bug(core): failed to get key device_canonical_map #574

New issue

Closed

opened 2026-03-28 04:25:54 +00:00 by mfreeman451 · 1 comment

mfreeman451 commented

2026-03-28 04:25:54 +00:00

Owner

Imported from GitHub.

Original GitHub issue: #1787
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1787
Original created: 2025-10-16T14:18:27Z

Describe the bug

{"level":"warn","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","key":"device_canonical_map/device-id/default=3A10.14
0.189.231","error":"all 3 retries failed: rpc error: code = Internal desc = failed to get key device_canonical_map/device-id/default=3A10.140.189.231: failed to ge
t key device_canonical_map/device-id/default=3A10.140.189.231: nats: connection closed","time":"2025-10-16T14:16:55Z","message":"KV identity publish exhausted retr
ies"}
{"level":"warn","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","key":"device_canonical_map/ip/10.43.220.55","error":
"all 3 retries failed: rpc error: code = Internal desc = failed to get key device_canonical_map/ip/10.43.220.55: failed to get key device_canonical_map/ip/10.43.22
0.55: nats: connection closed","time":"2025-10-16T14:17:02Z","message":"KV identity publish exhausted retries"}
{"level":"warn","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","key":"device_canonical_map/partition-ip/default=3A10
.43.220.55","error":"all 3 retries failed: rpc error: code = Internal desc = failed to get key device_canonical_map/partition-ip/default=3A10.43.220.55: failed to
get key device_canonical_map/partition-ip/default=3A10.43.220.55: nats: connection closed","time":"2025-10-16T14:17:08Z","message":"KV identity publish exhausted r
etries"}
{"level":"warn","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","error":"publish identity key device_canonical_map/de
vice-id/default=3A10.140.189.231: all 3 retries failed: rpc error: code = Internal desc = failed to get key device_canonical_map/device-id/default=3A10.140.189.231
: failed to get key device_canonical_map/device-id/default=3A10.140.189.231: nats: connection closed\npublish identity key device_canonical_map/ip/10.43.220.55: al
l 3 retries failed: rpc error: code = Internal desc = failed to get key device_canonical_map/ip/10.43.220.55: failed to get key device_canonical_map/ip/10.43.220.5
5: nats: connection closed\npublish identity key device_canonical_map/partition-ip/default=3A10.43.220.55: all 3 retries failed: rpc error: code = Internal desc =
failed to get key device_canonical_map/partition-ip/default=3A10.43.220.55: failed to get key device_canonical_map/partition-ip/default=3A10.43.220.55: nats: conne
ction closed","time":"2025-10-16T14:17:08Z","message":"Failed to publish identity map updates"}
{"level":"info","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","update_count":2,"total_batches":1,"time":"2025-10-16
T14:17:08Z","message":"Successfully published device updates to device_updates stream"}
{"level":"info","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","incoming_updates":1,"valid_updates":1,"published_upd
ates":2,"dropped_empty_ip":0,"canonicalized_by_armis_id":0,"canonicalized_by_netbox_id":0,"canonicalized_by_mac":0,"tombstones_emitted":1,"skipped_sweep_no_identit
y":0,"time":"2025-10-16T14:17:08Z","message":"Registry batch processed"}
{"level":"info","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","metric_count":0,"poller_id":"k8s-poller","timestamp"
:"2025-10-16T14:16:18Z","time":"2025-10-16T14:17:08Z","message":"Parsed rperf metrics"}
{"level":"warn","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","error":"all 3 retries failed: rpc error: code = Inte
rnal desc = failed to get key device_canonical_map/device-id/default=3A10.43.220.55: failed to get key device_canonical_map/device-id/default=3A10.43.220.55: nats:
 connection closed","time":"2025-10-16T14:17:11Z","message":"KV canonical hydration failed"}

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Imported from GitHub. Original GitHub issue: #1787 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1787 Original created: 2025-10-16T14:18:27Z --- **Describe the bug** ``` {"level":"warn","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","key":"device_canonical_map/device-id/default=3A10.14 0.189.231","error":"all 3 retries failed: rpc error: code = Internal desc = failed to get key device_canonical_map/device-id/default=3A10.140.189.231: failed to ge t key device_canonical_map/device-id/default=3A10.140.189.231: nats: connection closed","time":"2025-10-16T14:16:55Z","message":"KV identity publish exhausted retr ies"} {"level":"warn","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","key":"device_canonical_map/ip/10.43.220.55","error": "all 3 retries failed: rpc error: code = Internal desc = failed to get key device_canonical_map/ip/10.43.220.55: failed to get key device_canonical_map/ip/10.43.22 0.55: nats: connection closed","time":"2025-10-16T14:17:02Z","message":"KV identity publish exhausted retries"} {"level":"warn","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","key":"device_canonical_map/partition-ip/default=3A10 .43.220.55","error":"all 3 retries failed: rpc error: code = Internal desc = failed to get key device_canonical_map/partition-ip/default=3A10.43.220.55: failed to get key device_canonical_map/partition-ip/default=3A10.43.220.55: nats: connection closed","time":"2025-10-16T14:17:08Z","message":"KV identity publish exhausted r etries"} {"level":"warn","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","error":"publish identity key device_canonical_map/de vice-id/default=3A10.140.189.231: all 3 retries failed: rpc error: code = Internal desc = failed to get key device_canonical_map/device-id/default=3A10.140.189.231 : failed to get key device_canonical_map/device-id/default=3A10.140.189.231: nats: connection closed\npublish identity key device_canonical_map/ip/10.43.220.55: al l 3 retries failed: rpc error: code = Internal desc = failed to get key device_canonical_map/ip/10.43.220.55: failed to get key device_canonical_map/ip/10.43.220.5 5: nats: connection closed\npublish identity key device_canonical_map/partition-ip/default=3A10.43.220.55: all 3 retries failed: rpc error: code = Internal desc = failed to get key device_canonical_map/partition-ip/default=3A10.43.220.55: failed to get key device_canonical_map/partition-ip/default=3A10.43.220.55: nats: conne ction closed","time":"2025-10-16T14:17:08Z","message":"Failed to publish identity map updates"} {"level":"info","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","update_count":2,"total_batches":1,"time":"2025-10-16 T14:17:08Z","message":"Successfully published device updates to device_updates stream"} {"level":"info","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","incoming_updates":1,"valid_updates":1,"published_upd ates":2,"dropped_empty_ip":0,"canonicalized_by_armis_id":0,"canonicalized_by_netbox_id":0,"canonicalized_by_mac":0,"tombstones_emitted":1,"skipped_sweep_no_identit y":0,"time":"2025-10-16T14:17:08Z","message":"Registry batch processed"} {"level":"info","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","metric_count":0,"poller_id":"k8s-poller","timestamp" :"2025-10-16T14:16:18Z","time":"2025-10-16T14:17:08Z","message":"Parsed rperf metrics"} {"level":"warn","trace_id":"e4b2122d4076f523ff813fdae495ee24","span_id":"d0b9ca36ce571277","component":"core","error":"all 3 retries failed: rpc error: code = Inte rnal desc = failed to get key device_canonical_map/device-id/default=3A10.43.220.55: failed to get key device_canonical_map/device-id/default=3A10.43.220.55: nats: connection closed","time":"2025-10-16T14:17:11Z","message":"KV canonical hydration failed"} ``` **To Reproduce** Steps to reproduce the behavior: 1. Go to '...' 2. Click on '....' 3. Scroll down to '....' 4. See error **Expected behavior** A clear and concise description of what you expected to happen. **Screenshots** If applicable, add screenshots to help explain your problem. **Desktop (please complete the following information):** - OS: [e.g. iOS] - Browser [e.g. chrome, safari] - Version [e.g. 22] **Smartphone (please complete the following information):** - Device: [e.g. iPhone6] - OS: [e.g. iOS8.1] - Browser [e.g. stock browser, safari] - Version [e.g. 22] **Additional context** Add any other context about the problem here.

mfreeman451 commented

2026-03-28 04:25:55 +00:00

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1787#issuecomment-3446903979
Original created: 2025-10-25T17:04:22Z

closing, cant repro with new improved datasync service

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1787#issuecomment-3446903979 Original created: 2025-10-25T17:04:22Z --- closing, cant repro with new improved datasync service