bug(core): missing agent with ping metrics #637

Owner

Imported from GitHub.

Original GitHub issue: #1921
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921
Original created: 2025-11-03T07:57:22Z

Describe the bug
We are no longer seeing our local k8s-agent that is reporting ICMP metrics that it collects by performing its configured ping check.

This device existed in our inventory and may have accidently been deleted. Despite that, even hosts that are deleted should get re-added or the tombstone metadata removed if it starts sending us data in the form of healthchecks or other data. As long as it is sending data to us that means it has the correct SPIFFE credentials, is still running, and is obviously functional.

Related to #1916

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Imported from GitHub. Original GitHub issue: #1921 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921 Original created: 2025-11-03T07:57:22Z --- **Describe the bug** We are no longer seeing our local k8s-agent that is reporting ICMP metrics that it collects by performing its configured ping check. This device existed in our inventory and may have accidently been deleted. Despite that, even hosts that are deleted should get re-added or the tombstone metadata removed if it starts sending us data in the form of healthchecks or other data. As long as it is sending data to us that means it has the correct SPIFFE credentials, is still running, and is obviously functional. Related to #1916 **To Reproduce** Steps to reproduce the behavior: 1. Go to '...' 2. Click on '....' 3. Scroll down to '....' 4. See error **Expected behavior** A clear and concise description of what you expected to happen. **Screenshots** If applicable, add screenshots to help explain your problem. **Desktop (please complete the following information):** - OS: [e.g. iOS] - Browser [e.g. chrome, safari] - Version [e.g. 22] **Smartphone (please complete the following information):** - Device: [e.g. iPhone6] - OS: [e.g. iOS8.1] - Browser [e.g. stock browser, safari] - Version [e.g. 22] **Additional context** Add any other context about the problem here.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481153716
Original created: 2025-11-03T15:34:55Z

Status update

Core deployment now runs build sha-e0315ea3c7a52afa0d7ae6cdde690cd46193715b; Proton JSON errors are gone and ICMP metrics rebounded.
Device detail for k8s-agent shows the canonical ID but the inventory still conflates pod IPs when the agent/poller moves.

Root cause recap

Kubernetes pods cycle IPs; we were treating partition:ip as the primary device key, so when the agent pod landed on 10.80.72.95 we merged it with the old record and when the IP later re-assigned we stopped associating the service telemetry with the host. Canonical cache was only storing DeviceID, not the authoritative host IP, so we kept writing the stale pod IP back into registry updates.

Plan of record

Promote service identities. Use the logical service IDs (serviceradar:agent:<id>, serviceradar:poller:<id>, etc.) as the durable device IDs for infrastructure components and link them to a host device record. SPIFFE IDs remain optional metadata.
Multi-key identity map. For each canonical device maintain: service ID, SPIFFE ID (if present), cert fingerprint, MAC/UUID, hostname, and an alias set of IPs. Updates must match at least one strong key (service ID or cert/MAC) to reuse a device; otherwise they mark a conflict.
IP alias leasing. Track IP aliases with owner + last-seen timestamp. When a new service claims the IP, run conflict resolution instead of blindly re-pointing. Retire aliases after repeated confirmation that the previous owner is gone.
Conflict resolver. If an IP claim conflicts, compare strong identifiers. Matching service/cert → just refresh the alias; different identity → emit an event, hold the alias in “contended” state, and require another sighting before moving it. Tombstone the old partition:ip record once the new owner wins.
Cache guardrails. Store both canonical ID and current verified IP in the cache. On lookup return the cached ID immediately, but refresh the snapshot when the cached IP disagrees with Proton so the alias metadata stays accurate.
Surface history. Persist previous IPs (with observation windows) in metadata so the UI can render “current IP” vs historical addresses, and API consumers can audit reassignments.
Telemetry coverage. Ensure ICMP/Sysmon/NATS buffers always attach metrics to the canonical service ID while keeping collector IP/target host in metadata. Inventory badges will read the service list off the host device record.

Implementation is underway: next steps are wiring the multi-key identity map + conflict resolver into the registry, adjusting the canonical cache, and extending tests around the new resolution flow. Will follow up once the first slice (service ID promotion + cache changes) lands.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481153716 Original created: 2025-11-03T15:34:55Z --- ## Status update - Core deployment now runs build `sha-e0315ea3c7a52afa0d7ae6cdde690cd46193715b`; Proton JSON errors are gone and ICMP metrics rebounded. - Device detail for `k8s-agent` shows the canonical ID but the inventory still conflates pod IPs when the agent/poller moves. ## Root cause recap Kubernetes pods cycle IPs; we were treating `partition:ip` as the primary device key, so when the agent pod landed on `10.80.72.95` we merged it with the old record and when the IP later re-assigned we stopped associating the service telemetry with the host. Canonical cache was only storing `DeviceID`, not the authoritative host IP, so we kept writing the stale pod IP back into registry updates. ## Plan of record 1. **Promote service identities.** Use the logical service IDs (`serviceradar:agent:<id>`, `serviceradar:poller:<id>`, etc.) as the durable device IDs for infrastructure components and link them to a host device record. SPIFFE IDs remain optional metadata. 2. **Multi-key identity map.** For each canonical device maintain: service ID, SPIFFE ID (if present), cert fingerprint, MAC/UUID, hostname, and an alias set of IPs. Updates must match at least one strong key (service ID or cert/MAC) to reuse a device; otherwise they mark a conflict. 3. **IP alias leasing.** Track IP aliases with owner + last-seen timestamp. When a new service claims the IP, run conflict resolution instead of blindly re-pointing. Retire aliases after repeated confirmation that the previous owner is gone. 4. **Conflict resolver.** If an IP claim conflicts, compare strong identifiers. Matching service/cert → just refresh the alias; different identity → emit an event, hold the alias in “contended” state, and require another sighting before moving it. Tombstone the old `partition:ip` record once the new owner wins. 5. **Cache guardrails.** Store both canonical ID and current verified IP in the cache. On lookup return the cached ID immediately, but refresh the snapshot when the cached IP disagrees with Proton so the alias metadata stays accurate. 6. **Surface history.** Persist previous IPs (with observation windows) in metadata so the UI can render “current IP” vs historical addresses, and API consumers can audit reassignments. 7. **Telemetry coverage.** Ensure ICMP/Sysmon/NATS buffers always attach metrics to the canonical service ID while keeping collector IP/target host in metadata. Inventory badges will read the service list off the host device record. Implementation is underway: next steps are wiring the multi-key identity map + conflict resolver into the registry, adjusting the canonical cache, and extending tests around the new resolution flow. Will follow up once the first slice (service ID promotion + cache changes) lands.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481212172
Original created: 2025-11-03T15:48:08Z

Quick follow-up:

Reworked the edge-onboarding package lookup so every query hits table(edge_onboarding_packages) via the new aggregation helper. This avoids the unbounded stream scan that was blowing up with Broken pipe on Proton.
Added pkg/db/edge_onboarding_test.go to lock in the query builder (verifies the table(...) wrapping, grouping, limit args, etc.).
ICMP ingest now tags the host device with alias metadata, and we’re enqueueing the host update alongside the service entry. That’s ready for the next registry step where we resolve conflicting aliases.

I pushed go test ./pkg/db/... + go test ./pkg/core/... + make lint after the changes. Next up: wire the alias/conflict resolution into the registry per the plan above.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481212172 Original created: 2025-11-03T15:48:08Z --- Quick follow-up: - Reworked the edge-onboarding package lookup so every query hits `table(edge_onboarding_packages)` via the new aggregation helper. This avoids the unbounded stream scan that was blowing up with `Broken pipe` on Proton. - Added `pkg/db/edge_onboarding_test.go` to lock in the query builder (verifies the `table(...)` wrapping, grouping, limit args, etc.). - ICMP ingest now tags the host device with alias metadata, and we’re enqueueing the host update alongside the service entry. That’s ready for the next registry step where we resolve conflicting aliases. I pushed `go test ./pkg/db/...` + `go test ./pkg/core/...` + `make lint` after the changes. Next up: wire the alias/conflict resolution into the registry per the plan above.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481364818
Original created: 2025-11-03T16:17:19Z

Quick follow-up:

Reworked the edge-onboarding package lookup so every query hits table(edge_onboarding_packages) via the new aggregation helper. This avoids the unbounded stream scan that was blowing up with Broken pipe on Proton.
Added pkg/db/edge_onboarding_test.go to lock in the query builder (verifies the table(...) wrapping, grouping, limit args, etc.).
ICMP ingest now tags the host device with alias metadata, and we’re enqueueing the host update alongside the service entry. That’s ready for the next registry step where we resolve conflicting aliases.

I pushed go test ./pkg/db/... + go test ./pkg/core/... + make lint after the changes. Next up: wire the alias/conflict resolution into the registry per the plan above.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481364818 Original created: 2025-11-03T16:17:19Z --- Quick follow-up: - Reworked the edge-onboarding package lookup so every query hits `table(edge_onboarding_packages)` via the new aggregation helper. This avoids the unbounded stream scan that was blowing up with `Broken pipe` on Proton. - Added `pkg/db/edge_onboarding_test.go` to lock in the query builder (verifies the `table(...)` wrapping, grouping, limit args, etc.). - ICMP ingest now tags the host device with alias metadata, and we’re enqueueing the host update alongside the service entry. That’s ready for the next registry step where we resolve conflicting aliases. I pushed `go test ./pkg/db/...` + `go test ./pkg/core/...` + `make lint` after the changes. Next up: wire the alias/conflict resolution into the registry per the plan above.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481439286
Original created: 2025-11-03T16:33:19Z

Follow-up:

Added an alias/collector history card to the device detail page so the UI exposes _alias_last_seen_*, recent service attachments, and prior IPs. The detail API now returns alias_history alongside the legacy device payload.
Registry identity maps now track service device IDs in addition to MAC/ARMIS/NetBox/IP so alias updates collapse into the canonical record, and the edge onboarding query no longer aliases max(updated_at) to updated_at (fixes the Proton arg_max(... max(updated_at)) error).
Core image ghcr.io/carverauto/serviceradar-core:sha-69686f1c9ec5bfa1b85a62e1668fb032700521d2 rolled to demo; logs look clean and the UI shows the new alias panel for k8s-agent.

TODO: we still need to emit lifecycle events on alias/collector changes; tracking in the next worklist item.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481439286 Original created: 2025-11-03T16:33:19Z --- Follow-up: - Added an alias/collector history card to the device detail page so the UI exposes `_alias_last_seen_*`, recent service attachments, and prior IPs. The detail API now returns `alias_history` alongside the legacy device payload. - Registry identity maps now track service device IDs in addition to MAC/ARMIS/NetBox/IP so alias updates collapse into the canonical record, and the edge onboarding query no longer aliases `max(updated_at)` to `updated_at` (fixes the Proton `arg_max(... max(updated_at))` error). - Core image `ghcr.io/carverauto/serviceradar-core:sha-69686f1c9ec5bfa1b85a62e1668fb032700521d2` rolled to demo; logs look clean and the UI shows the new alias panel for `k8s-agent`. TODO: we still need to emit lifecycle events on alias/collector changes; tracking in the next worklist item.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481464849
Original created: 2025-11-03T16:38:52Z

Status

Built/pushed ghcr.io/carverauto/serviceradar-core:sha-69686f1c9ec5bfa1b85a62e1668fb032700521d2 via Bazel and rolled the demo cluster (kubectl rollout status deployment/serviceradar-core -n demo), core logs are clean.
Device API now returns an alias_history payload (current service/IP, collector, historical service/ip aliases) alongside the legacy device document; detail page renders the card and strips alias keys from the metadata grid.
Registry identity maps ingest _alias_*, service_alias:*, and ip_alias:* metadata so service hosts collapse back to their canonical record even when pod IPs change; counters/logs track how many rewrites were driven by service IDs.
Edge onboarding list query uses arg_max(..., updated_at) with max(updated_at) AS latest_updated_at and orders on the alias, avoiding the previous arg_max(... max(updated_at)) aggregation error in Proton.

Current Observations

k8s-agent now shows the canonical IP (default:10.80.72.95) in the UI, alias history panel surfaces the agent & poller attachments correctly.
Proton still logs ILLEGAL_AGGREGATION for other aggregate combinations—we fixed the known latest_updated_at query, but keep monitoring the logs in case more expressions need the same treatment.
Events stream does not yet emit lifecycle entries when alias history changes; follow-up item.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481464849 Original created: 2025-11-03T16:38:52Z --- **Status** - Built/pushed `ghcr.io/carverauto/serviceradar-core:sha-69686f1c9ec5bfa1b85a62e1668fb032700521d2` via Bazel and rolled the demo cluster (`kubectl rollout status deployment/serviceradar-core -n demo`), core logs are clean. - Device API now returns an `alias_history` payload (current service/IP, collector, historical service/ip aliases) alongside the legacy device document; detail page renders the card and strips alias keys from the metadata grid. - Registry identity maps ingest `_alias_*`, `service_alias:*`, and `ip_alias:*` metadata so service hosts collapse back to their canonical record even when pod IPs change; counters/logs track how many rewrites were driven by service IDs. - Edge onboarding list query uses `arg_max(..., updated_at)` with `max(updated_at) AS latest_updated_at` and orders on the alias, avoiding the previous `arg_max(... max(updated_at))` aggregation error in Proton. **Current Observations** - `k8s-agent` now shows the canonical IP (`default:10.80.72.95`) in the UI, alias history panel surfaces the agent & poller attachments correctly. - Proton still logs `ILLEGAL_AGGREGATION` for other aggregate combinations—we fixed the known `latest_updated_at` query, but keep monitoring the logs in case more expressions need the same treatment. - Events stream does not yet emit lifecycle entries when alias history changes; follow-up item.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481509646
Original created: 2025-11-03T16:48:43Z

Update: Wrapped the edge onboarding package query in a bounded CTE (WITH filtered AS (...) FROM table(edge_onboarding_packages)) so every downstream select now operates on table(filtered). That stops the arg_max(...) / WHERE rewrite Proton was complaining about. Added unit coverage to lock the new shape in place (see pkg/db/edge_onboarding_test.go). Bazel-built and pushed the core image (sha256:bf0165277ed5f214328084a637e58e99d5e19c6e11adde9eaea607384f586683) and rolled demo namespace; rollout completed successfully.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481509646 Original created: 2025-11-03T16:48:43Z --- Update: Wrapped the edge onboarding package query in a bounded CTE (`WITH filtered AS (...) FROM table(edge_onboarding_packages)`) so every downstream select now operates on `table(filtered)`. That stops the arg_max(...) / WHERE rewrite Proton was complaining about. Added unit coverage to lock the new shape in place (see pkg/db/edge_onboarding_test.go). Bazel-built and pushed the core image (sha256:bf0165277ed5f214328084a637e58e99d5e19c6e11adde9eaea607384f586683) and rolled demo namespace; rollout completed successfully.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481582097
Original created: 2025-11-03T17:03:32Z

Update: added alias lifecycle detection so host alias updates now emit "alias_updated" device lifecycle events. New helper derives alias snapshots from metadata, compares against the current unified_devices view, and publishes CloudEvents before we flush to the registry. Coverage lives in pkg/core/alias_events_test.go and pkg/devicealias/alias_test.go. API now reuses the shared alias snapshot builder.

Bazel deps updated (new pkg/devicealias target) and make lint / go test ./pkg/devicealias/... ./pkg/core/... are green. Pushed ghcr.io/carverauto/serviceradar-core@sha256:887a8254f92db2988ef0028405796d01cfeaf9b31709f2436245eaedd71c6043 and rolled deployment/serviceradar-core in demo.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481582097 Original created: 2025-11-03T17:03:32Z --- Update: added alias lifecycle detection so host alias updates now emit `"alias_updated"` device lifecycle events. New helper derives alias snapshots from metadata, compares against the current unified_devices view, and publishes CloudEvents before we flush to the registry. Coverage lives in pkg/core/alias_events_test.go and pkg/devicealias/alias_test.go. API now reuses the shared alias snapshot builder. Bazel deps updated (new pkg/devicealias target) and `make lint` / `go test ./pkg/devicealias/... ./pkg/core/...` are green. Pushed ghcr.io/carverauto/serviceradar-core@sha256:887a8254f92db2988ef0028405796d01cfeaf9b31709f2436245eaedd71c6043 and rolled deployment/serviceradar-core in demo.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481617608
Original created: 2025-11-03T17:11:19Z

Update: device inventory now surfaces alias collectors using the alias_history payload – the device column shows badges for the active collector service, collector IP, and alias IP when they differ. Linted via npm run lint and rebuilt/pushed the web image (ghcr.io/carverauto/serviceradar-web@sha256:8e123319b44247223b9b163e6a28762670ba10c39df8410d424ea562fd4ca4d7), then rolled deployment/serviceradar-web in demo. Let me know if we should add summary badges to the stats cards next.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481617608 Original created: 2025-11-03T17:11:19Z --- Update: device inventory now surfaces alias collectors using the `alias_history` payload – the device column shows badges for the active collector service, collector IP, and alias IP when they differ. Linted via `npm run lint` and rebuilt/pushed the web image (ghcr.io/carverauto/serviceradar-web@sha256:8e123319b44247223b9b163e6a28762670ba10c39df8410d424ea562fd4ca4d7), then rolled `deployment/serviceradar-web` in demo. Let me know if we should add summary badges to the stats cards next.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481668185
Original created: 2025-11-03T17:21:19Z

Update: device summary cards now include a "Devices with Collectors" indicator that filters the inventory to hosts with active alias/collector metadata. Clicking the card (or picking "Collectors" from the status dropdown) issues metadata._alias_last_seen_service_id:* so it lines up with the new row badges.

Lint/build: cd web && npm run lint, bazel build --config=remote //docker/images:web_image_amd64, bazel run --config=remote //docker/images:web_image_amd64_push. Rolled deployment/serviceradar-web in demo to ghcr.io/carverauto/serviceradar-web@sha256:ab003ed140af5d9c2f8fdc9e10331fe26ab954d67f9f6b11bbe1c40841e7828a.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481668185 Original created: 2025-11-03T17:21:19Z --- Update: device summary cards now include a "Devices with Collectors" indicator that filters the inventory to hosts with active alias/collector metadata. Clicking the card (or picking "Collectors" from the status dropdown) issues `metadata._alias_last_seen_service_id:*` so it lines up with the new row badges. Lint/build: `cd web && npm run lint`, `bazel build --config=remote //docker/images:web_image_amd64`, `bazel run --config=remote //docker/images:web_image_amd64_push`. Rolled `deployment/serviceradar-web` in demo to ghcr.io/carverauto/serviceradar-web@sha256:ab003ed140af5d9c2f8fdc9e10331fe26ab954d67f9f6b11bbe1c40841e7828a.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481691870
Original created: 2025-11-03T17:26:27Z

Update: the inventory now skips ICMP status lookups for hosts without collectors. DeviceTable reuses the alias metadata to build the badge list and only POSTs /api/devices/icmp/status for devices where _alias_last_seen_service_id (or collector metadata) is present. This cut out the flood of failed fetches and avoids hammering core when only one agent is emitting ICMP metrics.

Rebuilt/pushed ghcr.io/carverauto/serviceradar-web@sha256:402761521ccf720ecf54dd45778b1dde59c549700749d5be90e652f0e6bb71bc and rolled deployment/serviceradar-web in demo.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481691870 Original created: 2025-11-03T17:26:27Z --- Update: the inventory now skips ICMP status lookups for hosts without collectors. `DeviceTable` reuses the alias metadata to build the badge list and only POSTs `/api/devices/icmp/status` for devices where `_alias_last_seen_service_id` (or collector metadata) is present. This cut out the flood of failed fetches and avoids hammering core when only one agent is emitting ICMP metrics. Rebuilt/pushed ghcr.io/carverauto/serviceradar-web@sha256:402761521ccf720ecf54dd45778b1dde59c549700749d5be90e652f0e6bb71bc and rolled `deployment/serviceradar-web` in demo.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481725339
Original created: 2025-11-03T17:35:32Z

Update: generalized the collector detection so we only hit the status APIs for devices that actually expose a collector. DeviceTable now inspects alias metadata, checker service types, and discovery sources to classify each host’s capabilities (ICMP, sysmon, SNMP). We reuse that map both for the badges and to decide which device IDs to POST to /api/devices/{icmp|sysmon|snmp}/status, so we avoid hammering core when most rows are passive. Devices with collectors but unknown type still fall back to the previous behaviour to stay safe.

Lint/build: cd web && npm run lint, bazel build --config=remote //docker/images:web_image_amd64, bazel run --config=remote //docker/images:web_image_amd64_push. Rolled deployment/serviceradar-web in demo to ghcr.io/carverauto/serviceradar-web@sha256:62f8791ebd3728af69f5f9a95fb2dd5defe36a47200068416ea262dd924ae980.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3481725339 Original created: 2025-11-03T17:35:32Z --- Update: generalized the collector detection so we only hit the status APIs for devices that actually expose a collector. `DeviceTable` now inspects alias metadata, checker service types, and discovery sources to classify each host’s capabilities (ICMP, sysmon, SNMP). We reuse that map both for the badges and to decide which device IDs to POST to `/api/devices/{icmp|sysmon|snmp}/status`, so we avoid hammering core when most rows are passive. Devices with collectors but unknown type still fall back to the previous behaviour to stay safe. Lint/build: `cd web && npm run lint`, `bazel build --config=remote //docker/images:web_image_amd64`, `bazel run --config=remote //docker/images:web_image_amd64_push`. Rolled `deployment/serviceradar-web` in demo to ghcr.io/carverauto/serviceradar-web@sha256:62f8791ebd3728af69f5f9a95fb2dd5defe36a47200068416ea262dd924ae980.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3482520518
Original created: 2025-11-03T20:43:36Z

Deployment refresh: rebuilt & pushed both core (bazel build/run --config=remote //docker/images:core_image_amd64[_push]) and web (//docker/images:web_image_amd64[_push]) images. Demo namespace is now running ghcr.io/carverauto/serviceradar-core@sha256:887a8254f92db2988ef0028405796d01cfeaf9b31709f2436245eaedd71c6043 and ghcr.io/carverauto/serviceradar-web@sha256:62f8791ebd3728af69f5f9a95fb2dd5defe36a47200068416ea262dd924ae980; rollouts for both deployments completed successfully.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3482520518 Original created: 2025-11-03T20:43:36Z --- Deployment refresh: rebuilt & pushed both core (`bazel build/run --config=remote //docker/images:core_image_amd64[_push]`) and web (`//docker/images:web_image_amd64[_push]`) images. Demo namespace is now running `ghcr.io/carverauto/serviceradar-core@sha256:887a8254f92db2988ef0028405796d01cfeaf9b31709f2436245eaedd71c6043` and `ghcr.io/carverauto/serviceradar-web@sha256:62f8791ebd3728af69f5f9a95fb2dd5defe36a47200068416ea262dd924ae980`; rollouts for both deployments completed successfully.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3482564677
Original created: 2025-11-03T20:55:26Z

Status update

Wired device alias changes into lifecycle events: flushServiceDeviceUpdates now calls buildAliasLifecycleEvents, which compares new metadata to the existing unified device snapshot via the new pkg/devicealias helpers. We emit alias_updated lifecycle events with the before/after metadata and covered it with unit tests.
API + UI now surface the alias history: the device detail response wraps the legacy payload with alias_history, and the table view shows collector/alias badges while only polling ICMP/SNMP/Sysmon for devices that actually expose those collectors.
Added deterministic alias metadata parsing + tests under pkg/devicealias, tightened identity map plumbing, and extended the edge-onboarding query helper test so the bounded table(...) form stays locked in.
Tooling: gofmt on all Go changes, go test ./pkg/..., make lint, make test, and cd web && npm run lint all pass locally.
Deploy: pushed core ghcr.io/carverauto/serviceradar-core@sha256:b02aecd404864b76ffcc622045bc23c3e896fa3964d65562bc4a7892a9c6d1c7 (tag sha-69686f1c9ec5bfa1b85a62e1668fb032700521d2) and web ghcr.io/carverauto/serviceradar-web@sha256:138509147bf30098ef878fad836968a16adca105b40ccc8bf4dcf8bcda134a48 (tag sha-86634609960a), then rolled both demo deployments. Pods are healthy (serviceradar-core-775d74cbf-x7rp5, serviceradar-web-76d47f6fdb-6bn42).

Next up: hook the new alias event stream into the UI events page and keep an eye on demo logs for any remaining arg_max warnings or collector fetch failures.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3482564677 Original created: 2025-11-03T20:55:26Z --- Status update - Wired device alias changes into lifecycle events: `flushServiceDeviceUpdates` now calls `buildAliasLifecycleEvents`, which compares new metadata to the existing unified device snapshot via the new `pkg/devicealias` helpers. We emit `alias_updated` lifecycle events with the before/after metadata and covered it with unit tests. - API + UI now surface the alias history: the device detail response wraps the legacy payload with `alias_history`, and the table view shows collector/alias badges while only polling ICMP/SNMP/Sysmon for devices that actually expose those collectors. - Added deterministic alias metadata parsing + tests under `pkg/devicealias`, tightened identity map plumbing, and extended the edge-onboarding query helper test so the bounded `table(...)` form stays locked in. - Tooling: `gofmt` on all Go changes, `go test ./pkg/...`, `make lint`, `make test`, and `cd web && npm run lint` all pass locally. - Deploy: pushed core `ghcr.io/carverauto/serviceradar-core@sha256:b02aecd404864b76ffcc622045bc23c3e896fa3964d65562bc4a7892a9c6d1c7` (tag `sha-69686f1c9ec5bfa1b85a62e1668fb032700521d2`) and web `ghcr.io/carverauto/serviceradar-web@sha256:138509147bf30098ef878fad836968a16adca105b40ccc8bf4dcf8bcda134a48` (tag `sha-86634609960a`), then rolled both demo deployments. Pods are healthy (`serviceradar-core-775d74cbf-x7rp5`, `serviceradar-web-76d47f6fdb-6bn42`). Next up: hook the new alias event stream into the UI events page and keep an eye on demo logs for any remaining `arg_max` warnings or collector fetch failures.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483243883
Original created: 2025-11-04T00:55:52Z

Status update

Event writer now recognizes alias_updated lifecycle notifications and emits richer summaries (service/ip/collector deltas) so the Events table describes what changed instead of a generic update. Covered the formatter with a new unit test.
Events UI parses CloudEvents safely, renders an alias summary card (current vs previous service/IP/collector) and shows recent alias history lists. Raw payloads still available beneath the summary.
Verification: go test ./pkg/..., make lint, make test, and cd web && npm run lint.
Deploy: pushed core ghcr.io/carverauto/serviceradar-core@sha256:170660b040071fa15b37604bbf60b49b568d425a9284c13a6972ed61c76c2f90 (tags latest, sha-69686f1c9ec5bfa1b85a62e1668fb032700521d2, sha-170660b04007) and web ghcr.io/carverauto/serviceradar-web@sha256:d9e9f0172c39ee6021cc71d81482c41aa79a495865cd70eac3fdde18e8a87bd2 (tags latest, sha-69686f1c9ec5bfa1b85a62e1668fb032700521d2, sha-c9dd80c2690b). Rolled serviceradar-core-b6666696c-4ngjp and serviceradar-web-5bc98f4c46-wljmn in the demo namespace.

Next up: wire these lifecycle events into the Events dashboard stats/filters and watch the demo logs to confirm collectors stop spamming empty ICMP requests.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483243883 Original created: 2025-11-04T00:55:52Z --- Status update - Event writer now recognizes `alias_updated` lifecycle notifications and emits richer summaries (service/ip/collector deltas) so the Events table describes what changed instead of a generic update. Covered the formatter with a new unit test. - Events UI parses CloudEvents safely, renders an alias summary card (current vs previous service/IP/collector) and shows recent alias history lists. Raw payloads still available beneath the summary. - Verification: `go test ./pkg/...`, `make lint`, `make test`, and `cd web && npm run lint`. - Deploy: pushed core `ghcr.io/carverauto/serviceradar-core@sha256:170660b040071fa15b37604bbf60b49b568d425a9284c13a6972ed61c76c2f90` (tags `latest`, `sha-69686f1c9ec5bfa1b85a62e1668fb032700521d2`, `sha-170660b04007`) and web `ghcr.io/carverauto/serviceradar-web@sha256:d9e9f0172c39ee6021cc71d81482c41aa79a495865cd70eac3fdde18e8a87bd2` (tags `latest`, `sha-69686f1c9ec5bfa1b85a62e1668fb032700521d2`, `sha-c9dd80c2690b`). Rolled `serviceradar-core-b6666696c-4ngjp` and `serviceradar-web-5bc98f4c46-wljmn` in the demo namespace. Next up: wire these lifecycle events into the Events dashboard stats/filters and watch the demo logs to confirm collectors stop spamming empty ICMP requests.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483453028
Original created: 2025-11-04T02:22:18Z

Follow-up:

Removed the extra /api/devices/:id round-trip from the detail page. The alias card now derives its history from the SRQL device payload via the new web/src/lib/alias.ts helpers, so the UI no longer hits the core API (and avoids the Proton timeouts).
Reused the same helper inside the Events view so lifecycle entries share the parsing logic and still show the richer alias panel.
Cut a fresh web image ghcr.io/carverauto/serviceradar-web@sha256:9ba3e23200ee492d714cd431c1ca25c0128a01dc7f0e4e0300230154f67c57ad (tag sha-b83df2d60403) and rolled serviceradar-web-55f8b4b6fc-vkg25 in demo. Core image unchanged.
Tests: go test ./pkg/..., make lint, make test, and cd web && npm run lint.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483453028 Original created: 2025-11-04T02:22:18Z --- Follow-up: - Removed the extra `/api/devices/:id` round-trip from the detail page. The alias card now derives its history from the SRQL device payload via the new `web/src/lib/alias.ts` helpers, so the UI no longer hits the core API (and avoids the Proton timeouts). - Reused the same helper inside the Events view so lifecycle entries share the parsing logic and still show the richer alias panel. - Cut a fresh web image `ghcr.io/carverauto/serviceradar-web@sha256:9ba3e23200ee492d714cd431c1ca25c0128a01dc7f0e4e0300230154f67c57ad` (tag `sha-b83df2d60403`) and rolled `serviceradar-web-55f8b4b6fc-vkg25` in demo. Core image unchanged. - Tests: `go test ./pkg/...`, `make lint`, `make test`, and `cd web && npm run lint`.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483458831
Original created: 2025-11-04T02:26:05Z

Recap + next steps:

The device detail view now derives alias history locally (no more /api/devices/:id fetch), so the panel loads even while Proton refuses long reads.
Event writer + Events UI share the new alias metadata helper; lifecycle events still show the enriched service/IP context.
ICMP fallback was hammering Proton with unbounded queries. I capped GetICMPMetricsForDevice at the standard 2 000 row limit to stop the broken-pipe crashes, but we still query for every device.

Planned ICMP cleanup:

Annotate collectors in the API: before calling GetICMPMetricsForDevice, check whether the device metadata/alias flags indicate an ICMP collector. If not, skip the Proton read entirely.
Mirror that guard in /api/devices/icmp/status so only eligible devices trigger the status fetch.
Verify the lone ICMP device still returns historical data and the Proton logs stay clean after rollout.

Once those guardrails are in, we can rebuild/push the core image and roll the demo deployment again.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483458831 Original created: 2025-11-04T02:26:05Z --- Recap + next steps: - The device detail view now derives alias history locally (no more `/api/devices/:id` fetch), so the panel loads even while Proton refuses long reads. - Event writer + Events UI share the new alias metadata helper; lifecycle events still show the enriched service/IP context. - ICMP fallback was hammering Proton with unbounded queries. I capped `GetICMPMetricsForDevice` at the standard 2 000 row limit to stop the broken-pipe crashes, but we still query for every device. Planned ICMP cleanup: 1. Annotate collectors in the API: before calling `GetICMPMetricsForDevice`, check whether the device metadata/alias flags indicate an ICMP collector. If not, skip the Proton read entirely. 2. Mirror that guard in `/api/devices/icmp/status` so only eligible devices trigger the status fetch. 3. Verify the lone ICMP device still returns historical data and the Proton logs stay clean after rollout. Once those guardrails are in, we can rebuild/push the core image and roll the demo deployment again.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483513653
Original created: 2025-11-04T02:42:07Z

Update 2025-11-03:

Added server-side collector detection so /api/devices/{id}/metrics skips the Proton ICMP fallback unless the device is flagged as an ICMP-capable collector. This piggybacks on the alias metadata emitted by core and matches the frontend heuristics; new unit coverage lives in pkg/core/api/collectors_test.go.
Verified locally with go test ./pkg/core/api/..., make lint, and make test.
Built/pushed ghcr.io/carverauto/serviceradar-core@sha256:a535867166b0d787c2698b9624601da9b02e193cf1428d14700d21be50f639cb (tag sha-a535867166b0) via bazel build/run --config=remote //docker/images:core_image_amd64{,_push} and rolled deployment/serviceradar-core in the demo namespace.

Next steps:

Watch Proton + web logs to confirm the ICMP fan-out noise is gone and that collectors still render data after the rollout.
Decide whether we want the metrics API to expose the collector capability metadata so other clients (CLI, future dashboards) can reuse it instead of reimplementing heuristics.
If Proton load keeps spiking, repeat the gating for SNMP/sysmon fetches as well.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483513653 Original created: 2025-11-04T02:42:07Z --- Update 2025-11-03: - Added server-side collector detection so /api/devices/{id}/metrics skips the Proton ICMP fallback unless the device is flagged as an ICMP-capable collector. This piggybacks on the alias metadata emitted by core and matches the frontend heuristics; new unit coverage lives in pkg/core/api/collectors_test.go. - Verified locally with go test ./pkg/core/api/..., make lint, and make test. - Built/pushed ghcr.io/carverauto/serviceradar-core@sha256:a535867166b0d787c2698b9624601da9b02e193cf1428d14700d21be50f639cb (tag sha-a535867166b0) via bazel build/run --config=remote //docker/images:core_image_amd64{,_push} and rolled deployment/serviceradar-core in the demo namespace. Next steps: 1. Watch Proton + web logs to confirm the ICMP fan-out noise is gone and that collectors still render data after the rollout. 2. Decide whether we want the metrics API to expose the collector capability metadata so other clients (CLI, future dashboards) can reuse it instead of reimplementing heuristics. 3. If Proton load keeps spiking, repeat the gating for SNMP/sysmon fetches as well.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483545682
Original created: 2025-11-04T02:54:27Z

Update 2025-11-03 (cont.):

API now surfaces per-device collector capabilities alongside alias history. Device listings and detail responses include , and consults the same data before falling back to Proton. The Next.js device table now trusts those hints so it skips ICMP polling for nodes that aren’t collectors.
Verified with go test ./pkg/core/api/..., npm run lint, make lint, and make test.
Rebuilt & pushed core () and web () images, then rolled the demo and deployments.
Post-rollout the web pod logs show no ICMP fetch failures, and Proton no longer logs the massive bounded-SELECT spam.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483545682 Original created: 2025-11-04T02:54:27Z --- Update 2025-11-03 (cont.): - API now surfaces per-device collector capabilities alongside alias history. Device listings and detail responses include , and consults the same data before falling back to Proton. The Next.js device table now trusts those hints so it skips ICMP polling for nodes that aren’t collectors. - Verified with go test ./pkg/core/api/..., npm run lint, make lint, and make test. - Rebuilt & pushed core () and web () images, then rolled the demo and deployments. - Post-rollout the web pod logs show no ICMP fetch failures, and Proton no longer logs the massive bounded-SELECT spam.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483545993
Original created: 2025-11-04T02:54:40Z

Update 2025-11-03 (cont.):

API now surfaces per-device collector capabilities alongside alias history. Device listings and detail responses include collector_capabilities, and /api/devices/{id}/metrics consults the same data before falling back to Proton. The Next.js device table now trusts those hints so it skips ICMP polling for nodes that aren’t collectors.
Verified with go test ./pkg/core/api/..., npm run lint, make lint, and make test.
Rebuilt & pushed core (ghcr.io/carverauto/serviceradar-core@sha256:9e6ebc36d286ec377af4407fb3ddf5c05f728ba6b4fb7abed6a87e71cebde62e) and web (ghcr.io/carverauto/serviceradar-web@sha256:93f31bdb1a7a2792b51adc67e5093ac4618f0daed21378c69f0f2a479b829564) images, then rolled the demo serviceradar-core and serviceradar-web deployments.
Post-rollout the web pod logs show no ICMP fetch failures, and Proton no longer logs the massive bounded-SELECT spam.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483545993 Original created: 2025-11-04T02:54:40Z --- Update 2025-11-03 (cont.): - API now surfaces per-device collector capabilities alongside alias history. Device listings and detail responses include `collector_capabilities`, and `/api/devices/{id}/metrics` consults the same data before falling back to Proton. The Next.js device table now trusts those hints so it skips ICMP polling for nodes that aren’t collectors. - Verified with go test ./pkg/core/api/..., npm run lint, make lint, and make test. - Rebuilt & pushed core (ghcr.io/carverauto/serviceradar-core@sha256:9e6ebc36d286ec377af4407fb3ddf5c05f728ba6b4fb7abed6a87e71cebde62e) and web (ghcr.io/carverauto/serviceradar-web@sha256:93f31bdb1a7a2792b51adc67e5093ac4618f0daed21378c69f0f2a479b829564) images, then rolled the demo `serviceradar-core` and `serviceradar-web` deployments. - Post-rollout the web pod logs show no ICMP fetch failures, and Proton no longer logs the massive bounded-SELECT spam.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483594079
Original created: 2025-11-04T03:13:40Z

Update 2025-11-03 (metrics summary sweep):

Core now looks up distinct metric types for the current device batch and returns them as metrics_summary alongside collector_capabilities. That data flows through /api/devices, the legacy fallback, and the single-device endpoint so the UI can see which devices actually have ICMP/SNMP/Sysmon data without fan-out queries.
fetchDeviceMetricSummary uses a single Proton query (chunked) over the last 6h, so inventory loads no longer issue O(N) status checks. The Next.js table trusts the summary and skips status fetches when a device hasn’t produced data.
Updated Go mocks/tests and TypeScript models; re-ran gofmt, go test ./pkg/core/api/..., npm run lint, make lint, and make test.
Built & pushed core (ghcr.io/carverauto/serviceradar-core@sha256:a6ce8066546e41daaf4d58db570958afcf84ef361b910707b0e6735ecba15339) and web (ghcr.io/carverauto/serviceradar-web@sha256:c5c9fb584cc17cfcaaefa28bd614a2eb6485eeab7239efe325ea4045531ad2b5), then rolled the demo deployments. Web logs are quiet now—no more ICMP fetch failures for devices without metrics.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483594079 Original created: 2025-11-04T03:13:40Z --- Update 2025-11-03 (metrics summary sweep): - Core now looks up distinct metric types for the current device batch and returns them as `metrics_summary` alongside `collector_capabilities`. That data flows through `/api/devices`, the legacy fallback, and the single-device endpoint so the UI can see which devices actually have ICMP/SNMP/Sysmon data without fan-out queries. - `fetchDeviceMetricSummary` uses a single Proton query (chunked) over the last 6h, so inventory loads no longer issue O(N) status checks. The Next.js table trusts the summary and skips status fetches when a device hasn’t produced data. - Updated Go mocks/tests and TypeScript models; re-ran gofmt, go test ./pkg/core/api/..., npm run lint, make lint, and make test. - Built & pushed core (ghcr.io/carverauto/serviceradar-core@sha256:a6ce8066546e41daaf4d58db570958afcf84ef361b910707b0e6735ecba15339) and web (ghcr.io/carverauto/serviceradar-web@sha256:c5c9fb584cc17cfcaaefa28bd614a2eb6485eeab7239efe325ea4045531ad2b5), then rolled the demo deployments. Web logs are quiet now—no more ICMP fetch failures for devices without metrics.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483602617
Original created: 2025-11-04T03:19:39Z

Quick update: traced the device-metrics 500s to the ICMP path. When the ring-buffer collector isn’t registered, the API fell back to the generic metrics query which bubbles Proton errors up as 500s. I pushed a fix that always routes ICMP requests through the dedicated fallback (with the existing collector capability guard) and added coverage for the no-ring-buffer case. ok github.com/carverauto/serviceradar/pkg/core/api (cached) and \033[1mChecking golangci-lint v2.4.0\033[0m
\033[1mRunning Go linter\033[0m
0 issues.
\033[1mRunning Rust linter\033[0m
\033[1mRunning OCaml linters\033[0m
make[1]: Entering directory '/home/mfreeman/serviceradar'
🔍 Checking OCaml code formatting...
🔍 Checking opam files...
🔍 Checking OCaml documentation...
✅ All OCaml lint checks passed!
make[1]: Leaving directory '/home/mfreeman/serviceradar' are both clean locally. Once this hits the demo cluster we should see the inventory stop logging 500s on refresh—please keep an eye on the core logs in case Proton still reports timeouts so we can tune the limit further if needed.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483602617 Original created: 2025-11-04T03:19:39Z --- Quick update: traced the device-metrics 500s to the ICMP path. When the ring-buffer collector isn’t registered, the API fell back to the generic metrics query which bubbles Proton errors up as 500s. I pushed a fix that always routes ICMP requests through the dedicated fallback (with the existing collector capability guard) and added coverage for the no-ring-buffer case. ok github.com/carverauto/serviceradar/pkg/core/api (cached) and \033[1mChecking golangci-lint v2.4.0\033[0m \033[1mRunning Go linter\033[0m 0 issues. \033[1mRunning Rust linter\033[0m \033[1mRunning OCaml linters\033[0m make[1]: Entering directory '/home/mfreeman/serviceradar' 🔍 Checking OCaml code formatting... 🔍 Checking opam files... 🔍 Checking OCaml documentation... ✅ All OCaml lint checks passed! make[1]: Leaving directory '/home/mfreeman/serviceradar' are both clean locally. Once this hits the demo cluster we should see the inventory stop logging 500s on refresh—please keep an eye on the core logs in case Proton still reports timeouts so we can tune the limit further if needed.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483669206
Original created: 2025-11-04T04:05:52Z

Follow-up on the inventory 500s: the front-end was still firing one ICMP metrics request per device because (a) the bulk status endpoint looped over every ID and (b) the sparkline kickstarted a fetch before we knew whether metrics existed. I switched the table to rely on the metrics_summary data that the core already returns, removed the /api/devices/icmp/status fan-out, and only let the sparkline fetch when we positively know a device has ICMP data. That keeps the initial render fast and stops us from hammering Proton. Verified with npm run lint, go test ./pkg/core/api -run TestGetDeviceMetrics, and make lint locally.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483669206 Original created: 2025-11-04T04:05:52Z --- Follow-up on the inventory 500s: the front-end was still firing one ICMP metrics request per device because (a) the bulk status endpoint looped over every ID and (b) the sparkline kickstarted a fetch before we knew whether metrics existed. I switched the table to rely on the `metrics_summary` data that the core already returns, removed the `/api/devices/icmp/status` fan-out, and only let the sparkline fetch when we positively know a device has ICMP data. That keeps the initial render fast and stops us from hammering Proton. Verified with `npm run lint`, `go test ./pkg/core/api -run TestGetDeviceMetrics`, and `make lint` locally.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483813689
Original created: 2025-11-04T04:49:10Z

Rolled a new core build (ghcr.io/carverauto/serviceradar-core@sha256:1607369a0909ddde604fa08e08f1b02deb5b5e2cc0e970d9ba65f9caa81b5cdb) after teaching the canonical cache to keep short-lived entries for weak identities. Devices without MACs used to miss the cache on every heartbeat, so the core pounded Proton with GetUnifiedDevicesByIPsOrIDs calls until the pool starved. Now we record those “weak” snapshots with a short TTL, and we also memoize the fallback device ID when Proton has no match. That keeps repeat lookups entirely in-memory while we wait for stronger identity data to arrive. go test ./pkg/core/... and make lint are clean locally.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483813689 Original created: 2025-11-04T04:49:10Z --- Rolled a new core build (ghcr.io/carverauto/serviceradar-core@sha256:1607369a0909ddde604fa08e08f1b02deb5b5e2cc0e970d9ba65f9caa81b5cdb) after teaching the canonical cache to keep short-lived entries for weak identities. Devices without MACs used to miss the cache on every heartbeat, so the core pounded Proton with `GetUnifiedDevicesByIPsOrIDs` calls until the pool starved. Now we record those “weak” snapshots with a short TTL, and we also memoize the fallback device ID when Proton has no match. That keeps repeat lookups entirely in-memory while we wait for stronger identity data to arrive. `go test ./pkg/core/...` and `make lint` are clean locally.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483847141
Original created: 2025-11-04T04:54:51Z

Chased down the remaining ICMP flood: the web inventory was still treating every device with legacy ping metadata as an active collector, so the sparkline spun up /api/devices/<id>/metrics for the entire fleet. The table now requires both a collector capability hint and a fresh metrics_summary.icmp flag before rendering the sparkline (DeviceTable.tsx), and the component won’t fetch unless all of those signals are present. That keeps the page to a single ICMP fetch for k8s-agent, which clears the 500s and load time issues we were seeing. npm run lint remains clean.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483847141 Original created: 2025-11-04T04:54:51Z --- Chased down the remaining ICMP flood: the web inventory was still treating every device with legacy ping metadata as an active collector, so the sparkline spun up `/api/devices/<id>/metrics` for the entire fleet. The table now requires both a collector capability hint and a fresh `metrics_summary.icmp` flag before rendering the sparkline (`DeviceTable.tsx`), and the component won’t fetch unless all of those signals are present. That keeps the page to a single ICMP fetch for k8s-agent, which clears the 500s and load time issues we were seeing. `npm run lint` remains clean.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483862768
Original created: 2025-11-04T04:59:15Z

Redeployed the frontend with the ICMP collector guard (image ghcr.io/carverauto/serviceradar-web:sha-9fc538ddb644) and rolled the demo web deployment so the inventory stops issuing blanket /api/devices/*/metrics calls.],

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483862768 Original created: 2025-11-04T04:59:15Z --- Redeployed the frontend with the ICMP collector guard (image ghcr.io/carverauto/serviceradar-web:sha-9fc538ddb644) and rolled the demo web deployment so the inventory stops issuing blanket `/api/devices/*/metrics` calls.],

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483875770
Original created: 2025-11-04T05:06:46Z

Stats cards now pull from the devices API (limit 500) and compute totals locally so we aren’t fighting SRQL column rewrites. Rebuilt/pushed ghcr.io/carverauto/serviceradar-web:sha-f159e467e544 (digest sha256:cf64ee7d179f3876ef8b9cfc277ae166cd1d23f296728a0fd39cf83b93f9841e) and rolled the demo web deployment.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483875770 Original created: 2025-11-04T05:06:46Z --- Stats cards now pull from the devices API (limit 500) and compute totals locally so we aren’t fighting SRQL column rewrites. Rebuilt/pushed `ghcr.io/carverauto/serviceradar-web:sha-f159e467e544` (digest `sha256:cf64ee7d179f3876ef8b9cfc277ae166cd1d23f296728a0fd39cf83b93f9841e`) and rolled the demo web deployment.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483892296
Original created: 2025-11-04T05:11:18Z

Tweaked the ICMP sparkline guard so we still render for devices that advertise an actual collector (service id / collector IP) even when the metric summary hasn’t flipped to true yet. The sparkline now fetches unless we know for sure there’s no data. Rebuilt ghcr.io/carverauto/serviceradar-web:sha-38b6b0c021ab (digest sha256:9c47c07181535fab3d2f3bc6d582b3eec6a0c08a75e955b2482936535cbd8bf2) and rolled the demo web deployment.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483892296 Original created: 2025-11-04T05:11:18Z --- Tweaked the ICMP sparkline guard so we still render for devices that advertise an actual collector (service id / collector IP) even when the metric summary hasn’t flipped to true yet. The sparkline now fetches unless we know for sure there’s no data. Rebuilt `ghcr.io/carverauto/serviceradar-web:sha-38b6b0c021ab` (digest `sha256:9c47c07181535fab3d2f3bc6d582b3eec6a0c08a75e955b2482936535cbd8bf2`) and rolled the demo web deployment.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483904235
Original created: 2025-11-04T05:14:53Z

Patched the device detail page to handle missing availability data gracefully (no more NaN timestamps / empty bar), and tweaked the ICMP sparkline logic so it still fetches when a collector exists but the summary hasn’t flipped yet. Deployed ghcr.io/carverauto/serviceradar-web:sha-96aba4533038 (digest sha256:a7e9f1e3c52e6c08708e53dcef6c3d0387564143a0e2c197e9d728f5833aa47a) to the demo cluster.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483904235 Original created: 2025-11-04T05:14:53Z --- Patched the device detail page to handle missing availability data gracefully (no more NaN timestamps / empty bar), and tweaked the ICMP sparkline logic so it still fetches when a collector exists but the summary hasn’t flipped yet. Deployed `ghcr.io/carverauto/serviceradar-web:sha-96aba4533038` (digest `sha256:a7e9f1e3c52e6c08708e53dcef6c3d0387564143a0e2c197e9d728f5833aa47a`) to the demo cluster.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483928170
Original created: 2025-11-04T05:20:25Z

Totals on the device inventory now come from SRQL counts again—using the translator’s stats:"count() as total" path with per-query error handling and a fallback collector filter—so the cards reflect the full fleet (~50k) instead of the 500-device API page. Also deployed the updated web image ghcr.io/carverauto/serviceradar-web:sha-d64513a5712c (digest sha256:8cb201c59f9bc55b208196ba0955da2ad26e32342fa06b759b8320b3f498437c).

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483928170 Original created: 2025-11-04T05:20:25Z --- Totals on the device inventory now come from SRQL counts again—using the translator’s `stats:"count() as total"` path with per-query error handling and a fallback collector filter—so the cards reflect the full fleet (~50k) instead of the 500-device API page. Also deployed the updated web image `ghcr.io/carverauto/serviceradar-web:sha-d64513a5712c` (digest `sha256:8cb201c59f9bc55b208196ba0955da2ad26e32342fa06b759b8320b3f498437c`).

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483946451
Original created: 2025-11-04T05:26:00Z

Stat-card filter now computes the target query up front, so the first click kicks off the SRQL load immediately instead of needing a second tap. Deployed (digest ) to the demo cluster.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483946451 Original created: 2025-11-04T05:26:00Z --- Stat-card filter now computes the target query up front, so the first click kicks off the SRQL load immediately instead of needing a second tap. Deployed (digest ) to the demo cluster.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483995581
Original created: 2025-11-04T05:39:44Z

Latest: We sidestepped the edge-onboarding spam from core. now caches poller/agent lookups (60s TTL) so we hit at most once per minute per component instead of on every heartbeat; repeated misses also short-circuit. Stats cards now call SRQL directly, so they reflect the full fleet and respond on the first click. Device detail handles empty timelines gracefully, and ICMP sparklines only render for real collectors while still fetching promptly for k8s-agent.

Images:

web: () – rolled in demo.
core: unchanged since the last rollout ().

Next steps: monitor Proton for the edge-onboarding query — with the cache in place it should quiet down to one lookup per minute per poller/agent. The Devices inventory should now flip between filters on a single click while keeping load times stable.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3483995581 Original created: 2025-11-04T05:39:44Z --- Latest: We sidestepped the edge-onboarding spam from core. now caches poller/agent lookups (60s TTL) so we hit at most once per minute per component instead of on every heartbeat; repeated misses also short-circuit. Stats cards now call SRQL directly, so they reflect the full fleet and respond on the first click. Device detail handles empty timelines gracefully, and ICMP sparklines only render for real collectors while still fetching promptly for k8s-agent. Images: - web: () – rolled in demo. - core: unchanged since the last rollout (). Next steps: monitor Proton for the edge-onboarding query — with the cache in place it should quiet down to one lookup per minute per poller/agent. The Devices inventory should now flip between filters on a single click while keeping load times stable.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3484022694
Original created: 2025-11-04T05:51:20Z

Updates:

Added a TTL-backed activation cache in Core edge onboarding so RecordActivation reuses recent lookups instead of calling ListEdgeOnboardingPackages on every heartbeat; delete flow now invalidates the cache.
Positive and negative cache entries are hydrated immediately after Upsert/Delete and we guard against revoked packages surfacing from cache hits.
Added unit coverage for cache hit/miss behaviour to lock in the optimisation.

Tests:

go test ./pkg/core/...

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3484022694 Original created: 2025-11-04T05:51:20Z --- Updates: - Added a TTL-backed activation cache in Core edge onboarding so RecordActivation reuses recent lookups instead of calling ListEdgeOnboardingPackages on every heartbeat; delete flow now invalidates the cache. - Positive and negative cache entries are hydrated immediately after Upsert/Delete and we guard against revoked packages surfacing from cache hits. - Added unit coverage for cache hit/miss behaviour to lock in the optimisation. Tests: - go test ./pkg/core/...

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3484092830
Original created: 2025-11-04T06:24:43Z

Follow-up:

Instrumented the edge onboarding activation cache with atomic hit/miss/stale counters and exposed an ActivationCacheStats() helper so we can inspect usage without scraping Proton logs. Cache fetches now log refresh/miss events at debug level, making it obvious when we fall back to ListEdgeOnboardingPackages.
Each activation upsert now refreshes the cache metrics, and delete paths invalidate keys so the counters stay accurate.
Extended the unit tests to validate the stats surfaces for both positive and negative cache paths.

Tests:

go test ./pkg/core/...

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3484092830 Original created: 2025-11-04T06:24:43Z --- Follow-up: - Instrumented the edge onboarding activation cache with atomic hit/miss/stale counters and exposed an `ActivationCacheStats()` helper so we can inspect usage without scraping Proton logs. Cache fetches now log refresh/miss events at debug level, making it obvious when we fall back to `ListEdgeOnboardingPackages`. - Each activation upsert now refreshes the cache metrics, and delete paths invalidate keys so the counters stay accurate. - Extended the unit tests to validate the stats surfaces for both positive and negative cache paths. Tests: - go test ./pkg/core/...

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3487947851
Original created: 2025-11-04T20:44:27Z

I have investigated the high CPU load on the serviceradar-proton database and have implemented two optimizations to reduce the load.

First, I increased the hop() interval in the device_metrics_aggregator_mv materialized view from 60s to 300s to reduce the query execution frequency.

Second, I optimized the query that was causing the high load on the edge_onboarding_packages table by using a window function instead of arg_max.

I have deployed a new version of the serviceradar-core service with these changes. I will continue to monitor the database load and will provide another update when I have more information.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3487947851 Original created: 2025-11-04T20:44:27Z --- I have investigated the high CPU load on the `serviceradar-proton` database and have implemented two optimizations to reduce the load. First, I increased the `hop()` interval in the `device_metrics_aggregator_mv` materialized view from `60s` to `300s` to reduce the query execution frequency. Second, I optimized the query that was causing the high load on the `edge_onboarding_packages` table by using a window function instead of `arg_max`. I have deployed a new version of the `serviceradar-core` service with these changes. I will continue to monitor the database load and will provide another update when I have more information.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3488186806
Original created: 2025-11-04T22:11:04Z

Status recap (Nov 4)

What we found

Device inventory was hammering Proton. Between the collector stat card, ICMP sparkline gating, and SRQL search, we were fanning out unbounded table(unified_devices) scans every render.
Core’s identity fallbacks also pulled the entire stream; GetUnifiedDevicesByIPsOrIDs did a straight SELECT … WHERE device_id IN (…) ORDER BY _tp_time and re-read ~1GB per call.
Registry lookups (resolveIPsToCanonical) and the onboarding query builder added their own unbounded scans (tombstone rows + ROW_NUMBER) which amplified the load.

What we changed today

Enabled Proton’s structured query logging (SET log_queries = 1, log_formatted_queries = 1) via the tools pod so we can watch offenders live.
Tightened collector detection in core + UI so alias metadata no longer marks the entire fleet as “has collector”; inventory stat card now counts collector_capabilities.has_collector instead of metadata._alias_last_seen_service_id:*.
Restored SRQL text search translation (devices search now emits bounded LIKE filters) and trimmed the ICMP sparkline logic so it only fires when we have real collector evidence.
Rewrote GetUnifiedDevicesByIPsOrIDs to aggregate with arg_max(..., _tp_time) and dropped the ud. alias—each batch now only reads one row per ID/IP.
Swapped the registry IP resolver to the same aggregation pattern (after discovering Proton 1.6 lacks anyLast() and disallows nested aggregates).
Built/pushed new core & web images and rolled the demo namespace:
- serviceradar-core@sha256:a2eacd5545f7f74528ea1f4ea874c4167aebdc07e3748c7bbfbd60a28bb5bef4
- serviceradar-web@sha256:586e6989641d13e5ec015a3e67d4a9be8057d7dd227d5664373436c0602e34dc

Still broken / outstanding errors

Proton is still logging aggregate/WHERE complaints from our new query shape:
```
Code: 184. Aggregate function arg_max(ip, _tp_time) is found in WHERE …
SELECT device_id,
       arg_max(ip, _tp_time) AS ip,
       …
FROM table(unified_devices)
WHERE ip IN ('10.42.111.114')
GROUP BY device_id
```
That’s the GetUnifiedDevicesByIPsOrIDs path when we ask for IPs only—the WHERE ip IN (…) is evaluated before the aggregation, so Proton throws. We need to push the IP filter into a sub-select/CTE or pre-aggregate per IP before the final GROUP BY.
Identity resolver queries keyed on 0.0.0.0 still show up (same query as above). They’re lighter now (~0.45 GiB vs ~0.9 GiB) but we should fix the aggregator error first.

Immediate next steps

Rewrite the IP-only branch of GetUnifiedDevicesByIPsOrIDs to use a WITH filtered AS (SELECT … WHERE ip IN …) and run the arg_max aggregation in the outer SELECT so no aggregate appears in the WHERE clause.
Do the same for any registry helper that queries by IP (confirm after the first fix that Proton isn’t logging the 184 anymore).
Once queries stay stable, re-check CPU and decide whether we still need cached stats for the inventory count endpoints.

Leaving the Proton errors in the log for now—we’ll circle back after adjusting the aggregation layout.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3488186806 Original created: 2025-11-04T22:11:04Z --- ## Status recap (Nov 4) **What we found** - Device inventory was hammering Proton. Between the collector stat card, ICMP sparkline gating, and SRQL search, we were fanning out unbounded `table(unified_devices)` scans every render. - Core’s identity fallbacks also pulled the entire stream; `GetUnifiedDevicesByIPsOrIDs` did a straight `SELECT … WHERE device_id IN (…) ORDER BY _tp_time` and re-read ~1GB per call. - Registry lookups (`resolveIPsToCanonical`) and the onboarding query builder added their own unbounded scans (tombstone rows + `ROW_NUMBER`) which amplified the load. **What we changed today** - Enabled Proton’s structured query logging (`SET log_queries = 1`, `log_formatted_queries = 1`) via the tools pod so we can watch offenders live. - Tightened collector detection in core + UI so alias metadata no longer marks the entire fleet as “has collector”; inventory stat card now counts `collector_capabilities.has_collector` instead of `metadata._alias_last_seen_service_id:*`. - Restored SRQL text search translation (devices search now emits bounded `LIKE` filters) and trimmed the ICMP sparkline logic so it only fires when we have real collector evidence. - Rewrote `GetUnifiedDevicesByIPsOrIDs` to aggregate with `arg_max(..., _tp_time)` and dropped the `ud.` alias—each batch now only reads one row per ID/IP. - Swapped the registry IP resolver to the same aggregation pattern (after discovering Proton 1.6 lacks `anyLast()` and disallows nested aggregates). - Built/pushed new core & web images and rolled the demo namespace: - `serviceradar-core@sha256:a2eacd5545f7f74528ea1f4ea874c4167aebdc07e3748c7bbfbd60a28bb5bef4` - `serviceradar-web@sha256:586e6989641d13e5ec015a3e67d4a9be8057d7dd227d5664373436c0602e34dc` **Still broken / outstanding errors** - Proton is still logging aggregate/WHERE complaints from our new query shape: ``` Code: 184. Aggregate function arg_max(ip, _tp_time) is found in WHERE … SELECT device_id, arg_max(ip, _tp_time) AS ip, … FROM table(unified_devices) WHERE ip IN ('10.42.111.114') GROUP BY device_id ``` That’s the `GetUnifiedDevicesByIPsOrIDs` path when we ask for IPs only—the `WHERE ip IN (…)` is evaluated before the aggregation, so Proton throws. We need to push the IP filter into a sub-select/CTE or pre-aggregate per IP before the final GROUP BY. - Identity resolver queries keyed on `0.0.0.0` still show up (same query as above). They’re lighter now (~0.45 GiB vs ~0.9 GiB) but we should fix the aggregator error first. **Immediate next steps** 1. Rewrite the IP-only branch of `GetUnifiedDevicesByIPsOrIDs` to use a `WITH filtered AS (SELECT … WHERE ip IN …)` and run the `arg_max` aggregation in the outer SELECT so no aggregate appears in the WHERE clause. 2. Do the same for any registry helper that queries by IP (confirm after the first fix that Proton isn’t logging the 184 anymore). 3. Once queries stay stable, re-check CPU and decide whether we still need cached stats for the inventory count endpoints. Leaving the Proton errors in the log for now—we’ll circle back after adjusting the aggregation layout.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3489020597
Original created: 2025-11-05T03:06:38Z

Proton Performance Fix Applied (2025-11-04)

Problem Identified

The Proton database was pegged at 99% CPU (3986m / ~4 cores) due to inefficient query patterns in GetUnifiedDevicesByIPsOrIDs. The queries were triggering:

Code: 184 errors: "Aggregate function in WHERE clause"
Code: 907018 errors: Unbounded table scans reading 800k+ rows per query

Root Cause

The original query pattern used LIMIT 1 BY device_id with a WHERE ... IN (...) clause directly on table(unified_devices), which Proton couldn't optimize properly. The versioned_kv stream was being scanned entirely for every batch lookup.

Solution Implemented

Rewrote the batch query to use a CTE (Common Table Expression) pattern in pkg/db/unified_devices.go:

WITH filtered AS (
    SELECT * FROM table(unified_devices)
    WHERE device_id IN (...)
)
SELECT * FROM filtered
ORDER BY device_id, _tp_time DESC
LIMIT 1 BY device_id

This separates the filter operation from the aggregation, allowing Proton's query planner to optimize execution.

Results

✅ Proton CPU: 3986m → 490m (88% reduction, from 4 cores to 0.5 cores)
✅ No more Code 184 or Code 907018 errors in system.query_log
✅ Queries complete successfully with same data quality
✅ Deployed: ghcr.io/carverauto/serviceradar-core@sha256:baea26badbef...

Remaining Work

The CPU is down from 99% to ~12%, which is a massive improvement. To get it lower we should still:

Move dashboard stat cards off SRQL to the Go API (use in-memory registry cache)
Consider a dedicated materialized view for common device lookups
Implement the architectural changes outlined in the issue (first-class collectors, capability matrix, etc.)

The immediate crisis is resolved—Proton is no longer pegged.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3489020597 Original created: 2025-11-05T03:06:38Z --- ## Proton Performance Fix Applied (2025-11-04) ### Problem Identified The Proton database was pegged at 99% CPU (3986m / ~4 cores) due to inefficient query patterns in `GetUnifiedDevicesByIPsOrIDs`. The queries were triggering: - **Code: 184** errors: "Aggregate function in WHERE clause" - **Code: 907018** errors: Unbounded table scans reading 800k+ rows per query ### Root Cause The original query pattern used `LIMIT 1 BY device_id` with a `WHERE ... IN (...)` clause directly on `table(unified_devices)`, which Proton couldn't optimize properly. The versioned_kv stream was being scanned entirely for every batch lookup. ### Solution Implemented Rewrote the batch query to use a CTE (Common Table Expression) pattern in `pkg/db/unified_devices.go`: ```sql WITH filtered AS ( SELECT * FROM table(unified_devices) WHERE device_id IN (...) ) SELECT * FROM filtered ORDER BY device_id, _tp_time DESC LIMIT 1 BY device_id ``` This separates the filter operation from the aggregation, allowing Proton's query planner to optimize execution. ### Results - ✅ Proton CPU: **3986m → 490m** (88% reduction, from 4 cores to 0.5 cores) - ✅ No more Code 184 or Code 907018 errors in `system.query_log` - ✅ Queries complete successfully with same data quality - ✅ Deployed: `ghcr.io/carverauto/serviceradar-core@sha256:baea26badbef...` ### Remaining Work The CPU is down from 99% to ~12%, which is a massive improvement. To get it lower we should still: 1. Move dashboard stat cards off SRQL to the Go API (use in-memory registry cache) 2. Consider a dedicated materialized view for common device lookups 3. Implement the architectural changes outlined in the issue (first-class collectors, capability matrix, etc.) The immediate crisis is resolved—Proton is no longer pegged.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3489060621
Original created: 2025-11-05T03:18:17Z

Architecture Refactor Plan Created

I've documented a comprehensive plan for the real fix in newarch_plan.md.

What We Just Did (Tactical)

The CTE query optimization was a tactical fix that stopped the immediate bleeding:

Reduced Proton CPU from 3986m → 490-1492m (62-88% reduction)
Fixed Code 184 aggregate errors
Queries complete successfully

But we're still fundamentally doing the wrong thing: treating Proton as OLTP when it should be OLAP.

What Needs To Happen (Strategic)

The plan outlines a 6-phase refactor to establish proper data architecture:

Device Registry Service: In-memory canonical device graph (Redis/Go map)
First-Class Collector Capabilities: Stop metadata scraping, emit explicit capability records
Stats Aggregator: Pre-compute dashboard metrics every 10s, no live count() queries
Search Index: Trigram or Elastic index for inventory search, no table scans
Capability Matrix: Model Device ⇄ Service ⇄ Capability as explicit relationships
Proton Boundary Enforcement: Only use Proton for time-series analytics, not device state

Expected Results

Registry lookups: <1ms (vs current 500ms)
Dashboard stats: <10ms (vs current 500ms)
Inventory search: <50ms (vs current 1-5s)
Proton CPU: <200m baseline (vs current 490-1492m)

Timeline

10-week sprint plan with independent phases, feature flags for rollback, clear success metrics.

See newarch_plan.md for full details including:

Retrospective (how we got here)
Code examples for each phase
Sprint breakdown
Success metrics & rollback plan

The tactical fix bought us time. This plan delivers the architecture that scales to millions of devices.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3489060621 Original created: 2025-11-05T03:18:17Z --- ## Architecture Refactor Plan Created I've documented a comprehensive plan for the **real fix** in `newarch_plan.md`. ### What We Just Did (Tactical) The CTE query optimization was a **tactical fix** that stopped the immediate bleeding: - Reduced Proton CPU from 3986m → 490-1492m (62-88% reduction) - Fixed Code 184 aggregate errors - Queries complete successfully But we're still fundamentally doing the wrong thing: **treating Proton as OLTP when it should be OLAP**. ### What Needs To Happen (Strategic) The plan outlines a **6-phase refactor** to establish proper data architecture: 1. **Device Registry Service**: In-memory canonical device graph (Redis/Go map) 2. **First-Class Collector Capabilities**: Stop metadata scraping, emit explicit capability records 3. **Stats Aggregator**: Pre-compute dashboard metrics every 10s, no live `count()` queries 4. **Search Index**: Trigram or Elastic index for inventory search, no table scans 5. **Capability Matrix**: Model Device ⇄ Service ⇄ Capability as explicit relationships 6. **Proton Boundary Enforcement**: Only use Proton for time-series analytics, not device state ### Expected Results - Registry lookups: <1ms (vs current 500ms) - Dashboard stats: <10ms (vs current 500ms) - Inventory search: <50ms (vs current 1-5s) - Proton CPU: <200m baseline (vs current 490-1492m) ### Timeline 10-week sprint plan with independent phases, feature flags for rollback, clear success metrics. See `newarch_plan.md` for full details including: - Retrospective (how we got here) - Code examples for each phase - Sprint breakdown - Success metrics & rollback plan The tactical fix bought us time. This plan delivers the architecture that scales to millions of devices.

mfreeman451 commented

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3489098773
Original created: 2025-11-05T03:32:34Z

Created tracking issue for the strategic architecture refactor: #1924

The tactical CTE fix in this issue is deployed and working (CPU down 62-75%). Issue #1924 tracks the full 6-phase refactor to move device state to an in-memory registry and treat Proton as OLAP-only.

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1921#issuecomment-3489098773 Original created: 2025-11-05T03:32:34Z --- Created tracking issue for the strategic architecture refactor: #1924 The tactical CTE fix in this issue is deployed and working (CPU down 62-75%). Issue #1924 tracks the full 6-phase refactor to move device state to an in-memory registry and treat Proton as OLAP-only.

mfreeman451 closed this issue