carverauto/serviceradar

Fork

You've already forked serviceradar

Code Issues 190 Pull requests 16 Projects Releases 173 Packages Wiki Activity Actions

Agent device cache #273

New issue

Closed

opened 2026-03-28 04:22:58 +00:00 by mfreeman451 · 0 comments

mfreeman451 commented

2026-03-28 04:22:58 +00:00

Owner

Copy link

Imported from GitHub.

Original GitHub issue: #769
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/769
Original created: 2025-05-12T17:22:45Z

Enhancement: Implement Device Information Collection in ServiceRadar Agent

Overview

This issue tracks the implementation of device information collection in the serviceradar-agent as outlined in the [[ServiceRadar Agent Device Caching & Reporting PRD](https://github.com/carverauto/serviceradar/tree/main/sr-architecture-and-design/prd)](https://github.com/carverauto/serviceradar/tree/main/sr-architecture-and-design/prd). The agent will continue collecting raw DeviceInfo from edge services (sweep, ICMP, port, SNMP) without local caching, include AgentID in responses, and rely on the core service to derive the devices stream using a Proton materialized view. Pollers will attach PollerID to status messages, ensuring traceability.

Description

The current agent collects raw data but lacks agent_id and poller_id tracking, and a previous plan for a local DeviceCache raised memory footprint concerns. This implementation will:

Preserve existing raw data collection, adding only agent_id to StatusResponse.
Use existing config fields for AgentID and PollerID instead of context propagation.
Enable pollers to attach PollerID to service status messages.
Resolve the OpenPorts reference issue in the sweep service.
Minimize agent memory usage by removing all device caching code.

The core service will derive the devices stream using a materialized view, aggregating and deduplicating raw data from sweep_results, icmp_results, and snmp_results streams.

Implementation Tasks

Phase 1: Core Infrastructure (Est: 3 days)

Update data structures in pkg/agent/types.go
- Remove DeviceCache, DeviceState, and DeviceCacheConfig.
- Ensure ServerConfig includes AgentID.
Update Protocol Buffers in proto/serviceradar.proto
- Add agent_id field to StatusResponse.
- Add agent_id and poller_id fields to ServiceStatus.
- Remove DeviceStatusRequest, DeviceStatusResponse, and devices from PollerStatusRequest.
Generate updated gRPC code from proto definitions.

Phase 2: Agent and Poller Updates (Est: 4 days)

Update Agent
- Remove pkg/agent/devices.go (entire file).
- Update GetStatus() in pkg/agent/server.go to add AgentID from config to StatusResponse.
- Update edge service response handling to ensure proper raw data collection.
- Remove device cache initialization, maintenance, and extraction functions.
Update Poller
- Update ServiceCheck.execute() in pkg/poller/poller.go to preserve AgentID from agent responses.
- Update reportToCore() to add PollerID to each ServiceStatus before reporting.
- Remove device-specific handling from poller methods.

Phase 3: Testing (Est: 4 days)

Unit Tests
- Test GetStatus for AgentID inclusion in StatusResponse.
- Test ServiceCheck.execute() for preserving AgentID.
- Test reportToCore() for adding PollerID.
Integration Tests
- Test agent-poller communication with agent_id and poller_id.
- Verify raw data consistency with existing collection.
System Tests
- End-to-end test from edge services to core with agent_id and poller_id.
- Performance test with 10,000+ devices, monitoring agent memory.

Notes

Updated approach uses existing config fields for AgentID and PollerID rather than context propagation.
Preserves existing data flow and raw data collection without modification.
Focuses on identity propagation through the pipeline without changing core functionality.
Removes all device caching code to minimize memory footprint.
Relies on core service materialized view for device derivation.

Acceptance Criteria

Agent includes agent_id in all StatusResponse messages.
Poller preserves agent_id from agent responses and adds poller_id to ServiceStatus messages.
Raw data collection remains unchanged, with no local caching.
OpenPorts is consistently populated in sweep data.
Agent memory usage remains minimal (<1 MB for 10,000 devices).
Raw data reaches the core within 5 minutes.
All tests pass, including performance tests.

Dependencies

Coordination with core team for materialized view implementation.

Labels

enhancement
device-management
serviceradar-agent
backend

Imported from GitHub. Original GitHub issue: #769 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/769 Original created: 2025-05-12T17:22:45Z --- # Enhancement: Implement Device Information Collection in ServiceRadar Agent ## Overview This issue tracks the implementation of device information collection in the `serviceradar-agent` as outlined in the [[[ServiceRadar Agent Device Caching & Reporting PRD](https://github.com/carverauto/serviceradar/tree/main/sr-architecture-and-design/prd)](https://github.com/carverauto/serviceradar/tree/main/sr-architecture-and-design/prd)](https://github.com/carverauto/serviceradar/tree/main/sr-architecture-and-design/prd). The agent will continue collecting raw `DeviceInfo` from edge services (sweep, ICMP, port, SNMP) without local caching, include `AgentID` in responses, and rely on the core service to derive the devices stream using a Proton materialized view. Pollers will attach `PollerID` to status messages, ensuring traceability. ## Description The current agent collects raw data but lacks `agent_id` and `poller_id` tracking, and a previous plan for a local `DeviceCache` raised memory footprint concerns. This implementation will: - Preserve existing raw data collection, adding only `agent_id` to `StatusResponse`. - Use existing config fields for `AgentID` and `PollerID` instead of context propagation. - Enable pollers to attach `PollerID` to service status messages. - Resolve the `OpenPorts` reference issue in the sweep service. - Minimize agent memory usage by removing all device caching code. The core service will derive the devices stream using a materialized view, aggregating and deduplicating raw data from `sweep_results`, `icmp_results`, and `snmp_results` streams. ## Implementation Tasks ### Phase 1: Core Infrastructure (Est: 3 days) - [x] Update data structures in `pkg/agent/types.go` - [x] Remove `DeviceCache`, `DeviceState`, and `DeviceCacheConfig`. - [x] Ensure `ServerConfig` includes `AgentID`. - [x] Update Protocol Buffers in `proto/serviceradar.proto` - [x] Add `agent_id` field to `StatusResponse`. - [x] Add `agent_id` and `poller_id` fields to `ServiceStatus`. - [x] Remove `DeviceStatusRequest`, `DeviceStatusResponse`, and `devices` from `PollerStatusRequest`. - [x] Generate updated gRPC code from proto definitions. ### Phase 2: Agent and Poller Updates (Est: 4 days) - [ ] Update Agent - [x] Remove `pkg/agent/devices.go` (entire file). - [x] Update `GetStatus()` in `pkg/agent/server.go` to add `AgentID` from config to `StatusResponse`. - [ ] Update edge service response handling to ensure proper raw data collection. - [x] Remove device cache initialization, maintenance, and extraction functions. - [x] Update Poller - [x] Update `ServiceCheck.execute()` in `pkg/poller/poller.go` to preserve `AgentID` from agent responses. - [x] Update `reportToCore()` to add `PollerID` to each `ServiceStatus` before reporting. - [x] Remove device-specific handling from poller methods. ### Phase 3: Testing (Est: 4 days) - [ ] Unit Tests - [ ] Test `GetStatus` for `AgentID` inclusion in `StatusResponse`. - [ ] Test `ServiceCheck.execute()` for preserving `AgentID`. - [ ] Test `reportToCore()` for adding `PollerID`. - [ ] Integration Tests - [ ] Test agent-poller communication with `agent_id` and `poller_id`. - [ ] Verify raw data consistency with existing collection. - [ ] System Tests - [ ] End-to-end test from edge services to core with `agent_id` and `poller_id`. - [ ] Performance test with 10,000+ devices, monitoring agent memory. ## Notes - Updated approach uses existing config fields for `AgentID` and `PollerID` rather than context propagation. - Preserves existing data flow and raw data collection without modification. - Focuses on identity propagation through the pipeline without changing core functionality. - Removes all device caching code to minimize memory footprint. - Relies on core service materialized view for device derivation. ## Acceptance Criteria 1. Agent includes `agent_id` in all `StatusResponse` messages. 2. Poller preserves `agent_id` from agent responses and adds `poller_id` to `ServiceStatus` messages. 3. Raw data collection remains unchanged, with no local caching. 4. `OpenPorts` is consistently populated in sweep data. 5. Agent memory usage remains minimal (<1 MB for 10,000 devices). 6. Raw data reaches the core within 5 minutes. 7. All tests pass, including performance tests. ## Dependencies - Coordination with core team for materialized view implementation. ## Labels - enhancement - device-management - serviceradar-agent - backend

mfreeman451

2026-03-28 04:22:58 +00:00

closed this issue
added the
back-end

device-management

enhancement

serviceradar-agent
labels

No Branch/Tag specified

staging

2406-feat-agent-fleet-management-secure-self-update-system

bug/k8s-helm-deployments

chore/k8s-arc-update

rust-fix

2371-analytics-stats-cards-should-abbreviate-numbers

chore/perl-cleanup

2942-featweb-ng-add-logs-tab-to-device-details-page

192-feat-tftp-server

mikemiles-dev/feature/netflow_collection

testing

dependabot/cargo/hostname-0.4.1

dependabot/cargo/redis-1.0.1

dependabot/cargo/bb8-0.9.0

dependabot/cargo/rcgen-0.14.6

dependabot/cargo/hyper-1.8.1

dependabot/cargo/hyper-util-0.1.19

dependabot/cargo/clap-4.5.51

dependabot/cargo/thiserror-2.0.17

dependabot/cargo/time-tz-2.0.0

dependabot/cargo/tonic-build-0.14.2

main

dependabot/npm_and_yarn/docs/mdast-util-to-hast-13.2.1

815-feat-support-win32-for-agentpoller

gh-pages

v1.2.5

v1.2.4

v1.2.3

v1.2.2

v1.2.1

v1.2.0

v1.1.2

v1.1.0

v1.0.92

v1.0.91

v1.0.90

v1.0.89

v1.0.88

v1.0.87

v1.0.86

v1.0.85

v1.0.84

v1.0.83

v1.0.82

v1.0.81

v1.0.78

v1.0.77

v1.0.76

v1.0.70

v1.0.69

v1.0.68

v1.0.67

v1.0.66

v1.0.65

v1.0.64

v1.0.63

v1.0.62

v1.0.61

v1.0.60

v1.0.59

v1.0.58

1.0.57

v1.0.56

v1.0.55

v1.0.54-pre5

v1.0.53

v1.0.53-pre19

v1.0.53-pre18

v1.0.53-pre17

v1.0.53-pre15

1.0.53-pre10

1.0.53-pre9

1.0.53-pre8

1.0.53-pre7

1.0.53-pre6

1.0.53-pre5

1.0.53-pre4

1.0.53-pre3

1.0.53-pre2

1.0.53-pre1

1.0.52

1.0.51

1.0.50

1.0.49

1.0.49-pre5

1.0.49-pre4

1.0.49-pre3

1.0.49-pre2

1.0.48

1.0.48-rc2

1.0.48-rc1

1.0.48-pre8

1.0.48-pre7

1.0.48-pre6

1.0.48-pre5

1.0.48-pre4

1.0.48-pre3

1.0.48-pre2

1.0.48-pre1

1.0.47

1.0.47-pre8

1.0.47-pre7

1.0.47-pre6

1.0.47-pre5

1.0.47-pre4

1.0.47-pre3

1.0.47-pre2

1.0.47-pre1

1.0.46

1.0.46-pre9

1.0.46-pre8

1.0.46-pre7

1.0.46-pre6

1.0.46-pre5

1.0.46-pre4

1.0.46-pre3

1.0.46-pre2

1.0.46-pre1

1.0.45

1.0.44

1.0.44-pre12

1.0.44-pre11

1.0.44-pre10

1.0.44-pre9

1.0.44-pre8

1.0.44-pre7

1.0.44-pre6

1.0.44-pre5

1.0.44-pre4

1.0.44-pre3

1.0.44-pre2

1.0.44-pre1

1.0.43

1.0.42

1.0.41

1.0.41-pre1

1.0.40

1.0.40-pre11

1.0.40-pre10

1.0.40-pre9

1.0.40-pre8

1.0.40-pre7

1.0.40-pre6

1.0.40-pre5

1.0.40-pre4

1.0.40-pre3

1.0.40-pre2

1.0.40-pre1

1.0.39

1.0.38

1.0.37

1.0.36

1.0.36-pre5

1.0.36-pre4

1.0.36-pre3

1.0.36-pre2

1.0.35

1.0.35-pre3

1.0.35-pre2

1.0.35-pre

1.0.34-pre3

1.0.34-pre2

1.0.34-pre1

1.0.33

1.0.33-pre2

1.0.33-pre

1.0.32

1.0.31

1.0.30

1.0.29

1.0.28

1.0.27

1.0.26

1.0.25

1.0.24

1.0.23

1.0.22

1.0.21

1.0.20

1.0.19

1.0.18

1.0.17

1.0.16

1.0.15

1.0.14

1.0.13

1.0.11

1.0.10

1.0.9

1.0.8

1.0.7

1.0.6

1.0.5

1.0.4

1.0.3

1.0.2

1.0.1

1.0.0

Labels

Clear labels

1week

2weeks

Failed compliance check

IP cameras

NATS

NATS JetStream

Possible security concern

Something isn't working

build

checkers

ci-cd

continuous integration-continuous deployments

cleanup

cnpg

cloud-native postgres

Pull requests that update a dependency file

device-management

documentation

Improvements or additions to documentation

duplicate

This issue or pull request already exists

dusk

ebpf

enhancement

New feature or request

Pull requests that update GitHub Actions code

Pull requests that update Go code

good first issue

Good for newcomers

help wanted

Extra attention is needed

invalid

This doesn't seem right

javascript

Pull requests that update Javascript code

Oracle Linux related issues

otel

opentelemetry logs, traces, metrics

plug-in

proton

timeplus proton streaming database

python

question

Further information is requested

Pull requests that update rust code

sdk

security

serviceradar-agent

serviceradar-agent-gateway

This will not be worked on

zen-engine

No labels

1week

2weeks

Failed compliance check

IP cameras

NATS

Possible security concern

serviceradar-agent-gateway

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

carverauto/serviceradar#273

Reference in a new issue

Repository

carverauto/serviceradar

Title

Body

No description provided.

Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?

Rows
Columns