carverauto/serviceradar

Fork

You've already forked serviceradar

Code Issues 190 Pull requests 16 Projects Releases 173 Packages Wiki Activity Actions

ArangoDB - MVP #284

New issue

Closed

opened 2026-03-28 04:23:05 +00:00 by mfreeman451 · 1 comment

mfreeman451 commented

2026-03-28 04:23:05 +00:00

Owner

Copy link

Imported from GitHub.

Original GitHub issue: #787
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/787
Original created: 2025-05-16T02:39:12Z

Implementation: ServiceRadar Hybrid Data Organization and Relationship Modeling

Overview

This issue tracks the implementation of the ServiceRadar hybrid data organization and relationship modeling feature, as outlined in the ADR dated 2025-05-15.

Background

ServiceRadar currently lacks sophisticated mechanisms to express rich relationships between devices and enable advanced data correlation. The proposed solution is a hybrid architecture using Timeplus Proton for real-time data ingestion/streaming and ArangoDB for graph-based relationship modeling, following a phased implementation approach over 14 weeks.

Implementation Schedule

Phase 1: Data Foundation (4 weeks)

#789
Modify SNMP checkers/pollers to populate these new streams
Implement metadata enrichment (target_device_ip, ifIndex) in timeseries_metrics
#849
Create tests for data flow into Proton streams

Phase 2: ArangoDB Integration (3 weeks)

Set up ArangoDB instance(s) with security configurations
Define schemas for all vertex and edge collections:
- Vertices: Devices, Interfaces, Collections
- Edges: DeviceHasInterface, DeviceToDevice, DeviceToCollection, InterfaceToCollection
Develop serviceradar-arangodb-sync service with synchronization for:
- devices (Proton) → Devices (ArangoDB)
- discovered_interfaces (Proton) → Interfaces & DeviceHasInterface (ArangoDB)
- topology_discovery_events (Proton) → DeviceToDevice (ArangoDB)
Test synchronization between Proton and ArangoDB

Phase 3: Correlation API and SRQL v1 (3 weeks)

Develop API endpoints for device details with interfaces from ArangoDB
Implement initial SRQL support for querying devices and interfaces
Build NetFlow-SNMP correlation logic
Create new API endpoints for correlation data
Test correlation queries with sample data

Phase 4: Advanced SRQL and UI (2 weeks)

Extend SRQL for topology queries and advanced correlations
Develop API endpoints for topology visualization
Build UI components with vis.js for topology maps
Implement dashboards for correlated NetFlow-SNMP data
Test end-to-end functionality

Phase 5: Optimization and Hardening (2 weeks)

Performance testing with 10,000+ devices and relationships
Optimize ArangoDB sync service
Optimize ArangoDB queries with appropriate indexes
Optimize Proton streams and query performance
Conduct security review and hardening
Update documentation

Technical Requirements

Proton Stream Schemas

New discovered_interfaces stream to track device interfaces
New topology_discovery_events stream for network topology data
Enhanced metadata for existing streams

ArangoDB Collections

Vertex collections for devices, interfaces, and data collections
Edge collections for modeling relationships between entities
Synchronization service to populate from Proton streams

SRQL Enhancements

Support for contextual, correlation, and topology queries
New JOIN and PATH clauses for advanced correlations

API and UI

New endpoints for topology and correlation data
Topology visualization using vis.js
Dashboards for correlated metrics

Documentation Deliverables

Update docs/docs/architecture.md with hybrid model
Add docs/docs/topology.md for topology-aware monitoring
Add docs/docs/correlation.md for correlation examples
Update docs/docs/srql.md with new syntax
Create deployment guides for ArangoDB and sync service

Resources

ArangoDB documentation: [link]
Timeplus Proton documentation: [link]
ServiceRadar architecture diagrams: [link]

Assignees

@mfreeman (lead)
[Add additional team members]

Labels

feature, enhancement, serviceradar, topology, correlation, graphdb, arangodb, proton

Estimated Completion

Target completion date: [Current date + 14 weeks]

Imported from GitHub. Original GitHub issue: #787 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/787 Original created: 2025-05-16T02:39:12Z --- # Implementation: ServiceRadar Hybrid Data Organization and Relationship Modeling ## Overview This issue tracks the implementation of the ServiceRadar hybrid data organization and relationship modeling feature, as outlined in the ADR dated 2025-05-15. ## Background ServiceRadar currently lacks sophisticated mechanisms to express rich relationships between devices and enable advanced data correlation. The proposed solution is a hybrid architecture using Timeplus Proton for real-time data ingestion/streaming and ArangoDB for graph-based relationship modeling, following a phased implementation approach over 14 weeks. ## Implementation Schedule ### Phase 1: Data Foundation (4 weeks) - [x] #789 - [x] Modify SNMP checkers/pollers to populate these new streams - [x] Implement metadata enrichment (`target_device_ip`, `ifIndex`) in `timeseries_metrics` - [x] #849 - [ ] Create tests for data flow into Proton streams ### Phase 2: ArangoDB Integration (3 weeks) - [ ] Set up ArangoDB instance(s) with security configurations - [ ] Define schemas for all vertex and edge collections: - [ ] Vertices: `Devices`, `Interfaces`, `Collections` - [ ] Edges: `DeviceHasInterface`, `DeviceToDevice`, `DeviceToCollection`, `InterfaceToCollection` - [ ] Develop `serviceradar-arangodb-sync` service with synchronization for: - [ ] `devices` (Proton) → `Devices` (ArangoDB) - [ ] `discovered_interfaces` (Proton) → `Interfaces` & `DeviceHasInterface` (ArangoDB) - [ ] `topology_discovery_events` (Proton) → `DeviceToDevice` (ArangoDB) - [ ] Test synchronization between Proton and ArangoDB ### Phase 3: Correlation API and SRQL v1 (3 weeks) - [ ] Develop API endpoints for device details with interfaces from ArangoDB - [ ] Implement initial SRQL support for querying devices and interfaces - [ ] Build NetFlow-SNMP correlation logic - [ ] Create new API endpoints for correlation data - [ ] Test correlation queries with sample data ### Phase 4: Advanced SRQL and UI (2 weeks) - [ ] Extend SRQL for topology queries and advanced correlations - [ ] Develop API endpoints for topology visualization - [ ] Build UI components with vis.js for topology maps - [ ] Implement dashboards for correlated NetFlow-SNMP data - [ ] Test end-to-end functionality ### Phase 5: Optimization and Hardening (2 weeks) - [ ] Performance testing with 10,000+ devices and relationships - [ ] Optimize ArangoDB sync service - [ ] Optimize ArangoDB queries with appropriate indexes - [ ] Optimize Proton streams and query performance - [ ] Conduct security review and hardening - [ ] Update documentation ## Technical Requirements ### Proton Stream Schemas - New `discovered_interfaces` stream to track device interfaces - New `topology_discovery_events` stream for network topology data - Enhanced metadata for existing streams ### ArangoDB Collections - Vertex collections for devices, interfaces, and data collections - Edge collections for modeling relationships between entities - Synchronization service to populate from Proton streams ### SRQL Enhancements - Support for contextual, correlation, and topology queries - New JOIN and PATH clauses for advanced correlations ### API and UI - New endpoints for topology and correlation data - Topology visualization using vis.js - Dashboards for correlated metrics ## Documentation Deliverables - [ ] Update `docs/docs/architecture.md` with hybrid model - [ ] Add `docs/docs/topology.md` for topology-aware monitoring - [ ] Add `docs/docs/correlation.md` for correlation examples - [ ] Update `docs/docs/srql.md` with new syntax - [ ] Create deployment guides for ArangoDB and sync service ## Related Documents - PRD-02: Topology-Aware Monitoring - PRD-03: Real-Time Monitoring with Proton - PRD-04: Device Management System - PRD-05: Agent Device Caching & Reporting - PRD-01: NetFlow Integration ## Resources - ArangoDB documentation: [link] - Timeplus Proton documentation: [link] - ServiceRadar architecture diagrams: [link] ## Assignees @mfreeman (lead) [Add additional team members] ## Labels `feature`, `enhancement`, `serviceradar`, `topology`, `correlation`, `graphdb`, `arangodb`, `proton` ## Estimated Completion Target completion date: [Current date + 14 weeks]

mfreeman451 added the

enhancement

label

2026-03-28 04:23:05 +00:00

mfreeman451 commented

2026-03-28 04:23:05 +00:00

Author

Owner

Copy link

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/787#issuecomment-3056142338
Original created: 2025-07-10T07:52:36Z

closing, not planned anymore, arangodb is under BSL

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/787#issuecomment-3056142338 Original created: 2025-07-10T07:52:36Z --- closing, not planned anymore, arangodb is under BSL

mfreeman451 closed this issue

2026-03-28 04:23:05 +00:00

No Branch/Tag specified

staging

2406-feat-agent-fleet-management-secure-self-update-system

bug/k8s-helm-deployments

chore/k8s-arc-update

rust-fix

2371-analytics-stats-cards-should-abbreviate-numbers

chore/perl-cleanup

2942-featweb-ng-add-logs-tab-to-device-details-page

192-feat-tftp-server

mikemiles-dev/feature/netflow_collection

testing

dependabot/cargo/hostname-0.4.1

dependabot/cargo/redis-1.0.1

dependabot/cargo/bb8-0.9.0

dependabot/cargo/rcgen-0.14.6

dependabot/cargo/hyper-1.8.1

dependabot/cargo/hyper-util-0.1.19

dependabot/cargo/clap-4.5.51

dependabot/cargo/thiserror-2.0.17

dependabot/cargo/time-tz-2.0.0

dependabot/cargo/tonic-build-0.14.2

main

dependabot/npm_and_yarn/docs/mdast-util-to-hast-13.2.1

815-feat-support-win32-for-agentpoller

gh-pages

v1.2.5

v1.2.4

v1.2.3

v1.2.2

v1.2.1

v1.2.0

v1.1.2

v1.1.0

v1.0.92

v1.0.91

v1.0.90

v1.0.89

v1.0.88

v1.0.87

v1.0.86

v1.0.85

v1.0.84

v1.0.83

v1.0.82

v1.0.81

v1.0.78

v1.0.77

v1.0.76

v1.0.70

v1.0.69

v1.0.68

v1.0.67

v1.0.66

v1.0.65

v1.0.64

v1.0.63

v1.0.62

v1.0.61

v1.0.60

v1.0.59

v1.0.58

1.0.57

v1.0.56

v1.0.55

v1.0.54-pre5

v1.0.53

v1.0.53-pre19

v1.0.53-pre18

v1.0.53-pre17

v1.0.53-pre15

1.0.53-pre10

1.0.53-pre9

1.0.53-pre8

1.0.53-pre7

1.0.53-pre6

1.0.53-pre5

1.0.53-pre4

1.0.53-pre3

1.0.53-pre2

1.0.53-pre1

1.0.52

1.0.51

1.0.50

1.0.49

1.0.49-pre5

1.0.49-pre4

1.0.49-pre3

1.0.49-pre2

1.0.48

1.0.48-rc2

1.0.48-rc1

1.0.48-pre8

1.0.48-pre7

1.0.48-pre6

1.0.48-pre5

1.0.48-pre4

1.0.48-pre3

1.0.48-pre2

1.0.48-pre1

1.0.47

1.0.47-pre8

1.0.47-pre7

1.0.47-pre6

1.0.47-pre5

1.0.47-pre4

1.0.47-pre3

1.0.47-pre2

1.0.47-pre1

1.0.46

1.0.46-pre9

1.0.46-pre8

1.0.46-pre7

1.0.46-pre6

1.0.46-pre5

1.0.46-pre4

1.0.46-pre3

1.0.46-pre2

1.0.46-pre1

1.0.45

1.0.44

1.0.44-pre12

1.0.44-pre11

1.0.44-pre10

1.0.44-pre9

1.0.44-pre8

1.0.44-pre7

1.0.44-pre6

1.0.44-pre5

1.0.44-pre4

1.0.44-pre3

1.0.44-pre2

1.0.44-pre1

1.0.43

1.0.42

1.0.41

1.0.41-pre1

1.0.40

1.0.40-pre11

1.0.40-pre10

1.0.40-pre9

1.0.40-pre8

1.0.40-pre7

1.0.40-pre6

1.0.40-pre5

1.0.40-pre4

1.0.40-pre3

1.0.40-pre2

1.0.40-pre1

1.0.39

1.0.38

1.0.37

1.0.36

1.0.36-pre5

1.0.36-pre4

1.0.36-pre3

1.0.36-pre2

1.0.35

1.0.35-pre3

1.0.35-pre2

1.0.35-pre

1.0.34-pre3

1.0.34-pre2

1.0.34-pre1

1.0.33

1.0.33-pre2

1.0.33-pre

1.0.32

1.0.31

1.0.30

1.0.29

1.0.28

1.0.27

1.0.26

1.0.25

1.0.24

1.0.23

1.0.22

1.0.21

1.0.20

1.0.19

1.0.18

1.0.17

1.0.16

1.0.15

1.0.14

1.0.13

1.0.11

1.0.10

1.0.9

1.0.8

1.0.7

1.0.6

1.0.5

1.0.4

1.0.3

1.0.2

1.0.1

1.0.0

Labels

Clear labels

1week

2weeks

Failed compliance check

IP cameras

NATS

NATS JetStream

Possible security concern

Something isn't working

build

checkers

ci-cd

continuous integration-continuous deployments

cleanup

cnpg

cloud-native postgres

Pull requests that update a dependency file

device-management

documentation

Improvements or additions to documentation

duplicate

This issue or pull request already exists

dusk

ebpf

enhancement

New feature or request

Pull requests that update GitHub Actions code

Pull requests that update Go code

good first issue

Good for newcomers

help wanted

Extra attention is needed

invalid

This doesn't seem right

javascript

Pull requests that update Javascript code

Oracle Linux related issues

otel

opentelemetry logs, traces, metrics

plug-in

proton

timeplus proton streaming database

python

question

Further information is requested

Pull requests that update rust code

sdk

security

serviceradar-agent

serviceradar-agent-gateway

This will not be worked on

zen-engine

No labels

1week

2weeks

Failed compliance check

IP cameras

NATS

Possible security concern

serviceradar-agent-gateway

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

carverauto/serviceradar#284

Reference in a new issue

Repository

carverauto/serviceradar

Title

Body

No description provided.

Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?

Rows
Columns