feat: netflow collector (rust) and integration #699

New issue

Closed

opened 2026-03-28 04:27:34 +00:00 by mfreeman451 · 1 comment

mfreeman451 commented

2026-03-28 04:27:34 +00:00

Owner

Imported from GitHub.

Original GitHub issue: #2181
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/2181
Original created: 2025-12-18T05:40:18Z

Is your feature request related to a problem?

We need to integrate a netflow collector into ServiceRadar that can sit on the edge or in-cluster, receive netflow flow data, and write it to a message broker. The interface to the message broker should be abstracted so we could support additional / different message brokers in the future, right now we are currently targeting NATS JetStream but will be considering something like iggy.rs or some kind of hybrid architecture in the future. Once data is written to the message broker, we have several options of what to do next. Using the stateless rule-based zen-engine (serviceradar-zen), we can do really fast ETL to get it into the correct shape for an OCSF-based schema, write it to a different message subject, and then the db-event-writer consumer would process it off the queue and write it to the DB.

Describe the solution you'd like

I think it makes more sense to try and do the ETL in our pipeline and not really in the rust netflow collector itself so that it stays generic or in whatever existing format it's already in and we can keep using the upstream version.

https://schema.ocsf.io/1.7.0/classes/network_activity

rust-based netflow daemon that writes to a message broker
zen rule for processing into OCSF format if possible/necessary
db-event-writer updated
UI dashboards

Future work would be around analysis/data processing and could involve @marvin-hansen and his causal computation library (https://github.com/deepcausality-rs)

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Imported from GitHub. Original GitHub issue: #2181 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/2181 Original created: 2025-12-18T05:40:18Z --- **Is your feature request related to a problem?** We need to integrate a netflow collector into ServiceRadar that can sit on the edge or in-cluster, receive netflow flow data, and write it to a message broker. The interface to the message broker should be abstracted so we could support additional / different message brokers in the future, right now we are currently targeting NATS JetStream but will be considering something like iggy.rs or some kind of hybrid architecture in the future. Once data is written to the message broker, we have several options of what to do next. Using the stateless rule-based zen-engine (serviceradar-zen), we can do really fast ETL to get it into the correct shape for an OCSF-based schema, write it to a different message subject, and then the `db-event-writer` consumer would process it off the queue and write it to the DB. **Describe the solution you'd like** I think it makes more sense to try and do the ETL in our pipeline and not really in the rust netflow collector itself so that it stays generic or in whatever existing format it's already in and we can keep using the upstream version. https://schema.ocsf.io/1.7.0/classes/network_activity - [x] rust-based netflow daemon that writes to a message broker - [ ] zen rule for processing into OCSF format if possible/necessary - [x] db-event-writer updated - [x] UI dashboards Future work would be around analysis/data processing and could involve @marvin-hansen and his causal computation library (https://github.com/deepcausality-rs) **Describe alternatives you've considered** A clear and concise description of any alternative solutions or features you've considered. **Additional context** Related to https://github.com/carverauto/serviceradar/issues/611

mfreeman451 commented

2026-03-28 04:27:35 +00:00

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/2181#issuecomment-3978681918
Original created: 2026-03-01T00:14:06Z

closing, completed

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/2181#issuecomment-3978681918 Original created: 2026-03-01T00:14:06Z --- closing, completed