feat(observability): correlate netprobe attribution with NetFlow into attributed flows (#3425) #3516

Merged

mfreeman451 merged 1 commit from feat/attributed-flow-correlation into staging

2026-06-02 07:49:03 +00:00

mfreeman451 commented

2026-06-02 07:40:31 +00:00

Owner

Summary

Part of #3425. Makes the demo's /observability/flows/attributed page show real attributed flows from the live system, with an architecture that scales to fleets — NetFlow stays the flow source, netprobe only supplies the process, and the correlation is a set-based DB join in core-elx (no agent-as-flow-collector, no host-slice routing, no per-host config).

The model

NetFlow/sFlow keeps flowing from routers → flow-collector → ocsf_network_activity (unchanged — that's the 9.5k flow rows/24h already there).
netprobe agents push process attribution (5-tuple → pid/comm/uid/container) up the existing agent-gateway → core-elx pipeline. netprobe already emits FlowAttributionEvent; no netprobe/agent/flow-collector changes.
core-elx persists each pushed attribution to CNPG and a worker correlates it against the NetFlow already in the DB (direction-agnostic 5-tuple + partition + 15-min window), stamping matching flows event_type=attributed_flow with the process context.
web-ng reads those attributed_flow rows (was hardcoded fixtures).

Changes

migration platform.flow_process_attributions — 5-tuple + process + partition + observed_at, indexed.
ServiceRadar.FlowAttribution — persist/3 (StatusHandler writes pushed attributions), correlate/0 (the join/UPDATE), prune/0 (60-min retention).
ServiceRadar.FlowAttribution.Correlator — GenServer, correlate+prune every 30s, supervised under the EventWriter supervisor.
status_handler.ex — persist pushed attributions (additive; the existing real-time joiner path is untouched).
web-ng attributed_live.ex — real Repo.query against ocsf_network_activity.

Why not the alternatives

Static host_slices config (flow-collector per-agent IP map): doesn't scale to 100k hosts — rejected.
Agent emits complete flows (netprobe as flow collector): out of scope; we want network-wide NetFlow correlated with on-host process, not per-agent flow accounting.

Deploy / validation

Needs a core-elx + web-ng image build and the migration to run (core-migrations-job). After deploy, a netprobe-equipped agent (e.g. agent-sr-test-pve04) capturing on its interface will push attributions that correlate against its NetFlow (its host IP already appears in 142 flows/24h) → rows appear on the attributed-flows page. ⚠️ Elixir not compiled locally in this environment — CI build is the first compile/credo gate.

🤖 Generated with Claude Code

## Summary Part of #3425. Makes the demo's `/observability/flows/attributed` page show **real** attributed flows from the live system, with an architecture that scales to fleets — **NetFlow stays the flow source, netprobe only supplies the process, and the correlation is a set-based DB join in core-elx** (no agent-as-flow-collector, no host-slice routing, no per-host config). ### The model 1. **NetFlow/sFlow** keeps flowing from routers → flow-collector → `ocsf_network_activity` (unchanged — that's the 9.5k flow rows/24h already there). 2. **netprobe agents push process attribution** (5-tuple → pid/comm/uid/container) up the **existing** agent-gateway → core-elx pipeline. netprobe already emits `FlowAttributionEvent`; **no netprobe/agent/flow-collector changes.** 3. **core-elx persists** each pushed attribution to CNPG and a **worker correlates** it against the NetFlow already in the DB (direction-agnostic 5-tuple + partition + 15-min window), stamping matching flows `event_type=attributed_flow` with the process context. 4. **web-ng** reads those `attributed_flow` rows (was hardcoded fixtures). ### Changes - `migration` `platform.flow_process_attributions` — 5-tuple + process + partition + observed_at, indexed. - `ServiceRadar.FlowAttribution` — `persist/3` (StatusHandler writes pushed attributions), `correlate/0` (the join/UPDATE), `prune/0` (60-min retention). - `ServiceRadar.FlowAttribution.Correlator` — GenServer, correlate+prune every 30s, supervised under the EventWriter supervisor. - `status_handler.ex` — persist pushed attributions (additive; the existing real-time joiner path is untouched). - web-ng `attributed_live.ex` — real `Repo.query` against `ocsf_network_activity`. ### Why not the alternatives - *Static `host_slices` config* (flow-collector per-agent IP map): doesn't scale to 100k hosts — rejected. - *Agent emits complete flows* (netprobe as flow collector): out of scope; we want network-wide NetFlow correlated with on-host process, not per-agent flow accounting. ### Deploy / validation Needs a **core-elx + web-ng image build** and the **migration to run** (core-migrations-job). After deploy, a netprobe-equipped agent (e.g. `agent-sr-test-pve04`) capturing on its interface will push attributions that correlate against its NetFlow (its host IP already appears in 142 flows/24h) → rows appear on the attributed-flows page. ⚠️ Elixir not compiled locally in this environment — CI build is the first compile/credo gate. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Rows
Columns