feat: netflow enhancements #960

New issue

Closed

opened 2026-03-28 04:30:17 +00:00 by mfreeman451 · 0 comments

mfreeman451 commented

2026-03-28 04:30:17 +00:00

Owner

Imported from GitHub.

Original GitHub issue: #2681
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/2681
Original created: 2026-02-03T05:21:23Z

This PRD outlines the evolution of the ServiceRadar NetFlow Observability module from a basic flow logger to a high-context network intelligence dashboard.

PRD: ServiceRadar NetFlow Observability Enhancements

Status: Draft / Discovery
Author: Product Engineering
Version: 1.0

1. Executive Summary

The goal is to transform the existing NetFlow parser UI from a raw data table into an actionable diagnostic tool. By adding Data Enrichment, Hierarchical Visualizations, and Security Intelligence, we will enable users to reduce Mean Time to Resolution (MTTR) for network congestion and security incidents.

2. Target Audience

Network Engineers: Troubleshooting congestion and peering issues.
Security Analysts: Identifying data exfiltration or unauthorized access.
DevOps/SRE: Mapping application dependencies and cloud egress costs.

3. Feature Requirements

Phase 1: Data Enrichment (Contextualization)

Raw IPs are difficult to interpret. We must bridge the gap between "Network Address" and "Business Entity."

Feature	Description	Priority
Geo-IP Mapping	Resolve Source/Dest IPs to Country/City. Display flags in the flow table.	P0
Reverse DNS (rDNS)	Automatically resolve IPs to hostnames (e.g., `34.98.106.0` → `google-lb.com`) within the table view.	P0
Service Tagging	Map Common Ports to Service Names (e.g., 443 → `HTTPS`, 53 → `DNS`, 22 → `SSH`).	P1
ASN/ISP Lookup	Identify the owner of the IP block (e.g., `Amazon.com`, `Comcast`, `DigitalOcean`).	P1
Directionality Tagging	Label flows as `Inbound`, `Outbound`, or `Internal (East-West)` based on defined local subnets.	P1

Phase 2: Advanced Visualizations (Pattern Recognition)

Users need to see the "big picture" before diving into the "raw logs."

Feature	Description	Priority
Sankey Diagram	A flow-based visual showing the relationship between `Source Subnet` → `Protocol/Port` → `Destination`.	P1
Top Talkers (Bar/Pie)	Dynamic widgets showing Top 10 IPs by Bytes, Top 10 Ports, and Top 10 ASNs.	P0
Stacked Area Chart	A time-series chart showing traffic volume (bps/pps) color-coded by protocol or application.	P0
Global Heatmap	A world map visual showing traffic density by geographic region.	P2

Phase 3: Interactive UX & Filtering

The dashboard must act as a cohesive workspace where visuals and data are linked.

Feature	Description	Priority
Global Drill-Down	Clicking a segment in a chart (e.g., a "TCP" slice) automatically applies a filter to the raw table.	P0
CIDR Aggregation	Toggle to group individual flows into `/24` or `/16` subnets to reduce table noise.	P1
Relative Time Comparisons	"Compare to Yesterday" overlay on traffic charts to identify unusual spikes.	P2
Unit Auto-Scaling	Automatically convert raw bytes to `KB`, `MB`, `GB`, or `TB` based on volume.	P0

4. Security & Intelligence (Value-Add)

Moving from "What happened?" to "Is this dangerous?"

Threat Intel Integration: Cross-reference flow data with open-source threat feeds (e.g., AlienVault, CrowdSec). Flag known malicious IPs with a "Warning" icon in the table.
Anomaly Detection: Establish a 7-day baseline for specific ports. Alert the user if a port (like SSH/22) sees a 300% increase in traffic volume over the baseline.
Port Scan Detection: Identify and flag sources that are hitting a high number of unique destination ports in a short window.

5. Technical Considerations & Performance

High Cardinality: NetFlow generates massive datasets. The UI must utilize Pagination and Server-Side Filtering to remain responsive.
Caching: Geo-IP and rDNS lookups should be cached locally to prevent API rate-limiting and reduce latency in the table view.
Data Retention: Define a TTL (Time-To-Live) for raw flow data (e.g., 7 days raw, 30 days aggregated) to manage storage costs.

6. Success Metrics

Engagement: Increase in time spent on the "Observability" tab vs. the "Devices" tab.
Diagnostic Speed: Reduction in the number of manual CLI queries performed by users (tracked via user feedback).
Discoverability: Percentage of users who utilize the "Drill-down" filtering functionality.

7. UI/UX Mockup Notes (Iteration on Current Design)

Consolidate Cards: Merge "NetFlow v5", "NetFlow v9", and "IPFIX" into a single "Protocol Distribution" donut chart to save vertical space.
Compact Table: Offer a "Zebra-stripe" compact mode for the flow table to allow more rows to be visible on one screen.
Action Menu: Clicking a row in the table should open a side-panel showing full enrichment details (Map location, full WHOIS data, and related flows).

Imported from GitHub. Original GitHub issue: #2681 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/2681 Original created: 2026-02-03T05:21:23Z --- This PRD outlines the evolution of the **ServiceRadar NetFlow Observability** module from a basic flow logger to a high-context network intelligence dashboard. --- # PRD: ServiceRadar NetFlow Observability Enhancements **Status:** Draft / Discovery **Author:** Product Engineering **Version:** 1.0 ## 1. Executive Summary The goal is to transform the existing NetFlow parser UI from a raw data table into an actionable diagnostic tool. By adding **Data Enrichment**, **Hierarchical Visualizations**, and **Security Intelligence**, we will enable users to reduce Mean Time to Resolution (MTTR) for network congestion and security incidents. ## 2. Target Audience * **Network Engineers:** Troubleshooting congestion and peering issues. * **Security Analysts:** Identifying data exfiltration or unauthorized access. * **DevOps/SRE:** Mapping application dependencies and cloud egress costs. --- ## 3. Feature Requirements ### Phase 1: Data Enrichment (Contextualization) *Raw IPs are difficult to interpret. We must bridge the gap between "Network Address" and "Business Entity."* | Feature | Description | Priority | | :--- | :--- | :--- | | **Geo-IP Mapping** | Resolve Source/Dest IPs to Country/City. Display flags in the flow table. | P0 | | **Reverse DNS (rDNS)** | Automatically resolve IPs to hostnames (e.g., `34.98.106.0` → `google-lb.com`) within the table view. | P0 | | **Service Tagging** | Map Common Ports to Service Names (e.g., 443 → `HTTPS`, 53 → `DNS`, 22 → `SSH`). | P1 | | **ASN/ISP Lookup** | Identify the owner of the IP block (e.g., `Amazon.com`, `Comcast`, `DigitalOcean`). | P1 | | **Directionality Tagging** | Label flows as `Inbound`, `Outbound`, or `Internal (East-West)` based on defined local subnets. | P1 | ### Phase 2: Advanced Visualizations (Pattern Recognition) *Users need to see the "big picture" before diving into the "raw logs."* | Feature | Description | Priority | | :--- | :--- | :--- | | **Sankey Diagram** | A flow-based visual showing the relationship between `Source Subnet` → `Protocol/Port` → `Destination`. | P1 | | **Top Talkers (Bar/Pie)** | Dynamic widgets showing Top 10 IPs by Bytes, Top 10 Ports, and Top 10 ASNs. | P0 | | **Stacked Area Chart** | A time-series chart showing traffic volume (bps/pps) color-coded by protocol or application. | P0 | | **Global Heatmap** | A world map visual showing traffic density by geographic region. | P2 | ### Phase 3: Interactive UX & Filtering *The dashboard must act as a cohesive workspace where visuals and data are linked.* | Feature | Description | Priority | | :--- | :--- | :--- | | **Global Drill-Down** | Clicking a segment in a chart (e.g., a "TCP" slice) automatically applies a filter to the raw table. | P0 | | **CIDR Aggregation** | Toggle to group individual flows into `/24` or `/16` subnets to reduce table noise. | P1 | | **Relative Time Comparisons** | "Compare to Yesterday" overlay on traffic charts to identify unusual spikes. | P2 | | **Unit Auto-Scaling** | Automatically convert raw bytes to `KB`, `MB`, `GB`, or `TB` based on volume. | P0 | --- ## 4. Security & Intelligence (Value-Add) *Moving from "What happened?" to "Is this dangerous?"* 1. **Threat Intel Integration:** Cross-reference flow data with open-source threat feeds (e.g., AlienVault, CrowdSec). Flag known malicious IPs with a "Warning" icon in the table. 2. **Anomaly Detection:** Establish a 7-day baseline for specific ports. Alert the user if a port (like SSH/22) sees a 300% increase in traffic volume over the baseline. 3. **Port Scan Detection:** Identify and flag sources that are hitting a high number of unique destination ports in a short window. --- ## 5. Technical Considerations & Performance * **High Cardinality:** NetFlow generates massive datasets. The UI must utilize **Pagination** and **Server-Side Filtering** to remain responsive. * **Caching:** Geo-IP and rDNS lookups should be cached locally to prevent API rate-limiting and reduce latency in the table view. * **Data Retention:** Define a TTL (Time-To-Live) for raw flow data (e.g., 7 days raw, 30 days aggregated) to manage storage costs. --- ## 6. Success Metrics * **Engagement:** Increase in time spent on the "Observability" tab vs. the "Devices" tab. * **Diagnostic Speed:** Reduction in the number of manual CLI queries performed by users (tracked via user feedback). * **Discoverability:** Percentage of users who utilize the "Drill-down" filtering functionality. --- ## 7. UI/UX Mockup Notes (Iteration on Current Design) * **Consolidate Cards:** Merge "NetFlow v5", "NetFlow v9", and "IPFIX" into a single "Protocol Distribution" donut chart to save vertical space. * **Compact Table:** Offer a "Zebra-stripe" compact mode for the flow table to allow more rows to be visible on one screen. * **Action Menu:** Clicking a row in the table should open a side-panel showing full enrichment details (Map location, full WHOIS data, and related flows).