Device Identity Reconciliation Engine (DIRE) #2481
No reviewers
Labels
No labels
1week
2weeks
Failed compliance check
IP cameras
NATS
Possible security concern
Review effort 1/5
Review effort 2/5
Review effort 3/5
Review effort 4/5
Review effort 5/5
UI
aardvark
accessibility
amd64
api
arm64
auth
back-end
bgp
blog
bug
build
checkers
ci-cd
cleanup
cnpg
codex
core
dependencies
device-management
documentation
duplicate
dusk
ebpf
enhancement
eta 1d
eta 1hr
eta 3d
eta 3hr
feature
fieldsurvey
github_actions
go
good first issue
help wanted
invalid
javascript
k8s
log-collector
mapper
mtr
needs-triage
netflow
network-sweep
observability
oracle
otel
plug-in
proton
python
question
reddit
redhat
research
rperf
rperf-checker
rust
sdk
security
serviceradar-agent
serviceradar-agent-gateway
serviceradar-web
serviceradar-web-ng
siem
snmp
sysmon
topology
ubiquiti
wasm
wontfix
zen-engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
carverauto/serviceradar!2481
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "refs/pull/2481/head"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Imported from GitHub pull request.
Original GitHub pull request: #2022
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/pull/2022
Original created: 2025-11-27T04:53:09Z
Original updated: 2025-11-27T05:30:07Z
Original head: carverauto/serviceradar:core/device_count_creeping
Original base: main
Original merged: 2025-11-27T05:30:03Z by @mfreeman451
User description
IMPORTANT: Please sign the Developer Certificate of Origin
Thank you for your contribution to ServiceRadar. Please note, when contributing, the developer must include
a DCO sign-off statement indicating the DCO acceptance in one commit message. Here
is an example DCO Signed-off-by line in a commit message:
Describe your changes
Issue ticket number and link
Code checklist before requesting a review
PR Type
Enhancement, Tests, Bug fix, Documentation
Description
Implements comprehensive Identity Reconciliation Engine (IRE) to address device duplication from IP churn through network sightings lifecycle management
Adds policy-driven promotion pipeline with confidence tiers: Tier 3 sightings → Tier 2 promotions → Tier 1 anchored devices
Introduces database schema for
network_sightings,device_identifiers,fingerprints,subnet_policies, and audit tables with proper indexing and constraintsImplements core registry reconciliation with sighting ingestion, TTL management, and promotion eligibility evaluation based on policy gates
Adds background reapers for expiring stale sightings and cleaning up in-memory registry state
Provides comprehensive REST API endpoints for sighting management, promotion, dismissal, and audit trail queries
Implements OpenTelemetry metrics tracking promotion statistics (promoted, eligible, shadow-ready, blocked counts)
Adds frontend dashboard (
SightingsDashboard) with filtering, pagination, promotion actions, and merge history visualizationIntegrates promotion lineage visibility into device detail view showing sighting origin and audit events
Includes Prometheus alerting rules for monitoring promotion pipeline health and reconciliation staleness
Fixes SQL query formatting issues in hydrate.go with proper
constkeyword positioning and table reference correctionsAdds comprehensive documentation covering requirements, design, metrics, and alerting guidance
Updates Helm configuration with reaper settings, identity reconciliation profiles, and faker simulation parameters
Diagram Walkthrough
File Walkthrough
33 files
registry.go
Identity reconciliation engine with sighting promotion pipelinepkg/registry/registry.go
identityCfgandreconcileIntervalfields toDeviceRegistrystruct for identity reconciliation configuration
WithIdentityReconciliationConfig()andWithReconcileInterval()option functions for registry initializationProcessBatchDeviceUpdates()tofilter and store network sightings when identity reconciliation is
enabled
ReconcileSightings()method to promote eligible sightingsto unified devices based on policy gates
ingestSightings(),resolveSightingTTL(),copySightingMetadata(),buildUpdateFromNetworkSighting(),PromoteSighting(),DismissSighting(),ListSightingEvents(),ListSightings(),CountSightings(), andbuildIdentifiersFromUpdate()cnpg_identity_reconciliation_queries.go
Database queries for sighting and promotion managementpkg/db/cnpg_identity_reconciliation_queries.go
operations
promoted, listing active sightings, and counting sightings
ListPromotableSightings(),MarkSightingsPromoted(),ListActiveSightings(),CountActiveSightings(),GetNetworkSighting(),UpdateSightingStatus(),ListSightingEvents(),ListSubnetPolicies(), andListMergeAuditEvents()availability checks
cnpg_identity_reconciliation_upserts.go
Database upsert operations for identifiers and eventspkg/db/cnpg_identity_reconciliation_upserts.go
reconciliation
UpsertDeviceIdentifiers()to write device identifier rows withconflict resolution
InsertSightingEvents()to record audit events for sightingsproper error handling
cnpg_identity_reconciliation.go
Core sighting storage and expiration operationspkg/db/cnpg_identity_reconciliation.go
StoreNetworkSightings()to upsert active sightings withTTL management
ExpireNetworkSightings()to mark stale sightings asexpired based on TTL
identity.go
API endpoints for sighting management and reconciliationpkg/core/api/identity.go
endpoints
handleListSightings(),handleReconcileSightings(),handlePromoteSighting(),handleDismissSighting(),handleSightingEvents(),handleListSubnetPolicies(), andhandleMergeAuditHistory()and proper error responses
identity_metrics.go
OpenTelemetry metrics for promotion reconciliation trackingpkg/registry/identity_metrics.go
reconciliation
eligible, shadow-ready, blocked, etc.)
recordIdentityPromotionMetrics()to update metrics withlatest reconciliation run data
promotion_status.go
Sighting promotion eligibility evaluation logicpkg/registry/promotion_status.go
promotionStatusForSighting()to assess sighting readinessbased on policy gates
dedupeStrings()andformatDurationShort()forstatus formatting
shadow mode status
identity_reconciliation.go
Data models for identity reconciliation domainpkg/models/identity_reconciliation.go
NetworkSighting,DeviceIdentifier,SightingEvent,SubnetPolicy,MergeAuditEvent, andSightingPromotionStatusNetworkSightingStatusenum with states: active, promoted,expired, dismissed
config.go
Configuration models for identity reconciliation and reaperpkg/models/config.go
ReaperConfig,IdentityReaperProfile,IdentityReaperConfig,PromotionConfig, andFingerprintingConfigIdentityReconciliationConfigto gate the identity reconciliationengine
CoreServiceConfigwithReaperandIdentityfieldsreaper configuration
identity_reaper.go
Background reaper for stale network sightingspkg/core/identity_reaper.go
stale sightings
IdentityReaperstruct withStart()andreap()methodsevents
reaper.go
Enhanced stale device reaper with registry cleanuppkg/core/reaper.go
registryfield toStaleDeviceReaperstructNewStaleDeviceReaper()constructor to accept registryparameter
DeleteLocal()on registry when reaping staledevices
server.go
Server initialization for identity reconciliation pipelinepkg/core/server.go
reconciliation config and reconcile interval options
interval and TTL from config
enabled
reconciliation
server.go
API server routes for identity reconciliationpkg/core/api/server.go
WithIdentityConfig()option function to expose identityreconciliation config to API
endpoints
policies, merge audit, and reconciliation trigger
types.go
API server types for identity reconciliation supportpkg/core/api/types.go
identityConfigfield toAPIServerstructDeviceRegistryServiceinterface with 6 new methods forsighting operations and reconciliation
interfaces.go
Database service interface extensions for identity reconciliationpkg/db/interfaces.go
Serviceinterface with 11 new methods for identityreconciliation operations
updates, and audit queries
interfaces.go
Registry manager interface extensions for sighting operationspkg/registry/interfaces.go
DeleteLocal()method toManagerinterface for in-memory deviceremoval
ReconcileSightings(),ListSightings(),CountSightings(),PromoteSighting(),DismissSighting(), andListSightingEvents()unified_device.go
Added sighting discovery source typepkg/models/unified_device.go
DiscoverySourceSightingconstant to discovery source enumerationDeviceUpdatestructbootstrap.go
Bootstrap integration for identity configpkg/core/bootstrap/bootstrap.go
core service config
route.ts
Frontend API route for sighting eventsweb/src/app/api/identity/sightings/[id]/events/route.ts
and error responses
route.ts
Frontend API route for sighting dismissalweb/src/app/api/identity/sightings/[id]/dismiss/route.ts
error responses
route.ts
Add sightings API endpoint with paginationweb/src/app/api/identity/sightings/route.ts
support
partition,limit, andoffsetroute.ts
Add merge audit history API endpointweb/src/app/api/identity/merge-audit/route.ts
device_idand pagination withlimitparameterroute.ts
Add subnet policies API endpointweb/src/app/api/identity/policies/route.ts
limitquery parameteridentity.ts
Add identity reconciliation type definitionsweb/src/types/identity.ts
NetworkSighting,SightingEvent,SubnetPolicy, andMergeAuditEventtypesroute.ts
Add sighting promotion API endpointweb/src/app/api/identity/sightings/[id]/promote/route.ts
route.ts
Add reconciliation trigger API endpointweb/src/app/api/identity/reconcile/route.ts
endpoint
SightingsDashboard.tsx
Add comprehensive sightings management dashboard UIweb/src/components/Identity/SightingsDashboard.tsx
promotion
view panel
historical lookup features
explanatory hints
DeviceDetail.tsx
Add promotion lineage visibility to device detailweb/src/components/Devices/DeviceDetail.tsx
Infoicon andSightingEventtypemetadataFlag,metadataValue)sighting ID
promotion mode, and audit events
00000000000009_identity_reconciliation_schema.up.sql
Add identity reconciliation database schemapkg/db/cnpg/migrations/00000000000009_identity_reconciliation_schema.up.sql
subnet_policies,fingerprints,network_sightings,device_identifierssighting_eventsandmerge_auditwith properindexing
integrity
page.tsx
New Identity & Sightings dashboard page componentweb/src/app/identity/page.tsx
with Apache 2.0 license header
noStore()to disable caching forreal-time sighting data
sighting,history_actor, andhistory_partitionfiltersSightingsDashboardcomponent with Suspense fallback and passesfilter parameters
BUILD.bazel
Add identity reconciliation database layer filespkg/db/BUILD.bazel
cnpg_identity_reconciliation.go,cnpg_identity_reconciliation_queries.go, andcnpg_identity_reconciliation_upserts.goreconciliation operations
Sidebar.tsx
Add Identity navigation to sidebar menuweb/src/components/Sidebar.tsx
Fingerprinticon from lucide-react library/identityroute andFingerprinticonBUILD.bazel
Add identity reaper implementation filepkg/core/BUILD.bazel
identity_reaper.gofile to core package build configurationlow-confidence device cleanup per subnet policies
4 files
mock_db.go
Mock implementations for identity reconciliation database methodspkg/db/mock_db.go
identity reconciliation
and audit history
assertions
promotion_status_test.go
Unit tests for promotion status evaluationpkg/registry/promotion_status_test.go
disabled, shadow mode, and eligible scenarios
containsHint()for assertion validationreaper_test.go
Updated reaper tests for registry cleanuppkg/core/reaper_test.go
NewStaleDeviceReaper()DeleteLocal()calls on registry duringreaping
mock_registry.go
Mock implementations for registry sighting operationspkg/registry/mock_registry.go
event listing
5 files
config.go
Default configuration values for identity reconciliationpkg/core/config.go
configuration
applyIdentityDefaults()function to apply sensibledefaults for identity reconciliation config
normalizeConfig()function
BUILD.bazel
Bazel build configuration updatepkg/core/api/BUILD.bazel
identity.goto the go_library sources listvalues.yaml
Update Helm values with identity and reaper confighelm/serviceradar/values.yaml
latestto specific commit SHA forreproducibility
core.reaperconfiguration for stale device cleanupcore.identityconfiguration block with promotion, fingerprinting,and reaper profiles
faker.simulationconfiguration for IP shuffle behaviorserviceradar-config.yaml
Add identity reconciliation config to Helm templatehelm/serviceradar/files/serviceradar-config.yaml
reaperconfiguration section with interval and TTL settingsidentity_reconciliationconfiguration withpromotion, fingerprinting, and reaper profiles
shuffle behavior
kustomization.yaml
Add identity Prometheus rules to demo kustomizationk8s/demo/base/kustomization.yaml
identity-prometheus-rules.yamlresource filedeployment
1 files
hydrate.go
SQL query formatting and table reference fixespkg/registry/hydrate.go
constkeyword to proper positiontable(device_capability_registry)todevice_capability_registry6 files
spec.md
Add identity reconciliation requirements specificationopenspec/changes/add-identity-reconciliation-engine/specs/device-identity-reconciliation/spec.md
promotion context
identity-alerts.md
Add identity reconciliation alert rules documentationdocs/docs/identity-alerts.md
reconciliation
reconciliation stalls, and volume surges
design.md
Identity Reconciliation Engine design and architectureopenspec/changes/add-identity-reconciliation-engine/design.md
(IRE) to address device duplication from IP churn
network_sightings,device_identifiers,fingerprints,subnet_policies, and audit tablesTier 1 anchored devices) with policy-driven promotion rules
legacy path removal, including shadow-mode validation
tasks.md
Implementation tasks and deployment status for IREopenspec/changes/add-identity-reconciliation-engine/tasks.md
completion status (8 complete, 3 pending)
core engine, reaper, API/UI, and metrics
configuration API exposure
deployment to demo environment
identity-metrics.md
Identity reconciliation metrics and alerting guidedocs/docs/identity-metrics.md
serviceradar.identitymeter for promotion monitoring
staleness, throughput dips, and volume surges
trends, and run age tracking
15-minute staleness, 30-minute shadow backlog)
proposal.md
IRE proposal and impact analysisopenspec/changes/add-identity-reconciliation-engine/proposal.md
device duplication from IP churn
schema additions, API/UI surfaces, and metrics
sweep/poller/sync ingestion, CNPG schema, Helm values, and background
jobs
devices with policy-driven promotion
1 files
identity-prometheus-rules.yaml
Prometheus alerting rules for identity reconciliationk8s/demo/base/identity-prometheus-rules.yaml
health and promotion pipeline
reconciliation, throughput drops, and volume surges
annotations for operator guidance
reconciliation failures early
1 files
BUILD.bazel
Add OpenTelemetry dependencies to registrypkg/registry/BUILD.bazel
@io_opentelemetry_go_oteland@io_opentelemetry_go_otel_metricregistry package
1 files
Imported GitHub PR comment.
Original author: @qodo-code-review[bot]
Original URL: https://github.com/carverauto/serviceradar/pull/2022#issuecomment-3584200448
Original created: 2025-11-27T04:54:15Z
You are nearing your monthly Qodo Merge usage quota. For more information, please visit here.
PR Compliance Guide 🔍
Below is a summary of compliance checks for this PR:
Untrusted input logging
Description: The dismissal endpoint accepts an unvalidated JSON 'reason' field and echoes it back in
audit logs without sanitization, risking log injection or storing untrusted content that
could later be reflected (e.g., in UIs) and enable XSS if not escaped downstream.
identity.go [137-146]
Referred Code
Unvalidated metadata storage
Description: Sightings ingestion copies arbitrary metadata fields directly from updates (including
'hostname', 'mac', and other keys) into database without normalization or validation,
which can enable downstream injection risks (e.g., SQL/JSON query manipulation, log/HTML
injection) if later rendered or used to build queries.
registry.go [187-205]
Referred Code
Audit spoofing risk
Description: The promote sighting endpoint derives the 'actor' from the authenticated user's email and
records it without additional verification or normalization; if upstream identity
assertions are weak or email contains unexpected characters, audit trails could be spoofed
or contain malicious content.
identity.go [96-105]
Referred Code
Race/inconsistent upsert
Description: The upsert on 'network_sightings' uses ON CONFLICT (partition, ip) WHERE status='active',
which may allow bypass by toggling status or race conditions leading to duplicate active
rows if status changes between insert and update; this can be exploited to create
inconsistent state for promotion decisions.
cnpg_identity_reconciliation.go [30-37]
Referred Code
DoS via reconciliation triggers
Description: Background reconciliation is triggered on a fixed ticker without rate limiting per
tenant/partition or backoff; an authenticated user can also trigger '/identity/reconcile'
causing concurrent runs, potentially enabling resource exhaustion or DoS via repeated
promotions.
server.go [412-438]
Referred Code
🎫 No ticket provided
Codebase context is not defined
Follow the guide to enable codebase context checks.
Generic: Comprehensive Audit Trails
Objective: To create a detailed and reliable record of critical system actions for security analysis
and compliance.
Status: Passed
Generic: Meaningful Naming and Self-Documenting Code
Objective: Ensure all identifiers clearly express their purpose and intent, making code
self-documenting
Status: Passed
Generic: Secure Logging Practices
Objective: To ensure logs are useful for debugging and auditing without exposing sensitive
information like PII, PHI, or cardholder data.
Status: Passed
Generic: Robust Error Handling and Edge Case Management
Objective: Ensure comprehensive error handling that provides meaningful context and graceful
degradation
Status:
Error context: API handlers return internal error messages directly to clients without consistent
wrapping or redaction, which may lack robust context handling and uniform validation
across endpoints.
Referred Code
Generic: Secure Error Handling
Objective: To prevent the leakage of sensitive system information through error messages while
providing sufficient detail for internal debugging.
Status:
Error leakage: Handlers write err.Error() to HTTP responses (e.g., database or registry errors), which
could expose internal details to end users.
Referred Code
Generic: Security-First Input Validation and Data Handling
Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent
vulnerabilities
Status:
Input validation: Query and body parameters (e.g., limit, offset, reason) are minimally validated and path
IDs are only checked for emptiness, which may require stronger validation and constraints.
Referred Code
Compliance status legend
🟢 - Fully Compliant🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label
Imported GitHub PR comment.
Original author: @qodo-code-review[bot]
Original URL: https://github.com/carverauto/serviceradar/pull/2022#issuecomment-3584202374
Original created: 2025-11-27T04:55:27Z
You are nearing your monthly Qodo Merge usage quota. For more information, please visit here.
PR Code Suggestions ✨
Explore these optional code suggestions:
Fix incorrect promise handling for route parameters
Fix the
RoutePropsinterface and remove theawaitkeyword when accessingprops.paramsto prevent a runtime error in the Next.js route handler.web/src/app/api/identity/sightings/[id]/events/route.ts [20-25]
Suggestion importance[1-10]: 9
__
Why: The suggestion correctly identifies a critical runtime error in the new API route handler caused by incorrectly awaiting a non-promise
paramsobject, which would make the endpoint fail.Fix incorrect handling of searchParams
Correct the handling of the
searchParamsprop by removing the unnecessaryawaitand updating its type from a
Promiseto a plain object, as it is not a promisein Next.js App Router pages.
web/src/app/identity/page.tsx [26-40]
[To ensure code accuracy, apply this suggestion manually]Suggestion importance[1-10]: 9
__
Why: The suggestion correctly identifies a fundamental misunderstanding of the Next.js App Router's
searchParamsprop, which is an object, not a Promise. Correcting this prevents a runtime error and fixes the component's core logic for handling URL parameters.Fix an undefined function call
Add the missing helper function
toNullableStringinpkg/db/cnpg_identity_reconciliation_upserts.goto fix a compilation error inbuildSightingEventArgs.pkg/db/cnpg_identity_reconciliation_upserts.go [195-202]
Suggestion importance[1-10]: 8
__
Why: The suggestion correctly identifies a call to an undefined function
toNullableString, which would cause a compilation error, and provides the correct implementation to fix it.Add cascade delete to foreign key
Add
ON DELETE CASCADEto thedevice_idforeign key in thedevice_identifierstable to ensure associated identifiers are automatically removed when a device
is deleted.
pkg/db/cnpg/migrations/00000000000009_identity_reconciliation_schema.up.sql [59-70]
Suggestion importance[1-10]: 8
__
Why: This suggestion correctly identifies that the lack of an
ON DELETEclause on thedevice_idforeign key would block device deletions, leading to orphaned data or failed merges. AddingON DELETE CASCADEis a critical improvement for data integrity and the correct functioning of the reconciliation logic.Persist sighting status in shadow mode
In
ReconcileSightings, persist the updated sighting promotion status to thedatabase even when running in shadow mode to prevent displaying stale
information.
pkg/registry/registry.go [559-566]
Suggestion importance[1-10]: 7
__
Why: The suggestion correctly points out that sighting promotion statuses calculated in shadow mode are not persisted, which could be useful for observability. The proposed change is valid and improves the feature's utility.
Remove foreign keys from audit table
Remove the foreign key constraints on
from_device_idandto_device_idin themerge_audittable to decouple the audit log from the device lifecycle andprevent deletion blocks.
pkg/db/cnpg/migrations/00000000000009_identity_reconciliation_schema.up.sql [88-97]
Suggestion importance[1-10]: 7
__
Why: The suggestion correctly identifies that the foreign key constraints on the
merge_audittable would block device deletion, and correctly proposes removing them to ensure the audit trail's independence and prevent future operational failures.Handle JSON parsing errors properly
Instead of silently ignoring JSON parsing errors, return a
400 Bad Requestresponse to the client to improve API robustness and debuggability.
web/src/app/api/identity/sightings/[id]/dismiss/route.ts [34-42]
Suggestion importance[1-10]: 7
__
Why: The suggestion correctly points out that silently ignoring JSON parsing errors is poor practice and proposes a more robust error handling strategy by returning a
400error.