2067 bugcore duplicate key for sync #2513
No reviewers
Labels
No labels
1week
2weeks
Failed compliance check
IP cameras
NATS
Possible security concern
Review effort 1/5
Review effort 2/5
Review effort 3/5
Review effort 4/5
Review effort 5/5
UI
aardvark
accessibility
amd64
api
arm64
auth
back-end
bgp
blog
bug
build
checkers
ci-cd
cleanup
cnpg
codex
core
dependencies
device-management
documentation
duplicate
dusk
ebpf
enhancement
eta 1d
eta 1hr
eta 3d
eta 3hr
feature
fieldsurvey
github_actions
go
good first issue
help wanted
invalid
javascript
k8s
log-collector
mapper
mtr
needs-triage
netflow
network-sweep
observability
oracle
otel
plug-in
proton
python
question
reddit
redhat
research
rperf
rperf-checker
rust
sdk
security
serviceradar-agent
serviceradar-agent-gateway
serviceradar-web
serviceradar-web-ng
siem
snmp
sysmon
topology
ubiquiti
wasm
wontfix
zen-engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
carverauto/serviceradar!2513
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "refs/pull/2513/head"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Imported from GitHub pull request.
Original GitHub pull request: #2068
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/pull/2068
Original created: 2025-12-06T15:44:01Z
Original updated: 2025-12-06T20:32:43Z
Original head: carverauto/serviceradar:2067-bugcore-duplicate-key-for-sync
Original base: main
Original merged: 2025-12-06T20:32:37Z by @mfreeman451
User description
IMPORTANT: Please sign the Developer Certificate of Origin
Thank you for your contribution to ServiceRadar. Please note, when contributing, the developer must include
a DCO sign-off statement indicating the DCO acceptance in one commit message. Here
is an example DCO Signed-off-by line in a commit message:
Describe your changes
Issue ticket number and link
Code checklist before requesting a review
PR Type
Bug fix, Enhancement
Description
Resolve duplicate key constraint violations by enforcing IP uniqueness within batches and against existing database records
Implement strong identity (Armis ID, Netbox ID, MAC) aware IP conflict resolution to prevent incorrect device merges during IP churn
Add IP collision metrics and observability to track and diagnose duplicate key issues
Decouple AGE graph writes from synchronous registry ingest path to prevent backpressure stalls
Add drift tolerance thresholds to stats aggregation for handling faker-scale loads
Diagram Walkthrough
File Walkthrough
1 files
Ensure synchronous graph writes for age-backfill tool5 files
Add drift tolerance and improved discrepancy reportingImplement async/sync graph write modes with configurable behaviorNew helper functions for canonical device detectionAdd IP collision metrics and observabilityNew helper functions to extract strong identity markers2 files
Filter tombstoned devices from canonical IP resolutionImplement IP conflict resolution respecting strong identity6 files
Add test coverage for tombstone filtering in IP resolutionIntegration test for IP churn with different strong identitiesUnit tests for IP churn scenarios with strong identity handlingAdd tests for canonical IP mapping chain resolutionUpdate deduplication tests for strong identity and churn handlingUpdate existing tests for new conflict resolution behavior10 files
Design proposal for strong identity aware IP conflict resolutionRequirements for strong identity precedence in conflict resolutionImplementation tasks for strong identity fixDesign proposal for duplicate key constraint fixRequirements for batch-level IP uniqueness enforcementImplementation tasks for duplicate key constraint fixDesign proposal for AGE graph backpressure decouplingRequirements for inventory consistency under backpressureRequirements for AGE graph write decouplingImplementation tasks for backpressure stabilizationImported GitHub PR comment.
Original author: @qodo-code-review[bot]
Original URL: https://github.com/carverauto/serviceradar/pull/2068#issuecomment-3620607479
Original created: 2025-12-06T15:44:45Z
PR Compliance Guide 🔍
Below is a summary of compliance checks for this PR:
Unsafe IP sentinel
Description: Clearing an existing device's IP by assigning the literal string "0.0.0.0" may be
interpreted as a real IPv4 address rather than an unset value, potentially colliding with
a legitimate device using 0.0.0.0 or violating invariants; consider using NULL/empty and
ensuring downstream code treats it as cleared.
registry.go [434-441]
Referred Code
Destructive test setup
Description: Integration test drops and recreates production-like tables unconditionally; if
misconfigured to point at a non-disposable database, execution could cause destructive
data loss.
integration_churn_test.go [61-76]
Referred Code
🎫 #2067
strong identities to avoid incorrect merges.
resolutions.
sync issues.
during sync by ensuring only one active device per IP is published.
Codebase context is not defined
Follow the guide to enable codebase context checks.
Generic: Secure Error Handling
Objective: To prevent the leakage of sensitive system information through error messages while
providing sufficient detail for internal debugging.
Status: Passed
Generic: Secure Logging Practices
Objective: To ensure logs are useful for debugging and auditing without exposing sensitive
information like PII, PHI, or cardholder data.
Status: Passed
Generic: Comprehensive Audit Trails
Objective: To create a detailed and reliable record of critical system actions for security analysis
and compliance.
Status:
Limited auditing: New critical actions (IP conflict resolution, tombstone creation, IP clearing) add logs
but do not implement structured audit logs with actor/user context, making it unclear if
audit trail requirements are fully met.
Referred Code
Generic: Meaningful Naming and Self-Documenting Code
Objective: Ensure all identifiers clearly express their purpose and intent, making code
self-documenting
Status:
Magic constant IP: The code uses the string literal "0.0.0.0" to clear IPs without a named constant
or documentation, which may reduce clarity and self-documentation.
Referred Code
Generic: Robust Error Handling and Edge Case Management
Objective: Ensure comprehensive error handling that provides meaningful context and graceful
degradation
Status:
Fallback handling: When identity/device fetches fail during conflict resolution the code logs a warning and
proceeds without strong identity checks, which could lead to incorrect merges without
compensating safeguards.
Referred Code
Generic: Security-First Input Validation and Data Handling
Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent
vulnerabilities
Status:
Unsafe SQL drops: The integration test executes raw DROP TABLE and CREATE TABLE statements derived from an
environment-provided connection string, which could be risky if accidentally pointed at a
non-disposable database.
Referred Code
Compliance status legend
🟢 - Fully Compliant🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label
Imported GitHub PR comment.
Original author: @qodo-code-review[bot]
Original URL: https://github.com/carverauto/serviceradar/pull/2068#issuecomment-3620608545
Original created: 2025-12-06T15:46:03Z
PR Code Suggestions ✨
Explore these optional code suggestions:
Fix incorrect map lookup logic
Correct the logic in
resolveIPsToCanonicalto use thedeviceIDfrom theoutmapas the key for looking up the canonical target in
resolvedChains, instead ofincorrectly using the
ip.pkg/registry/registry.go [2042-2056]
[To ensure code accuracy, apply this suggestion manually]Suggestion importance[1-10]: 9
__
Why: This suggestion fixes a critical bug in
resolveIPsToCanonicalwhere an incorrect map key (ipinstead ofdeviceID) is used, which would cause the tombstone chain resolution to fail and return incorrect data.✅
Avoid changing device availability statusSuggestion Impact:
The commit refactored the code and introduced createIPClearUpdate, but it still sets IsAvailable: false. However, the change centralizes this behavior; the relevant code path still controls IsAvailable for churn. The suggestion influenced the touched area, but was not adopted—the commit retains IsAvailable: false.code diff:
In
resolveIPConflictsWithDB, remove the line that setsIsAvailable: falsewhencreating a
clearUpdatefor IP churn to avoid incorrectly marking a device asunavailable.
pkg/registry/registry.go [598-618]
[Suggestion processed]Suggestion importance[1-10]: 6
__
Why: The suggestion correctly identifies that setting
IsAvailable: falseduring an IP churn event is an incorrect assumption, as the device may still be online, and addresses the uncertainty noted in the code's comment.✅
Simplify tombstone creation logicSuggestion Impact:
The commit introduced a createTombstoneUpdate helper that initially sets Metadata to only {"_merged_into": targetDeviceID}, but it still copies over other metadata from the update afterward. While not fully applying the suggestion, the refactor touched the exact area and centralized tombstone creation, making it easier to adjust. Thus, the suggestion partially influenced the change.code diff:
Simplify the tombstone creation in
resolveIPConflictsWithDBby removing theredundant copy of all metadata, leaving only the essential
_merged_intofield.pkg/registry/registry.go [629-649]
[Suggestion processed]Suggestion importance[1-10]: 4
__
Why: The suggestion correctly points out that copying all metadata to a tombstone is redundant, as it is already being merged into the canonical device, and simplifying it improves code clarity and reduces data duplication.