feat: logical deletion system #397

New issue

Closed

opened 2026-03-28 04:24:11 +00:00 by mfreeman451 · 1 comment

mfreeman451 commented

2026-03-28 04:24:11 +00:00

Owner

Imported from GitHub.

Original GitHub issue: #1207
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1207
Original created: 2025-07-08T04:17:23Z

Need to implement a logical deletion system using the retraction event pattern.

#1208
update srql to filter on metadata for tombstoned devices

Sync service

The sync service will publish a special "retraction event" (a specific kind of tombstone) for any device that is no longer detected by an integration (the integration being considered the source of truth, Netbox or Armis).

The SRQL query engine needs to be updated to filter out these retracted devices automatically.

For each removed device, the sync service must publish a final event to the sweep_results stream with the following structure:

device_id: The ID of the removed device (e.g., for deviceC).
discovery_source: 'armis'.
ip: The device's IP address.
available: false.
timestamp: The time of the sync.
metadata: Add a special flag to the metadata map:
{"_deleted": "true"}

The sync service can make a call into the SRQL Query API using
show devices where discovery_sources = "armis" and discovery_sources = "sweep", if it finds
a device in the list that is not in the list of devices from the latest Armis API call, we create the tombstone event.

SRQL

pkg/srql/parser/translator.go

buildProtonWhere function updates:

When the translator builds a query for the devices entity, it will automatically append a condition to filter out deleted records.

A query of:

show devices where discovery_sources = "armis"

Will be translated to the following Proton SQL:

SELECT * FROM table(unified_devices)
WHERE
  -- This part is added automatically and implicitly --
  NOT has(map_keys(metadata), '_deleted')
  -- This is the user's original condition --
  AND has(discovery_sources, 'armis')

Imported from GitHub. Original GitHub issue: #1207 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1207 Original created: 2025-07-08T04:17:23Z --- Need to implement a logical deletion system using the retraction event pattern. - [x] #1208 - [x] update srql to filter on metadata for tombstoned devices ## Sync service The `sync` service will publish a special "retraction event" (a specific kind of tombstone) for any device that is no longer detected by an integration (the integration being considered the source of truth, Netbox or Armis). The SRQL query engine needs to be updated to filter out these retracted devices automatically. For each removed device, the sync service must publish a final event to the sweep_results stream with the following structure: * device_id: The ID of the removed device (e.g., for deviceC). * discovery_source: 'armis'. * ip: The device's IP address. * available: false. * timestamp: The time of the sync. * metadata: Add a special flag to the metadata map: {"_deleted": "true"} The sync service can make a call into the SRQL Query API using `show devices where discovery_sources = "armis" and discovery_sources = "sweep"`, if it finds a device in the list that is not in the list of devices from the latest Armis API call, we create the tombstone event. ## SRQL `pkg/srql/parser/translator.go` **buildProtonWhere** function updates: When the translator builds a query for the devices entity, it will automatically append a condition to filter out deleted records. A query of: `show devices where discovery_sources = "armis"` Will be translated to the following Proton SQL: ```sql SELECT * FROM table(unified_devices) WHERE -- This part is added automatically and implicitly -- NOT has(map_keys(metadata), '_deleted') -- This is the user's original condition -- AND has(discovery_sources, 'armis') ```

mfreeman451 commented

2026-03-28 04:24:11 +00:00

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1207#issuecomment-3047521022
Original created: 2025-07-08T06:20:59Z

closing as completed

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1207#issuecomment-3047521022 Original created: 2025-07-08T06:20:59Z --- closing as completed