bug(core-elx): mapper and sweep ingestion failing #895

New issue

Closed

opened 2026-03-28 04:29:39 +00:00 by mfreeman451 · 0 comments

mfreeman451 commented

2026-03-28 04:29:39 +00:00

Owner

Imported from GitHub.

Original GitHub issue: #2574
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/2574
Original created: 2026-01-29T00:39:18Z

Describe the bug
as seen in demo namespace (k8s) on core-elx

00:29:59.207 [info] StatusHandler received: service_type=sweep source=status service=network_sweep                                                                                               00:29:59.215 [info] ResultsRouter received: service_type=sysmon source=sysmon-metrics service=sysmon                                                                                             00:29:59.252 [info] ResultsRouter received: service_type=mapper source=results service=mapper                                                                                                    00:29:59.264 [info] ResultsRouter received: service_type=mapper_interfaces source=results service=mapper
00:29:59.268 [warning] Device UID lookup failed: %Postgrex.Error{message: nil, postgres: %{code: :undefined_table, line: "1449", message: "relation \"ocsf_devices\" does not exist", position: "31", file: "parse_relation.c", unknown: "ERROR", severity: "ERROR", pg_code: "42P01", routine: "parserOpenTable"}, connection_id: 18040, query: "SELECT o0.\"ip\", o0.\"uid\" FROM \"ocsf_devices\" AS o0 WHERE (o0.\"ip\" = ANY($1))"}                                                                                                                                                           00:29:59.511 [info] Coalesced 1 sync chunks into 3 updates
00:29:59.512 [info] Processing sync results                                                                                                                                                      00:29:59.512 [info] Decoded 3 sync updates                                                                                                                                                       00:29:59.512 [info] SyncIngestor: Processing 3 updates in batches of 500                                                                                                                         00:29:59.513 [warning] Bulk identifier lookup failed: %Postgrex.Error{message: nil, postgres: %{code: :undefined_table, line: "1449", message: "relation \"device_identifiers\" does not exist",
position: "89", file: "parse_relation.c", unknown: "ERROR", severity: "ERROR", pg_code: "42P01", routine: "parserOpenTable"}, connection_id: 18058, query: "SELECT d0.\"identifier_type\", d0.\"identifier_value\", d0.\"partition\", d0.\"device_id\" FROM \"device_identifiers\" AS d0 WHERE ((((d0.\"identifier_type\" = $1) AND (d0.\"identifier_value\" = $2)) AND (d0.\"partition\" = $3)) OR (((d0.\"identifier_type\" = $4) AND (d0.\"identifier_value\" = $5)) AND (d0.\"partition\" = $6)))"}                                                                                            00:29:59.515 [warning] Bulk device upsert failed: %Postgrex.Error{message: nil, postgres: %{code: :undefined_table, line: "1449", message: "relation \"ocsf_devices\" does not exist", position:
"13", file: "parse_relation.c", unknown: "ERROR", severity: "ERROR", pg_code: "42P01", routine: "parserOpenTable"}, connection_id: 18068, query: "INSERT INTO \"ocsf_devices\" AS o0 (\"name\",\"ip\",\"metadata\",\"uid\",\"hostname\",\"mac\",\"tags\",\"vendor_name\",\"first_seen_time\",\"last_seen_time\",\"created_time\",\"modified_time\",\"model\",\"discovery_sources\",\"is_available\") VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15),($16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30) ON CONFLICT (\"uid\") DO UPDATE SET \"ip\" = COALESCE(EXCLUDED.ip, o0.\"ip\"), \"mac\" = COALESCE(EXCLUDED.mac, o0.\"mac\"), \"hostname\" = COALESCE(EXCLUDED.hostname, o0.\"hostname\"), \"name\" = COALESCE(EXCLUDED.name, o0.\"name\"), \"vendor_name\" = COALESCE
(EXCLUDED.vendor_name, o0.\"vendor_name\"), \"model\" = COALESCE(EXCLUDED.model, o0.\"model\"), \"is_available\" = COALESCE(EXCLUDED.is_available, o0.\"is_available\"), \"metadata\" = COALESCE(EXCLUDED.metadata, o0.\"metadata\"), \"discovery_sources\" = (SELECT array_agg(DISTINCT src) FROM unnest(array_cat(COALESCE(o0.\"discovery_sources\", ARRAY[]::text[]), EXCLUDED.discovery_sources)) AS src WHERE src IS NOT NULL AND src <> ''), \"last_seen_time\" = EXCLUDED.last_seen_time, \"modified_time\" = EXCLUDED.modified_time"}
00:29:59.516 [warning] Bulk identifier upsert failed: %Postgrex.Error{message: nil, postgres: %{code: :undefined_table, line: "1449", message: "relation \"device_identifiers\" does not exist", position: "13", file: "parse_relation.c", unknown: "ERROR", severity: "ERROR", pg_code: "42P01", routine: "parserOpenTable"}, connection_id: 18042, query: "INSERT INTO \"device_identifiers\" (\"partition\",\"source\",\"identifier_value\",\"device_id\",\"first_seen\",\"last_seen\",\"identifier_type\",\"confidence\") VALUES ($1,$2,$3,$4,$5,$6,$7,$8),($9,$10,$11,$12,$13,$14,$15,$16) ON
CONFLICT (\"identifier_type\",\"identifier_value\",\"partition\") DO UPDATE SET \"device_id\" = EXCLUDED.\"device_id\",\"last_seen\" = EXCLUDED.\"last_seen\""}                                  00:29:59.516 [info] SyncIngestor: Completed 3 updates in 4ms (750.0 devices/sec)                                                                                                                 00:29:59.516 [info] SyncIngestor result: 🆗

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Imported from GitHub. Original GitHub issue: #2574 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/2574 Original created: 2026-01-29T00:39:18Z --- **Describe the bug** as seen in `demo` namespace (k8s) on `core-elx` ``` 00:29:59.207 [info] StatusHandler received: service_type=sweep source=status service=network_sweep 00:29:59.215 [info] ResultsRouter received: service_type=sysmon source=sysmon-metrics service=sysmon 00:29:59.252 [info] ResultsRouter received: service_type=mapper source=results service=mapper 00:29:59.264 [info] ResultsRouter received: service_type=mapper_interfaces source=results service=mapper 00:29:59.268 [warning] Device UID lookup failed: %Postgrex.Error{message: nil, postgres: %{code: :undefined_table, line: "1449", message: "relation \"ocsf_devices\" does not exist", position: "31", file: "parse_relation.c", unknown: "ERROR", severity: "ERROR", pg_code: "42P01", routine: "parserOpenTable"}, connection_id: 18040, query: "SELECT o0.\"ip\", o0.\"uid\" FROM \"ocsf_devices\" AS o0 WHERE (o0.\"ip\" = ANY($1))"} 00:29:59.511 [info] Coalesced 1 sync chunks into 3 updates 00:29:59.512 [info] Processing sync results 00:29:59.512 [info] Decoded 3 sync updates 00:29:59.512 [info] SyncIngestor: Processing 3 updates in batches of 500 00:29:59.513 [warning] Bulk identifier lookup failed: %Postgrex.Error{message: nil, postgres: %{code: :undefined_table, line: "1449", message: "relation \"device_identifiers\" does not exist", position: "89", file: "parse_relation.c", unknown: "ERROR", severity: "ERROR", pg_code: "42P01", routine: "parserOpenTable"}, connection_id: 18058, query: "SELECT d0.\"identifier_type\", d0.\"identifier_value\", d0.\"partition\", d0.\"device_id\" FROM \"device_identifiers\" AS d0 WHERE ((((d0.\"identifier_type\" = $1) AND (d0.\"identifier_value\" = $2)) AND (d0.\"partition\" = $3)) OR (((d0.\"identifier_type\" = $4) AND (d0.\"identifier_value\" = $5)) AND (d0.\"partition\" = $6)))"} 00:29:59.515 [warning] Bulk device upsert failed: %Postgrex.Error{message: nil, postgres: %{code: :undefined_table, line: "1449", message: "relation \"ocsf_devices\" does not exist", position: "13", file: "parse_relation.c", unknown: "ERROR", severity: "ERROR", pg_code: "42P01", routine: "parserOpenTable"}, connection_id: 18068, query: "INSERT INTO \"ocsf_devices\" AS o0 (\"name\",\"ip\",\"metadata\",\"uid\",\"hostname\",\"mac\",\"tags\",\"vendor_name\",\"first_seen_time\",\"last_seen_time\",\"created_time\",\"modified_time\",\"model\",\"discovery_sources\",\"is_available\") VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15),($16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30) ON CONFLICT (\"uid\") DO UPDATE SET \"ip\" = COALESCE(EXCLUDED.ip, o0.\"ip\"), \"mac\" = COALESCE(EXCLUDED.mac, o0.\"mac\"), \"hostname\" = COALESCE(EXCLUDED.hostname, o0.\"hostname\"), \"name\" = COALESCE(EXCLUDED.name, o0.\"name\"), \"vendor_name\" = COALESCE (EXCLUDED.vendor_name, o0.\"vendor_name\"), \"model\" = COALESCE(EXCLUDED.model, o0.\"model\"), \"is_available\" = COALESCE(EXCLUDED.is_available, o0.\"is_available\"), \"metadata\" = COALESCE(EXCLUDED.metadata, o0.\"metadata\"), \"discovery_sources\" = (SELECT array_agg(DISTINCT src) FROM unnest(array_cat(COALESCE(o0.\"discovery_sources\", ARRAY[]::text[]), EXCLUDED.discovery_sources)) AS src WHERE src IS NOT NULL AND src <> ''), \"last_seen_time\" = EXCLUDED.last_seen_time, \"modified_time\" = EXCLUDED.modified_time"} 00:29:59.516 [warning] Bulk identifier upsert failed: %Postgrex.Error{message: nil, postgres: %{code: :undefined_table, line: "1449", message: "relation \"device_identifiers\" does not exist", position: "13", file: "parse_relation.c", unknown: "ERROR", severity: "ERROR", pg_code: "42P01", routine: "parserOpenTable"}, connection_id: 18042, query: "INSERT INTO \"device_identifiers\" (\"partition\",\"source\",\"identifier_value\",\"device_id\",\"first_seen\",\"last_seen\",\"identifier_type\",\"confidence\") VALUES ($1,$2,$3,$4,$5,$6,$7,$8),($9,$10,$11,$12,$13,$14,$15,$16) ON CONFLICT (\"identifier_type\",\"identifier_value\",\"partition\") DO UPDATE SET \"device_id\" = EXCLUDED.\"device_id\",\"last_seen\" = EXCLUDED.\"last_seen\""} 00:29:59.516 [info] SyncIngestor: Completed 3 updates in 4ms (750.0 devices/sec) 00:29:59.516 [info] SyncIngestor result: 🆗 ``` **To Reproduce** Steps to reproduce the behavior: 1. Go to '...' 2. Click on '....' 3. Scroll down to '....' 4. See error **Expected behavior** A clear and concise description of what you expected to happen. **Screenshots** If applicable, add screenshots to help explain your problem. **Desktop (please complete the following information):** - OS: [e.g. iOS] - Browser [e.g. chrome, safari] - Version [e.g. 22] **Smartphone (please complete the following information):** - Device: [e.g. iPhone6] - OS: [e.g. iOS8.1] - Browser [e.g. stock browser, safari] - Version [e.g. 22] **Additional context** Add any other context about the problem here.