Clarify config precedence between Helm values and KV #660

New issue

Closed

opened 2026-03-28 04:27:09 +00:00 by mfreeman451 · 1 comment

mfreeman451 commented

2026-03-28 04:27:09 +00:00

Owner

Imported from GitHub.

Original GitHub issue: #2030
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/2030
Original created: 2025-11-28T05:46:23Z

Problem\nIn k8s we keep a persistent /var/lib/serviceradar/core.json that is seeded once from the Helm ConfigMap. Later Helm updates (or values changes) do not overwrite the file, and CONFIG_SOURCE/kv env vars are empty. KV overlays can exist (via config/core.json in NATS), but it is unclear which source wins. In practice we set promotion/sightings-only in Helm, but core kept running with the old file, which caused identity settings to be wrong. Users also edit config via the UI/KV, so it is easy for Helm and KV to disagree with no visibility.\n\n## Why it matters\nOperators need a predictable source of truth for identity settings (and other config). Today Helm changes can silently be ignored because the persisted file remains, and KV overlays are opaque. This makes it hard to recover from drift (e.g., promotion accidentally on/off) and to reason about inventory counts.\n\n## Proposed work\n- Define and document clear precedence: Helm vs persisted file vs KV/UI (and whether PV persistence should be kept).\n- Expose the active config source and revision in a status endpoint/logs so operators can see what is actually applied.\n- Optional: disable or gate the persistent file seeding, or implement a managed reconciliation so Helm/values changes can overwrite/update the runtime file when intended.\n- Ensure CONFIG_SOURCE and KV settings are set/used consistently in the core deployment and init scripts.\n- Add tests/CI coverage for the precedence rules and a doc page for k8s users explaining how to change config safely.\n\n## Notes\nDiscovered while trying to switch identity to sightings-only/promotion-off: Helm values said off, KV was updated, but core kept the old on-disk config and ran with promotion on. The only reliable fix was hand-editing /var/lib/serviceradar/core.json.

Imported from GitHub. Original GitHub issue: #2030 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/2030 Original created: 2025-11-28T05:46:23Z --- ## Problem\nIn k8s we keep a persistent /var/lib/serviceradar/core.json that is seeded once from the Helm ConfigMap. Later Helm updates (or values changes) do not overwrite the file, and CONFIG_SOURCE/kv env vars are empty. KV overlays can exist (via config/core.json in NATS), but it is unclear which source wins. In practice we set promotion/sightings-only in Helm, but core kept running with the old file, which caused identity settings to be wrong. Users also edit config via the UI/KV, so it is easy for Helm and KV to disagree with no visibility.\n\n## Why it matters\nOperators need a predictable source of truth for identity settings (and other config). Today Helm changes can silently be ignored because the persisted file remains, and KV overlays are opaque. This makes it hard to recover from drift (e.g., promotion accidentally on/off) and to reason about inventory counts.\n\n## Proposed work\n- Define and document clear precedence: Helm vs persisted file vs KV/UI (and whether PV persistence should be kept).\n- Expose the active config source and revision in a status endpoint/logs so operators can see what is actually applied.\n- Optional: disable or gate the persistent file seeding, or implement a managed reconciliation so Helm/values changes can overwrite/update the runtime file when intended.\n- Ensure CONFIG_SOURCE and KV settings are set/used consistently in the core deployment and init scripts.\n- Add tests/CI coverage for the precedence rules and a doc page for k8s users explaining how to change config safely.\n\n## Notes\nDiscovered while trying to switch identity to sightings-only/promotion-off: Helm values said off, KV was updated, but core kept the old on-disk config and ran with promotion on. The only reliable fix was hand-editing /var/lib/serviceradar/core.json.

mfreeman451 commented

2026-03-28 04:27:09 +00:00

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/2030#issuecomment-3813980341
Original created: 2026-01-28T21:17:19Z

closing, stale

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/2030#issuecomment-3813980341 Original created: 2026-01-28T21:17:19Z --- closing, stale