chore(kong): enable jwks/oidc plugin #545

New issue

Closed

opened 2026-03-28 04:25:34 +00:00 by mfreeman451 · 1 comment

mfreeman451 commented

2026-03-28 04:25:34 +00:00

Owner

Imported from GitHub.

Original GitHub issue: #1716
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1716
Original created: 2025-10-06T00:32:00Z

Title: Investigate using Kong’s OIDC/JWKS plugin for automatic key refresh

Summary
Core rotates its RS256 signing key on restart (when the key file is missing), but Kong only fetches /auth/jwks.json during pod init. Whenever Core comes up with a new kid, Kong keeps the old key and every JWT validation fails (401 “No credentials found for given ‘kid’”) until Kong is recycled. We recently hit this in demo after restarting Core.

Current Behavior

Core init job regenerates keys and publishes them via /auth/jwks.json.
Kong’s init container renders a static kong.yml with the public key and never reloads it.
Result: Core restarts (or key rotations) require a manual Kong restart to sync keys.

Why This Matters

Operational overhead: pods must be bounced in the right order.
Risk of unexpected outages if Core restarts independently (CrashLoop, node drain, etc.).
We just removed the --force flag so Core retains its key on restart, but pod rescheduling or PVC recreation would still rotate the key and break auth.

Proposal
Evaluate switching to a Kong plugin that supports JWKS refresh (e.g. kong-oidc) or otherwise teaching the existing setup to re-pull JWKS on a timer. Key questions:

Can we configure Kong to poll JWKS and cache keys automatically?
Do we need to expose additional OIDC metadata from Core?
Are there compatibility concerns with our current routing/auth assumptions?
What is the migration path from the current static JWT plugin?

Acceptance Criteria

Kong accepts new JWT kids without requiring a manual restart.
Document the configuration (intervals, caching behavior, failure modes).
Validate in demo that a forced Core key rotation doesn’t break auth.

we would also need to roll back this https://github.com/carverauto/serviceradar/pull/1715

Imported from GitHub. Original GitHub issue: #1716 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1716 Original created: 2025-10-06T00:32:00Z --- **Title:** Investigate using Kong’s OIDC/JWKS plugin for automatic key refresh **Summary** Core rotates its RS256 signing key on restart (when the key file is missing), but Kong only fetches `/auth/jwks.json` during pod init. Whenever Core comes up with a new `kid`, Kong keeps the old key and every JWT validation fails (401 “No credentials found for given ‘kid’”) until Kong is recycled. We recently hit this in `demo` after restarting Core. **Current Behavior** - Core init job regenerates keys and publishes them via `/auth/jwks.json`. - Kong’s init container renders a static `kong.yml` with the public key and never reloads it. - Result: Core restarts (or key rotations) require a manual Kong restart to sync keys. **Why This Matters** - Operational overhead: pods must be bounced in the right order. - Risk of unexpected outages if Core restarts independently (CrashLoop, node drain, etc.). - We just removed the `--force` flag so Core retains its key on restart, but pod rescheduling or PVC recreation would still rotate the key and break auth. **Proposal** Evaluate switching to a Kong plugin that supports JWKS refresh (e.g. `kong-oidc`) or otherwise teaching the existing setup to re-pull JWKS on a timer. Key questions: - Can we configure Kong to poll JWKS and cache keys automatically? - Do we need to expose additional OIDC metadata from Core? - Are there compatibility concerns with our current routing/auth assumptions? - What is the migration path from the current static JWT plugin? **Acceptance Criteria** 1. Kong accepts new JWT `kid`s without requiring a manual restart. 2. Document the configuration (intervals, caching behavior, failure modes). 3. Validate in `demo` that a forced Core key rotation doesn’t break auth. we would also need to roll back this https://github.com/carverauto/serviceradar/pull/1715

mfreeman451 commented

2026-03-28 04:25:34 +00:00

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1716#issuecomment-3814009175
Original created: 2026-01-28T21:24:20Z

closing, stale

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1716#issuecomment-3814009175 Original created: 2026-01-28T21:24:20Z --- closing, stale