feat(poller): nested spire agent #621

New issue

Closed

opened 2026-03-28 04:26:32 +00:00 by mfreeman451 · 0 comments

mfreeman451 commented

2026-03-28 04:26:32 +00:00

Owner

Imported from GitHub.

Original GitHub issue: #1894
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1894
Original created: 2025-10-26T04:11:54Z

Implementation plan for embedding nested SPIRE servers in pollers

This plan outlines the steps to introduce a nested SPIRE architecture. A root SPIRE server will reside in your core network, and downstream SPIRE servers will be embedded within your pollers at the edge. This provides localized identity management, resilience, and reduced network traffic to the core.

Phase 1: Preparation and core setup

Initial architecture and prerequisites
Core network: Hosts the root SPIRE server(s) and its SPIRE agent. This agent will attest and issue an intermediate CA to the downstream SPIRE server in the poller.
Edge network: Hosts the poller containing the nested SPIRE server and agent. Edge microservices connect to this local agent.
Network connectivity: Ensure the following connections are allowed:
Edge poller (specifically its SPIRE agent) can connect to the core SPIRE server on the SPIFFE server API port (default 8081).
Microservices at the edge can connect to the local poller's SPIRE agent on the Workload API socket.
Configure and deploy the core SPIRE server
Set up your core SPIRE server and its corresponding agent. This is your trust anchor.
The server.conf file for the root server requires no special nested configuration.
The agent.conf for the root agent should be configured to attest the downstream server. Using a node attestation plugin like x509pop or join_token is common.
Create a registration entry for the downstream server
On the root SPIRE server, create a registration entry for the poller's SPIRE server. This entry is how the root server authorizes the downstream server to receive its intermediate signing certificate.
Use the -downstream flag to indicate that this entry is for another SPIRE server.

The parentID will be the SPIFFE ID of the root SPIRE agent, and the spiffeID will be the identity you assign to the downstream server, e.g., spiffe:///nested/poller-1.

Example command on the root server:

# This example uses the 'x509pop' attestor and a 'docker:label' selector.
# Replace with your actual attestor and selector.
spire-server entry create \
    -parentID "spiffe://<trust-domain>/spire/agent/x509pop/..." \
    -spiffeID "spiffe://<trust-domain>/nested/poller-1" \
    -selector "docker:label:org.example.name:poller-server" \
    -downstream

Use code with caution.

Phase 2: Poller and nested server integration

Design the poller architecture
The poller will run two SPIRE components:
A SPIRE agent, which connects upstream to the root server.
A SPIRE server, which uses the agent's Workload API to act as a downstream authority for edge microservices.
Configure and deploy the nested SPIRE server in the poller
The nested server's configuration must specify the spire UpstreamAuthority plugin, which tells it to get its intermediate CA from an upstream SPIRE server.

This plugin will connect to the local SPIRE agent's Workload API socket.

Example server.conf for the nested server:

server {
    bind_address = "127.0.0.1"
    bind_port = "8081"
    trust_domain = "<your-trust-domain>"
    upstream_authority {
        plugin_type = "UpstreamAuthority"
        plugin_name = "spire"
        plugin_data {
            server_address = "spire-root-server.internal" // Or another address for your root server
            server_port = "8081"
            workload_api_socket = "/tmp/spire-workload-api.sock" // Path to local agent socket
        }
    }
    # ... other server settings
}

Use code with caution.

Configure and deploy the SPIRE agent in the poller
The poller's SPIRE agent will point to the root server for attestation.

It is crucial that the nested server and agent can share the same Workload API socket, likely through a volume mount, as shown in the example Docker Compose scenarios.

Example agent.conf for the nested agent:

agent {
    server_address = "spire-root-server.internal" // Address of the root SPIRE server
    server_port = "8081"
    trust_domain = "<your-trust-domain>"
    data_dir = "/run/spire/data/agent"
    workload_api_socket = "/tmp/spire-workload-api.sock" // Shared socket for the nested server
    # ... other agent settings, including node attestor configuration
}

Phase 3: Edge microservice and poller integration

Configure edge microservices
Instead of talking to a remote SPIRE agent, each microservice at the edge will connect to the local poller's SPIRE agent via the Workload API socket.
The poller's SPIRE agent will then issue credentials to the microservices, signed by the nested server's intermediate CA.
Register edge workloads on the nested server
Registration entries for the edge microservices are now managed on the nested SPIRE server.

This keeps edge-specific configuration localized and simplifies the root server's responsibilities.

Example command on the nested server:

spire-server entry create \
    -parentID "spiffe://<trust-domain>/nested/poller-1" \
    -spiffeID "spiffe://<trust-domain>/edge/service-a" \
    -selector "unix:uid:1001"

Phase 4: Testing and validation

Verify SVID issuance: After deployment, confirm that edge microservices successfully retrieve SVIDs from the local poller's agent. Check the SPIFFE ID and the certificate's issuer to ensure it comes from the nested server's intermediate CA.

Test mTLS: Validate that microservices can use their SVIDs to establish mTLS connections with each other and with services in the core network. This confirms the trust chain is working end-to-end.
Simulate failure: Disconnect the poller from the core network. The nested SPIRE server should continue to function and issue/rotate certificates for edge services based on its cached intermediate CA, proving its resilience.

Alternative: SPIRE agent on each poller

This nested server approach adds complexity by embedding a full SPIRE server in each poller. For simpler deployments, consider running only a SPIRE agent on the poller nodes.
Simpler setup: Fewer components to manage on the poller.
Reduced resource usage: No full SPIRE server instance, only the agent.
Requires connectivity: Pollers must maintain connectivity to the core SPIRE server to renew their intermediate CA and receive trust bundle updates.
Less resilient: If the core SPIRE server is unreachable, certificate issuance and rotation will eventually fail once the grace period expires.

References

https://spiffe.io/docs/latest/architecture/nested/readme/
github.com/spiffe/spiffe.io@2a4440103b/content/docs/latest/spire-helm-charts-hardened-advanced/nested-spire.md

Imported from GitHub. Original GitHub issue: #1894 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1894 Original created: 2025-10-26T04:11:54Z --- # Implementation plan for embedding nested SPIRE servers in pollers This plan outlines the steps to introduce a nested SPIRE architecture. A root SPIRE server will reside in your core network, and downstream SPIRE servers will be embedded within your pollers at the edge. This provides localized identity management, resilience, and reduced network traffic to the core. ## Phase 1: Preparation and core setup 1. Initial architecture and prerequisites Core network: Hosts the root SPIRE server(s) and its SPIRE agent. This agent will attest and issue an intermediate CA to the downstream SPIRE server in the poller. Edge network: Hosts the poller containing the nested SPIRE server and agent. Edge microservices connect to this local agent. Network connectivity: Ensure the following connections are allowed: Edge poller (specifically its SPIRE agent) can connect to the core SPIRE server on the SPIFFE server API port (default 8081). Microservices at the edge can connect to the local poller's SPIRE agent on the Workload API socket. 2. Configure and deploy the core SPIRE server Set up your core SPIRE server and its corresponding agent. This is your trust anchor. The server.conf file for the root server requires no special nested configuration. The agent.conf for the root agent should be configured to attest the downstream server. Using a node attestation plugin like x509pop or join_token is common. 3. Create a registration entry for the downstream server On the root SPIRE server, create a registration entry for the poller's SPIRE server. This entry is how the root server authorizes the downstream server to receive its intermediate signing certificate. Use the -downstream flag to indicate that this entry is for another SPIRE server. The parentID will be the SPIFFE ID of the root SPIRE agent, and the spiffeID will be the identity you assign to the downstream server, e.g., spiffe://<trust-domain>/nested/poller-1. Example command on the root server: ```bash # This example uses the 'x509pop' attestor and a 'docker:label' selector. # Replace with your actual attestor and selector. spire-server entry create \ -parentID "spiffe://<trust-domain>/spire/agent/x509pop/..." \ -spiffeID "spiffe://<trust-domain>/nested/poller-1" \ -selector "docker:label:org.example.name:poller-server" \ -downstream ``` **Use code with caution.** ### Phase 2: Poller and nested server integration 1. Design the poller architecture The poller will run two SPIRE components: A SPIRE agent, which connects upstream to the root server. A SPIRE server, which uses the agent's Workload API to act as a downstream authority for edge microservices. 2. Configure and deploy the nested SPIRE server in the poller The nested server's configuration must specify the spire UpstreamAuthority plugin, which tells it to get its intermediate CA from an upstream SPIRE server. This plugin will connect to the local SPIRE agent's Workload API socket. Example server.conf for the nested server: ```hcl server { bind_address = "127.0.0.1" bind_port = "8081" trust_domain = "<your-trust-domain>" upstream_authority { plugin_type = "UpstreamAuthority" plugin_name = "spire" plugin_data { server_address = "spire-root-server.internal" // Or another address for your root server server_port = "8081" workload_api_socket = "/tmp/spire-workload-api.sock" // Path to local agent socket } } # ... other server settings } ``` **Use code with caution.** 3. Configure and deploy the SPIRE agent in the poller The poller's SPIRE agent will point to the root server for attestation. It is crucial that the nested server and agent can share the same Workload API socket, likely through a volume mount, as shown in the example Docker Compose scenarios. Example agent.conf for the nested agent: ```hcl agent { server_address = "spire-root-server.internal" // Address of the root SPIRE server server_port = "8081" trust_domain = "<your-trust-domain>" data_dir = "/run/spire/data/agent" workload_api_socket = "/tmp/spire-workload-api.sock" // Shared socket for the nested server # ... other agent settings, including node attestor configuration } ``` ### Phase 3: Edge microservice and poller integration 1. Configure edge microservices Instead of talking to a remote SPIRE agent, each microservice at the edge will connect to the local poller's SPIRE agent via the Workload API socket. The poller's SPIRE agent will then issue credentials to the microservices, signed by the nested server's intermediate CA. 2. Register edge workloads on the nested server Registration entries for the edge microservices are now managed on the nested SPIRE server. This keeps edge-specific configuration localized and simplifies the root server's responsibilities. Example command on the nested server: ```bash spire-server entry create \ -parentID "spiffe://<trust-domain>/nested/poller-1" \ -spiffeID "spiffe://<trust-domain>/edge/service-a" \ -selector "unix:uid:1001" ``` ### Phase 4: Testing and validation Verify SVID issuance: After deployment, confirm that edge microservices successfully retrieve SVIDs from the local poller's agent. Check the SPIFFE ID and the certificate's issuer to ensure it comes from the nested server's intermediate CA. Test mTLS: Validate that microservices can use their SVIDs to establish mTLS connections with each other and with services in the core network. This confirms the trust chain is working end-to-end. Simulate failure: Disconnect the poller from the core network. The nested SPIRE server should continue to function and issue/rotate certificates for edge services based on its cached intermediate CA, proving its resilience. Alternative: SPIRE agent on each poller This nested server approach adds complexity by embedding a full SPIRE server in each poller. For simpler deployments, consider running only a SPIRE agent on the poller nodes. Simpler setup: Fewer components to manage on the poller. Reduced resource usage: No full SPIRE server instance, only the agent. Requires connectivity: Pollers must maintain connectivity to the core SPIRE server to renew their intermediate CA and receive trust bundle updates. Less resilient: If the core SPIRE server is unreachable, certificate issuance and rotation will eventually fail once the grace period expires. ## References https://spiffe.io/docs/latest/architecture/nested/readme/ https://github.com/spiffe/spiffe.io/blob/2a4440103b344a7582221ff1c6ed6a25be781c5c/content/docs/latest/spire-helm-charts-hardened-advanced/nested-spire.md