bug(core-elx: core unavailable, zen rule reconcile failed #733

New issue

Closed

opened 2026-03-28 04:27:57 +00:00 by mfreeman451 · 1 comment

mfreeman451 commented

2026-03-28 04:27:57 +00:00

Owner

Imported from GitHub.

Original GitHub issue: #2271
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/2271
Original created: 2026-01-12T07:38:20Z

Describe the bug

06:18:44.037 [error] gRPC error putting key agents/default-agent/events/logs.snmp/snmp_severity.json: :down: :noproc
06:18:44.037 [warning] Zen rule reconcile failed: %GRPC.RPCError{status: 2, message: ":down: :noproc", details: nil}
06:18:55.432 [warning] [message: "Oban.Notifier.status/1 check failed due to {:timeout, {GenServer, :call, [{:via, Registry, {Oban.Registry, {{:via, Registry, {ServiceRadar.LocalRegistry, {:tenant_oban, \"tenant_platform\"}}}, Oban.Sonar}}}, :get_status, 5000]}}", source: :oban, module: Oban.Notifier]
06:18:56.364 [warning] [message: "Oban.Notifier.status/1 check failed due to {:timeout, {GenServer, :call, [{:via, Registry, {Oban.Registry, {Oban, Oban.Sonar}}}, :get_status, 5000]}}", source: :oban, module: Oban.Notifier]
06:19:01.438 [warning] [message: "Oban.Notifier.status/1 check failed due to {:timeout, {GenServer, :call, [{:via, Registry, {Oban.Registry, {{:via, Registry, {ServiceRadar.LocalRegistry, {:tenant_oban, \"tenant_platform\"}}}, Oban.Sonar}}}, :get_status, 5000]}}", source: :oban, module: Oban.Notifier]
06:19:02.366 [warning] [message: "Oban.Notifier.status/1 check failed due to {:timeout, {GenServer, :call, [{:via, Registry, {Oban.Registry, {Oban, Oban.Sonar}}}, :get_status, 5000]}}", source: :oban, module: Oban.Notifier]
06:19:05.398 [error] Postgrex.Protocol (#PID<0.5849.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.6091.0> timed out because it queued and checked out the connection for longer than 15000ms

#PID<0.6091.0> was at location:

    (stdlib 7.1) gen.erl:243: :gen.do_call/4
    (stdlib 7.1) gen_statem.erl:3261: :gen_statem.call/3
    (ssl 11.4) ssl_gen_statem.erl:1284: :ssl_gen_statem.call/2
    (postgrex 0.21.1) lib/postgrex/protocol.ex:3261: Postgrex.Protocol.msg_recv/4
    (postgrex 0.21.1) lib/postgrex/protocol.ex:2263: Postgrex.Protocol.recv_bind/3
    (postgrex 0.21.1) lib/postgrex/protocol.ex:2118: Postgrex.Protocol.bind_execute_close/4
    (db_connection 2.8.1) lib/db_connection/holder.ex:354: DBConnection.Holder.holder_apply/4
    (db_connection 2.8.1) lib/db_connection.ex:1555: DBConnection.run_execute/5

06:19:05.398 [error] Postgrex.Protocol (#PID<0.5866.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.6209.0> timed out because it queued and checked out the connection for longer than 15000ms

#PID<0.6209.0> was at location:

    (stdlib 7.1) gen.erl:243: :gen.do_call/4
    (stdlib 7.1) gen_statem.erl:3261: :gen_statem.call/3
    (ssl 11.4) ssl_gen_statem.erl:1284: :ssl_gen_statem.call/2
    (postgrex 0.21.1) lib/postgrex/protocol.ex:3261: Postgrex.Protocol.msg_recv/4
    (postgrex 0.21.1) lib/postgrex/protocol.ex:2263: Postgrex.Protocol.recv_bind/3
    (postgrex 0.21.1) lib/postgrex/protocol.ex:2118: Postgrex.Protocol.bind_execute_close/4
    (db_connection 2.8.1) lib/db_connection/holder.ex:354: DBConnection.Holder.holder_apply/4
    (db_connection 2.8.1) lib/db_connection.ex:1555: DBConnection.run_execute/5

06:19:06.440 [warning] [message: "Oban.Peer.leader?/2 check failed due to {:timeout, {GenServer, :call, [#PID<0.6208.0>, :leader?, 5000]}}", source: :oban, module: Oban.Peer]
...
07:33:44.264 [warning] Zen rule reconcile failed: %GRPC.RPCError{status: 2, message: ":down: :noproc", details: nil}
07:33:44.264 [warning] The connection process #PID<0.6128.0> is not alive, please create a new channel via GRPC.Stub.connect/2
07:33:44.264 [error] gRPC error putting key agents/default-agent/events/logs.snmp/snmp_severity.json: :down: :noproc
07:33:44.264 [warning] Zen rule reconcile failed: %GRPC.RPCError{status: 2, message: ":down: :noproc", details: nil}

The agent gateway can't talk to the core right now, presumably because of this issue. Issues like this should not make the core unavailable to the agent gateway.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Imported from GitHub. Original GitHub issue: #2271 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/2271 Original created: 2026-01-12T07:38:20Z --- **Describe the bug** ``` 06:18:44.037 [error] gRPC error putting key agents/default-agent/events/logs.snmp/snmp_severity.json: :down: :noproc 06:18:44.037 [warning] Zen rule reconcile failed: %GRPC.RPCError{status: 2, message: ":down: :noproc", details: nil} 06:18:55.432 [warning] [message: "Oban.Notifier.status/1 check failed due to {:timeout, {GenServer, :call, [{:via, Registry, {Oban.Registry, {{:via, Registry, {ServiceRadar.LocalRegistry, {:tenant_oban, \"tenant_platform\"}}}, Oban.Sonar}}}, :get_status, 5000]}}", source: :oban, module: Oban.Notifier] 06:18:56.364 [warning] [message: "Oban.Notifier.status/1 check failed due to {:timeout, {GenServer, :call, [{:via, Registry, {Oban.Registry, {Oban, Oban.Sonar}}}, :get_status, 5000]}}", source: :oban, module: Oban.Notifier] 06:19:01.438 [warning] [message: "Oban.Notifier.status/1 check failed due to {:timeout, {GenServer, :call, [{:via, Registry, {Oban.Registry, {{:via, Registry, {ServiceRadar.LocalRegistry, {:tenant_oban, \"tenant_platform\"}}}, Oban.Sonar}}}, :get_status, 5000]}}", source: :oban, module: Oban.Notifier] 06:19:02.366 [warning] [message: "Oban.Notifier.status/1 check failed due to {:timeout, {GenServer, :call, [{:via, Registry, {Oban.Registry, {Oban, Oban.Sonar}}}, :get_status, 5000]}}", source: :oban, module: Oban.Notifier] 06:19:05.398 [error] Postgrex.Protocol (#PID<0.5849.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.6091.0> timed out because it queued and checked out the connection for longer than 15000ms #PID<0.6091.0> was at location: (stdlib 7.1) gen.erl:243: :gen.do_call/4 (stdlib 7.1) gen_statem.erl:3261: :gen_statem.call/3 (ssl 11.4) ssl_gen_statem.erl:1284: :ssl_gen_statem.call/2 (postgrex 0.21.1) lib/postgrex/protocol.ex:3261: Postgrex.Protocol.msg_recv/4 (postgrex 0.21.1) lib/postgrex/protocol.ex:2263: Postgrex.Protocol.recv_bind/3 (postgrex 0.21.1) lib/postgrex/protocol.ex:2118: Postgrex.Protocol.bind_execute_close/4 (db_connection 2.8.1) lib/db_connection/holder.ex:354: DBConnection.Holder.holder_apply/4 (db_connection 2.8.1) lib/db_connection.ex:1555: DBConnection.run_execute/5 06:19:05.398 [error] Postgrex.Protocol (#PID<0.5866.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.6209.0> timed out because it queued and checked out the connection for longer than 15000ms #PID<0.6209.0> was at location: (stdlib 7.1) gen.erl:243: :gen.do_call/4 (stdlib 7.1) gen_statem.erl:3261: :gen_statem.call/3 (ssl 11.4) ssl_gen_statem.erl:1284: :ssl_gen_statem.call/2 (postgrex 0.21.1) lib/postgrex/protocol.ex:3261: Postgrex.Protocol.msg_recv/4 (postgrex 0.21.1) lib/postgrex/protocol.ex:2263: Postgrex.Protocol.recv_bind/3 (postgrex 0.21.1) lib/postgrex/protocol.ex:2118: Postgrex.Protocol.bind_execute_close/4 (db_connection 2.8.1) lib/db_connection/holder.ex:354: DBConnection.Holder.holder_apply/4 (db_connection 2.8.1) lib/db_connection.ex:1555: DBConnection.run_execute/5 06:19:06.440 [warning] [message: "Oban.Peer.leader?/2 check failed due to {:timeout, {GenServer, :call, [#PID<0.6208.0>, :leader?, 5000]}}", source: :oban, module: Oban.Peer] ... 07:33:44.264 [warning] Zen rule reconcile failed: %GRPC.RPCError{status: 2, message: ":down: :noproc", details: nil} 07:33:44.264 [warning] The connection process #PID<0.6128.0> is not alive, please create a new channel via GRPC.Stub.connect/2 07:33:44.264 [error] gRPC error putting key agents/default-agent/events/logs.snmp/snmp_severity.json: :down: :noproc 07:33:44.264 [warning] Zen rule reconcile failed: %GRPC.RPCError{status: 2, message: ":down: :noproc", details: nil} ``` The agent gateway can't talk to the core right now, presumably because of this issue. Issues like this should not make the core unavailable to the agent gateway. **To Reproduce** Steps to reproduce the behavior: 1. Go to '...' 2. Click on '....' 3. Scroll down to '....' 4. See error **Expected behavior** A clear and concise description of what you expected to happen. **Screenshots** If applicable, add screenshots to help explain your problem. **Desktop (please complete the following information):** - OS: [e.g. iOS] - Browser [e.g. chrome, safari] - Version [e.g. 22] **Smartphone (please complete the following information):** - Device: [e.g. iPhone6] - OS: [e.g. iOS8.1] - Browser [e.g. stock browser, safari] - Version [e.g. 22] **Additional context** Add any other context about the problem here.

mfreeman451 commented

2026-03-28 04:27:57 +00:00

Author

Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/2271#issuecomment-3747943134
Original created: 2026-01-14T06:11:01Z

closing as fixed

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/2271#issuecomment-3747943134 Original created: 2026-01-14T06:11:01Z --- closing as fixed