bug(sync): KV unable to do large writes #400

New issue

Closed

opened 2026-03-28 04:24:12 +00:00 by mfreeman451 · 0 comments

mfreeman451 commented

2026-03-28 04:24:12 +00:00

Owner

Imported from GitHub.

Original GitHub issue: #1217
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1217
Original created: 2025-07-09T15:58:53Z

Jul 09 10:55:06 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:06 Prepared 2 status updates to send to Armis.
Jul 09 10:55:07 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:07 Armis bulk update response: [{"result":{},"status":202},{"result":{},"status":202}]
Jul 09 10:55:07 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:07 Successfully invoked the Armis device status updater.
Jul 09 10:55:07 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:07 gRPC call: /proto.KVService/PutMany attempt: 1 target: 127.0.0.1:50057 duration: 8.435482ms error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5815512 vs. 4194304)
Jul 09 10:55:08 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:08 gRPC call: /proto.KVService/PutMany attempt: 2 target: 127.0.0.1:50057 duration: 4.962105ms error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5815512 vs. 4194304)
Jul 09 10:55:10 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:10 gRPC call: /proto.KVService/PutMany attempt: 3 target: 127.0.0.1:50057 duration: 5.437687ms error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5815512 vs. 4194304)
Jul 09 10:55:10 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:10 Failed to write batch to KV for source armis: all 3 retries failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5815512 vs. 4194304)
Jul 09 10:55:10 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:10 Updated discovery cache with 3114 devices
Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 gRPC call: /grpc.health.v1.Health/Check Duration: 1.484µs Error:
Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 GetStatus called by poller for service 'sync' (type: 'grpc')
Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 Returning 3114 cached devices to the poller.
Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 gRPC call: /monitoring.AgentService/GetStatus Duration: 8.927031ms Error:
Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 GetStatus called by poller for service 'sync' (type: 'grpc')
Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 Returning 3114 cached devices to the poller.
Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 gRPC call: /monitoring.AgentService/GetStatus Duration: 7.819029ms Error:

Imported from GitHub. Original GitHub issue: #1217 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1217 Original created: 2025-07-09T15:58:53Z --- Jul 09 10:55:06 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:06 Prepared 2 status updates to send to Armis. Jul 09 10:55:07 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:07 Armis bulk update response: [{"result":{},"status":202},{"result":{},"status":202}] Jul 09 10:55:07 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:07 Successfully invoked the Armis device status updater. Jul 09 10:55:07 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:07 gRPC call: /proto.KVService/PutMany attempt: 1 target: 127.0.0.1:50057 duration: 8.435482ms error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5815512 vs. 4194304) Jul 09 10:55:08 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:08 gRPC call: /proto.KVService/PutMany attempt: 2 target: 127.0.0.1:50057 duration: 4.962105ms error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5815512 vs. 4194304) Jul 09 10:55:10 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:10 gRPC call: /proto.KVService/PutMany attempt: 3 target: 127.0.0.1:50057 duration: 5.437687ms error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5815512 vs. 4194304) Jul 09 10:55:10 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:10 Failed to write batch to KV for source armis: all 3 retries failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5815512 vs. 4194304) Jul 09 10:55:10 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:10 Updated discovery cache with 3114 devices Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 gRPC call: /grpc.health.v1.Health/Check Duration: 1.484µs Error: <nil> Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 GetStatus called by poller for service 'sync' (type: 'grpc') Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 Returning 3114 cached devices to the poller. Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 gRPC call: /monitoring.AgentService/GetStatus Duration: 8.927031ms Error: <nil> Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 GetStatus called by poller for service 'sync' (type: 'grpc') Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 Returning 3114 cached devices to the poller. Jul 09 10:55:32 foobar serviceradar-sync[3936368]: 2025/07/09 10:55:32 gRPC call: /monitoring.AgentService/GetStatus Duration: 7.819029ms Error: <nil>

Rows
Columns