bug: flakey tests #447

Closed
opened 2026-03-28 04:24:39 +00:00 by mfreeman451 · 1 comment
Owner

Imported from GitHub.

Original GitHub issue: #1360
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1360
Original created: 2025-07-26T07:53:46Z


-test.shuffle 1753515890228528000
2025/07/26 02:44:50 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping
2025/07/26 02:44:50 Recovery alert for poller test-poller rate limited, but poller marked as recovered
2025/07/26 02:44:52 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping
2025/07/26 02:44:52 Recovery alert for poller test-poller rate limited, but poller marked as recovered
2025/07/26 02:44:52 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping
2025/07/26 02:44:52 Recovery alert for poller test-poller rate limited, but poller marked as recovered
2025/07/26 02:44:53 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping
2025/07/26 02:44:53 Recovery alert for poller test-poller rate limited, but poller marked as recovered
2025/07/26 02:44:53 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping
2025/07/26 02:44:53 Recovery alert for poller test-poller rate limited, but poller marked as recovered
2025/07/26 02:44:54 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping
coverage: 3.5% of statements in ./...
2025/07/26 02:44:54 Recovery alert for poller test-poller rate limited, but poller marked as recovered
2025/07/26 02:44:55 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping
2025/07/26 02:44:55 Recovery alert for poller test-poller rate limited, but poller marked as recovered
2025/07/26 02:44:56 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping
2025/07/26 02:44:57 Recovery alert for poller test-poller rate limited, but poller marked as recovered
2025/07/26 02:44:57 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping
2025/07/26 02:44:57 Recovery alert for poller test-poller rate limited, but poller marked as recovered
2025/07/26 02:44:58 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping
2025/07/26 02:44:58 Recovery alert for poller test-poller rate limited, but poller marked as recovered
PASS
panic: test timed out after 3s

goroutine 35 [running]:
testing.(*M).startAlarm.func1()
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2484 +0x4d8
created by time.goFunc
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/time/sleep.go:215 +0x44

goroutine 1 [runnable]:
internal/sync.runtime_SemacquireMutex(0xc0000478b8?, 0xc?, 0xc0000478e8?)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/runtime/sema.go:95 +0x28
internal/sync.(*Mutex).lockSlow(0xc000310b1c)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/internal/sync/mutex.go:149 +0x1e4
internal/sync.(*Mutex).Lock(0xc000310b1c)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/internal/sync/mutex.go:70 +0x5c
sync.(*Mutex).Lock(0xc000310b1c)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/mutex.go:46 +0x30
sync.(*Once).doSlow(0xc000310b18, 0xc000047a28)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/once.go:74 +0x40
sync.(*Once).Do(0xc000310b18, 0xc000047a28)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/once.go:69 +0x44
testing.(*M).after(0xc000310aa0)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2366 +0x5c
testing.(*M).Run(0xc000310aa0)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2185 +0x128c
main.main()
	_testmain.go:127 +0x114
FAIL	github.com/carverauto/serviceradar/pkg/core	12.067s
	github.com/carverauto/serviceradar/pkg/core/alerts		coverage: 0.0% of statements
ok  	github.com/carverauto/serviceradar/pkg/core/api	12.603s	coverage: 2.7% of statements in ./...
-test.shuffle 1753515892411088000
coverage: 0.3% of statements in ./...
PASS
panic: test timed out after 3s

goroutine 81 [running]:
testing.(*M).startAlarm.func1()
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2484 +0x4d8
created by time.goFunc
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/time/sleep.go:215 +0x44

goroutine 1 [runnable]:
internal/sync.runtime_SemacquireMutex(0xc0002c98b8?, 0xbc?, 0xc0002c98e8?)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/runtime/sema.go:95 +0x28
internal/sync.(*Mutex).lockSlow(0xc000157a1c)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/internal/sync/mutex.go:149 +0x1e4
internal/sync.(*Mutex).Lock(0xc000157a1c)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/internal/sync/mutex.go:70 +0x5c
sync.(*Mutex).Lock(0xc000157a1c)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/mutex.go:46 +0x30
sync.(*Once).doSlow(0xc000157a18, 0xc0000a1a28)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/once.go:74 +0x40
sync.(*Once).Do(0xc000157a18, 0xc0002c9a28)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/once.go:69 +0x44
testing.(*M).after(0xc0001579a0)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2366 +0x5c
testing.(*M).Run(0xc0001579a0)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2185 +0x128c
main.main()
	_testmain.go:65 +0x114
FAIL	github.com/carverauto/serviceradar/pkg/core/auth	15.999s
	github.com/carverauto/serviceradar/pkg/db		coverage: 0.0% of statements
-test.shuffle 1753515894815661000
coverage: 0.6% of statements in ./...
PASS
panic: test timed out after 3s

goroutine 104 [running]:
testing.(*M).startAlarm.func1()
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2484 +0x4d8
created by time.goFunc
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/time/sleep.go:215 +0x44

goroutine 1 [runnable]:
internal/sync.runtime_SemacquireMutex(0xc0000478b8?, 0xcc?, 0xc0000478e8?)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/runtime/sema.go:95 +0x28
internal/sync.(*Mutex).lockSlow(0xc00021c43c)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/internal/sync/mutex.go:149 +0x1e4
internal/sync.(*Mutex).Lock(0xc00021c43c)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/internal/sync/mutex.go:70 +0x5c
sync.(*Mutex).Lock(0xc00021c43c)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/mutex.go:46 +0x30
sync.(*Once).doSlow(0xc00021c438, 0xc0001dda28)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/once.go:74 +0x40
sync.(*Once).Do(0xc00021c438, 0xc000047a28)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/once.go:69 +0x44
testing.(*M).after(0xc00021c3c0)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2366 +0x5c
testing.(*M).Run(0xc00021c3c0)
	/opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2185 +0x128c
main.main()
	_testmain.go:63 +0x114
FAIL	github.com/carverauto/serviceradar/pkg/grpc	12.597s
ok  	github.com/carverauto/serviceradar/pkg/http	9.732s	coverage: 0.1% of statements in ./...
ok  	github.com/carverauto/serviceradar/pkg/integration_test	9.789s	coverage: 0.3% of statements in ./...
ok  	github.com/carverauto/serviceradar/pkg/kv	9.962s	coverage: 0.4% of statements in ./...
	github.com/carverauto/serviceradar/pkg/lifecycle		coverage: 0.0% of statements
FAIL
make: *** [test] Error 1
Imported from GitHub. Original GitHub issue: #1360 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1360 Original created: 2025-07-26T07:53:46Z --- ``` -test.shuffle 1753515890228528000 2025/07/26 02:44:50 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping 2025/07/26 02:44:50 Recovery alert for poller test-poller rate limited, but poller marked as recovered 2025/07/26 02:44:52 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping 2025/07/26 02:44:52 Recovery alert for poller test-poller rate limited, but poller marked as recovered 2025/07/26 02:44:52 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping 2025/07/26 02:44:52 Recovery alert for poller test-poller rate limited, but poller marked as recovered 2025/07/26 02:44:53 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping 2025/07/26 02:44:53 Recovery alert for poller test-poller rate limited, but poller marked as recovered 2025/07/26 02:44:53 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping 2025/07/26 02:44:53 Recovery alert for poller test-poller rate limited, but poller marked as recovered 2025/07/26 02:44:54 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping coverage: 3.5% of statements in ./... 2025/07/26 02:44:54 Recovery alert for poller test-poller rate limited, but poller marked as recovered 2025/07/26 02:44:55 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping 2025/07/26 02:44:55 Recovery alert for poller test-poller rate limited, but poller marked as recovered 2025/07/26 02:44:56 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping 2025/07/26 02:44:57 Recovery alert for poller test-poller rate limited, but poller marked as recovered 2025/07/26 02:44:57 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping 2025/07/26 02:44:57 Recovery alert for poller test-poller rate limited, but poller marked as recovered 2025/07/26 02:44:58 Alert 'Service Failure' for node 'test-poller' is within cooldown period, skipping 2025/07/26 02:44:58 Recovery alert for poller test-poller rate limited, but poller marked as recovered PASS panic: test timed out after 3s goroutine 35 [running]: testing.(*M).startAlarm.func1() /opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2484 +0x4d8 created by time.goFunc /opt/homebrew/Cellar/go/1.24.1/libexec/src/time/sleep.go:215 +0x44 goroutine 1 [runnable]: internal/sync.runtime_SemacquireMutex(0xc0000478b8?, 0xc?, 0xc0000478e8?) /opt/homebrew/Cellar/go/1.24.1/libexec/src/runtime/sema.go:95 +0x28 internal/sync.(*Mutex).lockSlow(0xc000310b1c) /opt/homebrew/Cellar/go/1.24.1/libexec/src/internal/sync/mutex.go:149 +0x1e4 internal/sync.(*Mutex).Lock(0xc000310b1c) /opt/homebrew/Cellar/go/1.24.1/libexec/src/internal/sync/mutex.go:70 +0x5c sync.(*Mutex).Lock(0xc000310b1c) /opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/mutex.go:46 +0x30 sync.(*Once).doSlow(0xc000310b18, 0xc000047a28) /opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/once.go:74 +0x40 sync.(*Once).Do(0xc000310b18, 0xc000047a28) /opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/once.go:69 +0x44 testing.(*M).after(0xc000310aa0) /opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2366 +0x5c testing.(*M).Run(0xc000310aa0) /opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2185 +0x128c main.main() _testmain.go:127 +0x114 FAIL github.com/carverauto/serviceradar/pkg/core 12.067s github.com/carverauto/serviceradar/pkg/core/alerts coverage: 0.0% of statements ok github.com/carverauto/serviceradar/pkg/core/api 12.603s coverage: 2.7% of statements in ./... -test.shuffle 1753515892411088000 coverage: 0.3% of statements in ./... PASS panic: test timed out after 3s goroutine 81 [running]: testing.(*M).startAlarm.func1() /opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2484 +0x4d8 created by time.goFunc /opt/homebrew/Cellar/go/1.24.1/libexec/src/time/sleep.go:215 +0x44 goroutine 1 [runnable]: internal/sync.runtime_SemacquireMutex(0xc0002c98b8?, 0xbc?, 0xc0002c98e8?) /opt/homebrew/Cellar/go/1.24.1/libexec/src/runtime/sema.go:95 +0x28 internal/sync.(*Mutex).lockSlow(0xc000157a1c) /opt/homebrew/Cellar/go/1.24.1/libexec/src/internal/sync/mutex.go:149 +0x1e4 internal/sync.(*Mutex).Lock(0xc000157a1c) /opt/homebrew/Cellar/go/1.24.1/libexec/src/internal/sync/mutex.go:70 +0x5c sync.(*Mutex).Lock(0xc000157a1c) /opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/mutex.go:46 +0x30 sync.(*Once).doSlow(0xc000157a18, 0xc0000a1a28) /opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/once.go:74 +0x40 sync.(*Once).Do(0xc000157a18, 0xc0002c9a28) /opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/once.go:69 +0x44 testing.(*M).after(0xc0001579a0) /opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2366 +0x5c testing.(*M).Run(0xc0001579a0) /opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2185 +0x128c main.main() _testmain.go:65 +0x114 FAIL github.com/carverauto/serviceradar/pkg/core/auth 15.999s github.com/carverauto/serviceradar/pkg/db coverage: 0.0% of statements -test.shuffle 1753515894815661000 coverage: 0.6% of statements in ./... PASS panic: test timed out after 3s goroutine 104 [running]: testing.(*M).startAlarm.func1() /opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2484 +0x4d8 created by time.goFunc /opt/homebrew/Cellar/go/1.24.1/libexec/src/time/sleep.go:215 +0x44 goroutine 1 [runnable]: internal/sync.runtime_SemacquireMutex(0xc0000478b8?, 0xcc?, 0xc0000478e8?) /opt/homebrew/Cellar/go/1.24.1/libexec/src/runtime/sema.go:95 +0x28 internal/sync.(*Mutex).lockSlow(0xc00021c43c) /opt/homebrew/Cellar/go/1.24.1/libexec/src/internal/sync/mutex.go:149 +0x1e4 internal/sync.(*Mutex).Lock(0xc00021c43c) /opt/homebrew/Cellar/go/1.24.1/libexec/src/internal/sync/mutex.go:70 +0x5c sync.(*Mutex).Lock(0xc00021c43c) /opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/mutex.go:46 +0x30 sync.(*Once).doSlow(0xc00021c438, 0xc0001dda28) /opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/once.go:74 +0x40 sync.(*Once).Do(0xc00021c438, 0xc000047a28) /opt/homebrew/Cellar/go/1.24.1/libexec/src/sync/once.go:69 +0x44 testing.(*M).after(0xc00021c3c0) /opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2366 +0x5c testing.(*M).Run(0xc00021c3c0) /opt/homebrew/Cellar/go/1.24.1/libexec/src/testing/testing.go:2185 +0x128c main.main() _testmain.go:63 +0x114 FAIL github.com/carverauto/serviceradar/pkg/grpc 12.597s ok github.com/carverauto/serviceradar/pkg/http 9.732s coverage: 0.1% of statements in ./... ok github.com/carverauto/serviceradar/pkg/integration_test 9.789s coverage: 0.3% of statements in ./... ok github.com/carverauto/serviceradar/pkg/kv 9.962s coverage: 0.4% of statements in ./... github.com/carverauto/serviceradar/pkg/lifecycle coverage: 0.0% of statements FAIL make: *** [test] Error 1 ```
Author
Owner

Imported GitHub comment.

Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/1360#issuecomment-3813958560
Original created: 2026-01-28T21:12:17Z


closing, stale

Imported GitHub comment. Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/1360#issuecomment-3813958560 Original created: 2026-01-28T21:12:17Z --- closing, stale
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
carverauto/serviceradar#447
No description provided.