-
Story
-
Resolution: Done
-
Major
-
None
-
None
-
None
-
None
-
False
-
-
False
-
None
-
None
-
None
-
None
-
None
During recent investigation of TRT-1926 a panic was detected in the kubelet logs for the failing jobs
2024-12-20 08:52:36
fatal error: concurrent map iteration and map write
2024-12-20 08:52:36
goroutine 64863 gp=0xc000b69a40 m=21 mp=0xc002991808 [running]:
2024-12-20 08:52:36
runtime.fatal({0x55d57cd30f9d?, 0x0?})
2024-12-20 08:52:36
/usr/lib/golang/src/runtime/panic.go:1042 +0x5e fp=0xc0017c33c0 sp=0xc0017c3390 pc=0x55d57b5477be
2024-12-20 08:52:36
runtime.mapiternext(0xc0017c34b0)
2024-12-20 08:52:36
/usr/lib/golang/src/runtime/map.go:869 +0x3b fp=0xc0017c3430 sp=0xc0017c33c0 pc=0x55d57b51af5b
2024-12-20 08:52:36
github.com/cri-o/cri-o/vendor/k8s.io/cri-api/pkg/apis/runtime/v1.(*PodSandbox).MarshalToSizedBuffer(0xc00054e1e0, {0xc0034c4000, 0x3a670, 0x3e11d})
2024-12-20 08:52:36
/builddir/build/BUILD/cri-o-b031f890da99234425d52c3f1c5b171356cba404/_output/src/github.com/cri-o/cri-o/vendor/k8s.io/cri-api/pkg/apis/runtime/v1/api.pb.go:14002 +0x169 fp=0xc0017c3520 sp=0xc0017c3430 pc=0x55d57bc4b109
The panic was not detected as this was coming from kublet logs vs standard pod logs. We want to enhance the eventsFromKubeletLogs to detect panics and add a kubeletelogcollector monitor test to fail when detected.
- duplicates
-
OCPBUGS-48186 A monitor test should catch a panic in a system process
-
- POST
-