-
Bug
-
Resolution: Done
-
Normal
-
None
-
4.13
-
None
-
Moderate
-
None
-
1
-
Workloads Sprint 258
-
1
-
False
-
The 3 pods for kube-controller-manager container keeps restarting with error code 2, referencing different go files/scripts,
$ for pod in `oc get pods -o custom-columns=POD:.metadata.name --no-headers -l app=kube-controller-manager -n openshift-kube-controller-manager`; do echo $pod; oc describe pod $pod | grep -A 8 "Last State"; echo " "; done
kube-controller-manager-ip-172-24-0-154.ec2.internal
Last State: Terminated
Reason: Error
Message: ync.(*Cond).Wait(0x553b3b8?)
/usr/lib/golang/src/sync/cond.go:70 +0x8c <<<<<<<<<<<<<<<
golang.org/x/net/http2.(*pipe).Read(0xc0061941b0,
vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc00a3a6300?, 0xc00a70b508?, 0x40b6f6?})
vendor/golang.org/x/net/http2/transport.go:2507 +0x75
io.ReadAtLeast({0x7f437f7d30f0, 0xc006194180}, {0xc00a3a6300, 0x4, 0x4}
, 0x4)
kube-controller-manager-ip-172-24-0-181.ec2.internal
Last State: Terminated
Reason: Error
Message: pointslice/endpointslice_controller.go:265 +0x32c <<<<<<<<<<<<<<
goroutine 47375 [running]:
k8s.io/apimachinery/pkg/util/sets.Set[...].Has(0xc002c421b0?,
)
vendor/k8s.io/apimachinery/pkg/util/sets/set.go:69 +0x25
k8s.io/kubernetes/pkg/controller/endpointslice/topologycache.(*TopologyCache).HasPopulatedHints(...)
kube-controller-manager-ip-172-24-0-210.ec2.internal
Last State: Terminated
Reason: Error
Message: c007a87ef0,
)
vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:76 +0x88 <<<<<<<<<
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc00a53e500, 0xc00add02d0?,
)
vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00819
==============
Doing oc logs -p, "fatal error: concurrent map read and map write" pop up right before the goroutine errors,
kube-controller-manager-ip-172-24-0-181.ec2.internal
2024-07-27T22:09:08.942084342Z fatal error: concurrent map read and map write
2024-07-27T22:09:08.942249300Z fatal error: concurrent map read and map write
2024-07-27T22:09:08.942275654Z fatal error: concurrent map read and map write
2024-07-27T22:09:08.949719129Z
2024-07-27T22:09:08.949719129Z goroutine 47699 [running]:
2024-07-27T22:09:08.949719129Z k8s.io/apimachinery/pkg/util/sets.Set[...].Has(0xc002930f30?,
)
2024-07-27T22:09:08.949743995Z vendor/k8s.io/apimachinery/pkg/util/sets/set.go:69 +0x25
2024-07-27T22:09:08.949743995Z k8s.io/kubernetes/pkg/controller/endpointslice/topologycache.(*TopologyCache).HasPopulatedHints(...)