Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-37739

The kube-controller-manager container keeps restarting pointing to errors in different go files, with this warning "fatal error: concurrent map read and map write"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • 4.13
    • None
    • Moderate
    • None
    • 1
    • Workloads Sprint 258
    • 1
    • False
    • Hide

      None

      Show
      None

      The 3 pods for kube-controller-manager container keeps restarting with error code 2, referencing different go files/scripts,

      $ for pod in `oc get pods -o custom-columns=POD:.metadata.name --no-headers -l app=kube-controller-manager -n openshift-kube-controller-manager`; do echo $pod; oc describe pod $pod | grep -A 8 "Last State"; echo " "; done
      kube-controller-manager-ip-172-24-0-154.ec2.internal
      Last State: Terminated
      Reason: Error
      Message: ync.(*Cond).Wait(0x553b3b8?)
      /usr/lib/golang/src/sync/cond.go:70 +0x8c <<<<<<<<<<<<<<<
      golang.org/x/net/http2.(*pipe).Read(0xc0061941b0,

      {0xc00a3a6300, 0x4, 0x4})
      vendor/golang.org/x/net/http2/pipe.go:76 +0xeb
      golang.org/x/net/http2.transportResponseBody.Read({0x0?}, {0xc00a3a6300?, 0xc00a70b508?, 0x40b6f6?})
      vendor/golang.org/x/net/http2/transport.go:2507 +0x75
      io.ReadAtLeast({0x7f437f7d30f0, 0xc006194180}, {0xc00a3a6300, 0x4, 0x4}

      , 0x4)

      kube-controller-manager-ip-172-24-0-181.ec2.internal
      Last State: Terminated
      Reason: Error
      Message: pointslice/endpointslice_controller.go:265 +0x32c <<<<<<<<<<<<<<

      goroutine 47375 [running]:
      k8s.io/apimachinery/pkg/util/sets.Set[...].Has(0xc002c421b0?,

      {0xc0059d05d0?, 0x2a?}

      )
      vendor/k8s.io/apimachinery/pkg/util/sets/set.go:69 +0x25
      k8s.io/kubernetes/pkg/controller/endpointslice/topologycache.(*TopologyCache).HasPopulatedHints(...)

      kube-controller-manager-ip-172-24-0-210.ec2.internal
      Last State: Terminated
      Reason: Error
      Message: c007a87ef0,

      {0xc0072da800, 0x400, 0x400}

      )
      vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:76 +0x88 <<<<<<<<<
      k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc00a53e500, 0xc00add02d0?,

      {0x552beb8, 0xc00d7bac40}

      )

      vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
      k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00819
      ==============

      Doing oc logs -p, "fatal error: concurrent map read and map write" pop up right before the goroutine errors,

      kube-controller-manager-ip-172-24-0-181.ec2.internal
      2024-07-27T22:09:08.942084342Z fatal error: concurrent map read and map write
      2024-07-27T22:09:08.942249300Z fatal error: concurrent map read and map write
      2024-07-27T22:09:08.942275654Z fatal error: concurrent map read and map write
      2024-07-27T22:09:08.949719129Z
      2024-07-27T22:09:08.949719129Z goroutine 47699 [running]:
      2024-07-27T22:09:08.949719129Z k8s.io/apimachinery/pkg/util/sets.Set[...].Has(0xc002930f30?,

      {0xc0051a0e10?, 0x2a?}

      )
      2024-07-27T22:09:08.949743995Z vendor/k8s.io/apimachinery/pkg/util/sets/set.go:69 +0x25
      2024-07-27T22:09:08.949743995Z k8s.io/kubernetes/pkg/controller/endpointslice/topologycache.(*TopologyCache).HasPopulatedHints(...)

              jchaloup@redhat.com Jan Chaloupka
              rhn-support-nestoros Nikkie Estorosos
              David Darrah David Darrah (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: