Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-60635

openshift-controller-manager crashes with a nil pointer dereference during Prometheus metrics collection

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Recently, we observed panic from kube-apiserver audit logs,

      "containers": [
                   {
                     "name": "controller-manager",
                     "image": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff225b25728a996f156a93e3f389ae4c3f0f85606a436d070c5a547559d14207",
                     "command": [
                       "openshift-controller-manager",
                       "start"
                     ],
                     "args": [
                       "--config=/var/run/configmaps/config/config.yaml",
                       "-v=2"
                     ],
                     "ports": [
                       {
                         "containerPort": 8443,
                         "protocol": "TCP"
                       }
                     ],
          ...
          "startTime": "2025-08-08T08:47:16Z",
                 "containerStatuses": [
                   {
                     "name": "controller-manager",
                     "state": {
                       "terminated": {
                         "exitCode": 0,
                         "reason": "Completed",
                         "startedAt": "2025-08-08T08:48:02Z",
                         "finishedAt": "2025-08-08T09:16:26Z",
                         "containerID": "cri-o://6fceef446287823e4d9e06913907be8aca8bbb2a279c78f3d1cb3c1c85b549fb"
                       }
                     },
                     "lastState": {
                       "terminated": {
                         "exitCode": 2,
                         "reason": "Error",
                         "message": "08 08:48:01.868632       1 reflector.go:430] \"Caches populated\" type=\"*v1.DeploymentConfig\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\nI0808 08:48:01.871065       1 reflector.go:430] \"Caches populated\" type=\"*v1.RoleBinding\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\nI0808 08:48:01.875101       1 reflector.go:430] \"Caches populated\" type=\"*v1.Deployment\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\nI0808 08:48:01.875341       1 buildconfig_controller.go:212] Starting buildconfig controller\nI0808 08:48:01.875478       1 warnings.go:110] \"Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+\"\nI0808 08:48:01.875616       1 reflector.go:430] \"Caches populated\" type=\"*v1.Build\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\nI0808 08:48:01.882434       1 reflector.go:430] \"Caches populated\" type=\"*v1.Image\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\nI0808 08:48:01.890204       1 reflector.go:430] \"Caches populated\" type=\"*v1.ImageStream\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\nI0808 08:48:01.909421       1 reflector.go:430] \"Caches populated\" type=\"*v1.Image\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\npanic: runtime error: invalid memory address or nil pointer dereference\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x29b38ad]\n\ngoroutine 743 [running]:\ngithub.com/openshift/openshift-controller-manager/pkg/image/metrics/prometheus.(*importStatusCollector).Collect(0x5c7d8a0, 0xc00002c2a0)\n\tgithub.com/openshift/openshift-controller-manager/pkg/image/metrics/prometheus/metrics.go:113 +0x2d\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()\n\tgithub.com/prometheus/client_golang@v1.22.0/prometheus/registry.go:456 +0x105\ncreated by github.com/prometheus/client_golang/prometheus.(*Registry).Gather in goroutine 742\n\tgithub.com/prometheus/client_golang@v1.22.0/prometheus/registry.go:467 +0x5b4\n",
                         "startedAt": "2025-08-08T08:47:17Z",
                         "finishedAt": "2025-08-08T08:48:01Z",
                         "containerID": "cri-o://e85b7efb545e0983f4ea257cacf81370764093445371d0664d34fc50cabbeeb5"
                       }
                     },
                     "ready": false,
                     "restartCount": 1,
                     "image": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff225b25728a996f156a93e3f389ae4c3f0f85606a436d070c5a547559d14207",
                     "imageID": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff225b25728a996f156a93e3f389ae4c3f0f85606a436d070c5a547559d14207",
                     "containerID": "cri-o://6fceef446287823e4d9e06913907be8aca8bbb2a279c78f3d1cb3c1c85b549fb",
                     "started": false,
                     "allocatedResources": {
                       "cpu": "100m",
                       "memory": "100Mi"
                     },
      

      openshift-controller-manager crashes with a nil pointer dereference during Prometheus metrics collection in importStatusCollector.Collect().
      The crash occurs because the callback function cbCollectISCounts is not initialized before being invoked at metrics.go:113: https://github.com/openshift/openshift-controller-manager/blob/master/pkg/image/metrics/prometheus/metrics.go#L113

      [Affects: ]

      Component: openshift-controller-manager (image import metrics)

      Functionality: Prometheus metrics scraping (/metrics endpoint)

      Impact:

      Controller Manager panics → Pod restarts (depending on deployment strategy).

      Image import metrics are unavailable.

      May disrupt ImageStream imports and degrade cluster stability.

      Versions: Any OpenShift build where importStatusCollector is initialized without assigning cbCollectISCounts. (Seen with k8s v0.33.2 libraries + openshift-controller-manager).

              rhn-support-ppalepu Prabhakar Palepu
              wk2019 Ke Wang
              None
              None
              Jitendar Singh Jitendar Singh
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: