Loading...

Type: Bug
Resolution: Unresolved
Priority: Undefined
Fix Version/s: None
Affects Version/s: 4.20.0
Component/s: openshift-controller-manager / controller-manager
Labels:
None

Activity Type:
Quality / Stability / Reliability
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Story Points:
None
Severity:
Moderate
Regression:
None

Target Backport Versions:
None
Target Version:
None
Release Blocker:
None
Sprint:
None

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

PX Impact Score:

Release Note Status:
None
Release Note Type:
None
Release Note Text:
None

Escape Reason:
None
Escape Impact:
None
Corrective Measures:
None
SDLC stage when should've been found:
None

Recently, we observed panic from kube-apiserver audit logs,

"containers": [
             {
               "name": "controller-manager",
               "image": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff225b25728a996f156a93e3f389ae4c3f0f85606a436d070c5a547559d14207",
               "command": [
                 "openshift-controller-manager",
                 "start"
               ],
               "args": [
                 "--config=/var/run/configmaps/config/config.yaml",
                 "-v=2"
               ],
               "ports": [
                 {
                   "containerPort": 8443,
                   "protocol": "TCP"
                 }
               ],
    ...
    "startTime": "2025-08-08T08:47:16Z",
           "containerStatuses": [
             {
               "name": "controller-manager",
               "state": {
                 "terminated": {
                   "exitCode": 0,
                   "reason": "Completed",
                   "startedAt": "2025-08-08T08:48:02Z",
                   "finishedAt": "2025-08-08T09:16:26Z",
                   "containerID": "cri-o://6fceef446287823e4d9e06913907be8aca8bbb2a279c78f3d1cb3c1c85b549fb"
                 }
               },
               "lastState": {
                 "terminated": {
                   "exitCode": 2,
                   "reason": "Error",
                   "message": "08 08:48:01.868632       1 reflector.go:430] \"Caches populated\" type=\"*v1.DeploymentConfig\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\nI0808 08:48:01.871065       1 reflector.go:430] \"Caches populated\" type=\"*v1.RoleBinding\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\nI0808 08:48:01.875101       1 reflector.go:430] \"Caches populated\" type=\"*v1.Deployment\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\nI0808 08:48:01.875341       1 buildconfig_controller.go:212] Starting buildconfig controller\nI0808 08:48:01.875478       1 warnings.go:110] \"Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+\"\nI0808 08:48:01.875616       1 reflector.go:430] \"Caches populated\" type=\"*v1.Build\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\nI0808 08:48:01.882434       1 reflector.go:430] \"Caches populated\" type=\"*v1.Image\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\nI0808 08:48:01.890204       1 reflector.go:430] \"Caches populated\" type=\"*v1.ImageStream\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\nI0808 08:48:01.909421       1 reflector.go:430] \"Caches populated\" type=\"*v1.Image\" reflector=\"k8s.io/client-go@v0.33.2/tools/cache/reflector.go:285\"\npanic: runtime error: invalid memory address or nil pointer dereference\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x29b38ad]\n\ngoroutine 743 [running]:\ngithub.com/openshift/openshift-controller-manager/pkg/image/metrics/prometheus.(*importStatusCollector).Collect(0x5c7d8a0, 0xc00002c2a0)\n\tgithub.com/openshift/openshift-controller-manager/pkg/image/metrics/prometheus/metrics.go:113 +0x2d\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()\n\tgithub.com/prometheus/client_golang@v1.22.0/prometheus/registry.go:456 +0x105\ncreated by github.com/prometheus/client_golang/prometheus.(*Registry).Gather in goroutine 742\n\tgithub.com/prometheus/client_golang@v1.22.0/prometheus/registry.go:467 +0x5b4\n",
                   "startedAt": "2025-08-08T08:47:17Z",
                   "finishedAt": "2025-08-08T08:48:01Z",
                   "containerID": "cri-o://e85b7efb545e0983f4ea257cacf81370764093445371d0664d34fc50cabbeeb5"
                 }
               },
               "ready": false,
               "restartCount": 1,
               "image": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff225b25728a996f156a93e3f389ae4c3f0f85606a436d070c5a547559d14207",
               "imageID": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff225b25728a996f156a93e3f389ae4c3f0f85606a436d070c5a547559d14207",
               "containerID": "cri-o://6fceef446287823e4d9e06913907be8aca8bbb2a279c78f3d1cb3c1c85b549fb",
               "started": false,
               "allocatedResources": {
                 "cpu": "100m",
                 "memory": "100Mi"
               },

openshift-controller-manager crashes with a nil pointer dereference during Prometheus metrics collection in importStatusCollector.Collect().
The crash occurs because the callback function cbCollectISCounts is not initialized before being invoked at metrics.go:113: https://github.com/openshift/openshift-controller-manager/blob/master/pkg/image/metrics/prometheus/metrics.go#L113

[Affects: ]

Component: openshift-controller-manager (image import metrics)

Functionality: Prometheus metrics scraping (/metrics endpoint)

Impact:

Controller Manager panics → Pod restarts (depending on deployment strategy).

Image import metrics are unavailable.

May disrupt ImageStream imports and degrade cluster stability.

Versions: Any OpenShift build where importStatusCollector is initialized without assigning cbCollectISCounts. (Seen with k8s v0.33.2 libraries + openshift-controller-manager).

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates