Loading...

XML

Word

Printable

Type: Bug
Resolution: Duplicate
Priority: Major
Fix Version/s: None
Affects Version/s: 4.17
Component/s: Image Registry
Labels:
None

Activity Type:
Quality / Stability / Reliability
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Story Points:
None
Severity:
None
Regression:
Yes

Target Backport Versions:
None
Target Version:
None
Release Blocker:
Approved
Sprint:
None

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Release Note Status:
None
Release Note Type:
None
Release Note Text:
None

Escape Reason:
None
Escape Impact:
None
Corrective Measures:
None
SDLC stage when should've been found:
None

Description of problem:

    saw this failure:

: Undiagnosed panic detected in pod expand_less0s{  pods/openshift-controller-manager-operator_openshift-controller-manager-operator-5b7798f5d9-85fhj_openshift-controller-manager-operator.log.gz:E0528 15:05:37.475672       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)}

in https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-techpreview/1795464246306279424

most recently seen on this payload: https://amd64.ocp.releases.ci.openshift.org/releasestream/4.17.0-0.nightly/release/4.17.0-0.nightly-2024-05-28-143311

the first payload where this panic was observed was in: https://sippy.dptools.openshift.org/sippy-ng/release/4.17/tags/4.17.0-0.nightly-2024-05-24-115812/pull_requests

It's an informing job so it didn't fail the payload.

024-05-28T15:05:37.476151619Z E0528 15:05:37.475672       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
2024-05-28T15:05:37.476151619Z goroutine 1055 [running]:
2024-05-28T15:05:37.476151619Z k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2598ea0?, 0x4514fb0})
2024-05-28T15:05:37.476151619Z     k8s.io/apimachinery@v0.29.0/pkg/util/runtime/runtime.go:75 +0x85
2024-05-28T15:05:37.476151619Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0xc0011456a0, 0x1, 0x23e83e0?})
2024-05-28T15:05:37.476151619Z     k8s.io/apimachinery@v0.29.0/pkg/util/runtime/runtime.go:49 +0x6b
2024-05-28T15:05:37.476151619Z panic({0x2598ea0?, 0x4514fb0?})
2024-05-28T15:05:37.476151619Z     runtime/panic.go:914 +0x21f
2024-05-28T15:05:37.476151619Z github.com/openshift/cluster-openshift-controller-manager-operator/pkg/operator/internalimageregistry.(*imagePullSecretCleanupController).cleanup(0xc000389420, {0x2eac7b0, 0xc0002ed4a0})
2024-05-28T15:05:37.476151619Z     github.com/openshift/cluster-openshift-controller-manager-operator/pkg/operator/internalimageregistry/cleanup_controller.go:146 +0xdce
2024-05-28T15:05:37.476151619Z github.com/openshift/cluster-openshift-controller-manager-operator/pkg/operator/internalimageregistry.(*imagePullSecretCleanupController).sync(0xc000389420, {0x2eac7b0, 0xc0002ed4a0}, {0x7f16fc259de8?, 0x7f174768ef18?})
2024-05-28T15:05:37.476151619Z     github.com/openshift/cluster-openshift-controller-manager-operator/pkg/operator/internalimageregistry/cleanup_controller.go:67 +0x67
2024-05-28T15:05:37.476151619Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).reconcile(0xc000e9ccf0, {0x2eac7b0, 0xc0002ed4a0}, {0x2e9d120?, 0xc0025f1890?})
2024-05-28T15:05:37.476151619Z     github.com/openshift/library-go@v0.0.0-20240116081341-964bcb3f545c/pkg/controller/factory/base_controller.go:201 +0x43
2024-05-28T15:05:37.476151619Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).processNextWorkItem(0xc000e9ccf0, {0x2eac7b0, 0xc0002ed4a0})
2024-05-28T15:05:37.476151619Z     github.com/openshift/library-go@v0.0.0-20240116081341-964bcb3f545c/pkg/controller/factory/base_controller.go:260 +0x1b4
2024-05-28T15:05:37.476151619Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker.func1({0x2eac7b0, 0xc0002ed4a0})
2024-05-28T15:05:37.476151619Z     github.com/openshift/library-go@v0.0.0-20240116081341-964bcb3f545c/pkg/controller/factory/base_controller.go:192 +0x89
2024-05-28T15:05:37.476151619Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
2024-05-28T15:05:37.476151619Z     k8s.io/apimachinery@v0.29.0/pkg/util/wait/backoff.go:259 +0x22
2024-05-28T15:05:37.476151619Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
2024-05-28T15:05:37.476151619Z     k8s.io/apimachinery@v0.29.0/pkg/util/wait/backoff.go:226 +0x33
2024-05-28T15:05:37.476151619Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0016f9680?, {0x2e7f560, 0xc003169740}, 0x1, 0xc0016f9680)
2024-05-28T15:05:37.476151619Z     k8s.io/apimachinery@v0.29.0/pkg/util/wait/backoff.go:227 +0xaf
2024-05-28T15:05:37.476151619Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0002ed4a0?, 0x3b9aca00, 0x0, 0x38?, 0xb221df?)
2024-05-28T15:05:37.476151619Z     k8s.io/apimachinery@v0.29.0/pkg/util/wait/backoff.go:204 +0x7f
2024-05-28T15:05:37.476151619Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x2eac7b0, 0xc0002ed4a0}, 0xc000e47770, 0x0?, 0x0?, 0x0?)
2024-05-28T15:05:37.476151619Z     k8s.io/apimachinery@v0.29.0/pkg/util/wait/backoff.go:259 +0x93
2024-05-28T15:05:37.476151619Z k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
2024-05-28T15:05:37.476151619Z     k8s.io/apimachinery@v0.29.0/pkg/util/wait/backoff.go:170
2024-05-28T15:05:37.476151619Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker(0xc003150600?, {0x2eac7b0?, 0xc0002ed4a0?})
2024-05-28T15:05:37.476151619Z     github.com/openshift/library-go@v0.0.0-20240116081341-964bcb3f545c/pkg/controller/factory/base_controller.go:183 +0x4d
2024-05-28T15:05:37.476151619Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).Run.func2()
2024-05-28T15:05:37.476151619Z     github.com/openshift/library-go@v0.0.0-20240116081341-964bcb3f545c/pkg/controller/factory/base_controller.go:117 +0x65
2024-05-28T15:05:37.476151619Z created by github.com/openshift/library-go/pkg/controller/factory.(*baseController).Run in goroutine 508
2024-05-28T15:05:37.476151619Z     github.com/openshift/library-go@v0.0.0-20240116081341-964bcb3f545c/pkg/controller/factory/base_controller.go:112 +0x2ba
2024-05-28T15:05:37.476151619Z I0528 15:05:37.476031       1 event.go:364] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-controller-manager-operator", Name:"openshift-controller-manager-operator", UID:"03e1b2be-8139-42d7-8992-89ae2e210b21", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'Openshift-Controller-Manager-OperatorPanic' Panic observed: runtime error: invalid memory address or nil pointer dereference

Version-Release number of selected component (if applicable):

    4.17

How reproducible:

    intermittent

Steps to Reproduce:

    1.
    2.
    3.

Actual results:

Expected results:

Additional info:

    I saw these jobs with similar failure (seems starting in may 24, 2024):

 2024-05-28 17:05:22.795177+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-telco5g/1795464272298381312
 2024-05-28 17:04:03.835375+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-techpreview/1795464246306279424
 2024-05-28 14:04:12.578335+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-agent-compact-ipv4-conformance/1795425218701824000
 2024-05-28 12:08:41.48392+00  | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-techpreview-serial/1795372046302056448
 2024-05-28 11:05:58.137624+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-agent-compact-ipv4-conformance/1795372088249290752
 2024-05-28 03:04:41.918835+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-techpreview-serial/1795233952940691456
 2024-05-28 03:03:59.720205+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-metal-ipi-ovn-serial-ipv4/1795234088836141056
 2024-05-28 01:04:04.834652+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-csi/1795233955457273856
 2024-05-28 00:02:55.235784+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-multiarch-master-nightly-4.17-ocp-e2e-ovn-remote-libvirt-s390x-heterogeneous/1795183484998782976
 2024-05-27 21:03:22.977843+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-techpreview-serial/1795142371764080640
 2024-05-27 20:03:01.457348+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-techpreview/1795151150786285568
 2024-05-27 15:05:10.928355+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-agent-compact-ipv4-conformance/1795062889120272384
 2024-05-27 12:04:12.989457+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-metal-ipi-ovn-serial-virtualmedia/1794995421739749376
 2024-05-27 12:03:23.650014+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn/1795025475114897408
 2024-05-27 10:03:31.344591+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-metal-ipi-ovn-bm/1794995073503465472
 2024-05-27 01:03:38.960364+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-serial/1794855540359172096
 2024-05-27 00:03:14.582971+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-csi/1794855432133545984
 2024-05-26 15:03:39.528855+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-agent-compact-ipv4-conformance/1794721718489583616
 2024-05-26 05:03:50.797444+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-techpreview-serial/1794547374559858688
 2024-05-26 05:03:22.322221+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-agent-ha-dualstack-conformance/1794547433288503296
 2024-05-26 02:04:18.221637+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn/1794516903792742400
 2024-05-25 22:03:43.152318+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-csi/1794455588655075328
 2024-05-25 14:03:43.738249+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-techpreview-serial/1794321127166709760
 2024-05-24 17:05:02.370531+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-agent-ha-dualstack-conformance/1793975788505141248
 2024-05-24 15:05:11.480878+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-metal-ipi-ovn-bm/1793976062107979776
 2024-05-24 15:03:05.545413+00 | https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn/1793975910169317376

links to

openshift/cluster-openshift-controller-manager-operator#369: wip: OCPBUGS-34610: test: image registry capability disabled but image registry installed

Assignee:: Flavian Missi

Reporter:: Dennis Periquet

Need Info From:: None

Contributors:: None

QA Contact:: Jitendar Singh

Doc Contact:: None

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Created:: 2024/05/29 4:33 PM

Updated:: 2025/07/22 5:38 PM

Resolved:: 2024/06/25 8:32 AM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates