-
Bug
-
Resolution: Done
-
Major
-
None
-
4.13, 4.12
-
None
-
Critical
-
No
-
Proposed
-
False
-
Description of problem:
$ oc patch certmanager/cluster --type=merge -p ' spec: cainjectorConfig: overrideArgs: - --v=2 logLevel: Trace webhookConfig: overrideArgs: - --v=4 '
Then observed cert-manager-operator pod panics.
Version-Release number of selected component (if applicable):
v1.11.1-6 in 4.13.0-0.nightly-2023-05-25-001936
How reproducible:
Tried two clusters, both hit.
Steps to Reproduce:
1. $ oc patch certmanager/cluster --type=merge -p ' spec: cainjectorConfig: overrideArgs: - --v=2 logLevel: Trace webhookConfig: overrideArgs: - --v=4 ' 2. Check cert-manager 3 pods, only cert-manager pod is renewed: $ oc get po -n cert-manager NAME READY STATUS RESTARTS AGE cert-manager-6884bb8c8-fqj64 1/1 Running 0 16s cert-manager-cainjector-5c4c4b59b-rhgsg 1/1 Running 0 10m51s cert-manager-webhook-5cfcfdb665-fj4ln 1/1 Running 0 20m54s 3. Check cert-manager-operator pod logs, found panic: ... 2023-05-26T08:52:06.742816737Z E0526 08:52:06.742494 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) 2023-05-26T08:52:06.742816737Z goroutine 666 [running]: 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/runtime.logPanic({0x22b8d80?, 0x3f82000}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0xc0010635f8, 0x1, 0x21978a0?}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75 2023-05-26T08:52:06.742816737Z panic({0x22b8d80, 0x3f82000}) 2023-05-26T08:52:06.742816737Z /usr/lib/golang/src/runtime/panic.go:884 +0x212 2023-05-26T08:52:06.742816737Z github.com/openshift/cert-manager-operator/pkg/controller/deployment.withPodLabelsValidateHook.func2(0x3f305e0?, 0xc0013b4500?) 2023-05-26T08:52:06.742816737Z /remote-source/app/pkg/controller/deployment/deployment_overrides_validation.go:182 +0x21b 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/operator/deploymentcontroller.(*DeploymentController).getDeployment(0xc000cec000, 0xc00127a3c0?) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/operator/deploymentcontroller/deployment_controller.go:226 +0x1ce 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/operator/deploymentcontroller.(*DeploymentController).syncManaged(0xc000cec000, {0x2ad2b70, 0xc00127a3c0}, 0x0?, 0xc000d421a0, {0x2ad1560, 0xc000bb98c0}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/operator/deploymentcontroller/deployment_controller.go:138 +0xde 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/operator/deploymentcontroller.(*DeploymentController).sync(0xc000cec000, {0x2ad2b70, 0xc00127a3c0}, {0x2ad1560, 0xc000bb98c0}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/operator/deploymentcontroller/deployment_controller.go:127 +0x15d 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).reconcile(0xc000cec7e0, {0x2ad2b70, 0xc00127a3c0}, {0x2ad1560?, 0xc000bb98c0?}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:205 +0x47 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).processNextWorkItem(0xc000cec7e0, {0x2ad2b70, 0xc00127a3c0}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:264 +0x23e 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker.func1({0x2ad2b70, 0xc00127a3c0}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:196 +0x96 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1() 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:190 +0x25 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x3feedb2?) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0008d0720?, {0x2aac600, 0xc000bb9890}, 0x1, 0xc0008d0720) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00127a3c0?, 0x3b9aca00, 0x0, 0xb0?, 0xc000065e60?) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x2ad2b70, 0xc00127a3c0}, 0xc000c9b770, 0xc000c9b7d0?, 0xb7e3f1?, 0xb0?) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:190 +0x99 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:101 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker(0x1?, {0x2ad2b70?, 0xc00127a3c0?}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:187 +0x53 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).Run.func2() 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:117 +0x72 2023-05-26T08:52:06.742816737Z created by github.com/openshift/library-go/pkg/controller/factory.(*baseController).Run 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:112 +0x297 2023-05-26T08:52:06.742816737Z I0526 08:52:06.742586 1 base_controller.go:73] Caches are synced for cert-manager-cainjector-static-resources- 2023-05-26T08:52:06.742816737Z I0526 08:52:06.742593 1 base_controller.go:110] Starting #1 worker of cert-manager-cainjector-static-resources- controller ... 2023-05-26T08:52:06.742816737Z E0526 08:52:06.742659 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) 2023-05-26T08:52:06.742816737Z goroutine 677 [running]: 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/runtime.logPanic({0x22b8d80?, 0x3f82000}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0xc0010675f8, 0x1, 0x21978a0?}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75 2023-05-26T08:52:06.742816737Z panic({0x22b8d80, 0x3f82000}) 2023-05-26T08:52:06.742816737Z /usr/lib/golang/src/runtime/panic.go:884 +0x212 2023-05-26T08:52:06.742816737Z github.com/openshift/cert-manager-operator/pkg/controller/deployment.withPodLabelsValidateHook.func2(0x3f3df40?, 0xc00015ca00?) 2023-05-26T08:52:06.742816737Z /remote-source/app/pkg/controller/deployment/deployment_overrides_validation.go:178 +0x2a5 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/operator/deploymentcontroller.(*DeploymentController).getDeployment(0xc000cdb0e0, 0xc000f4b900?) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/operator/deploymentcontroller/deployment_controller.go:226 +0x1ce 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/operator/deploymentcontroller.(*DeploymentController).syncManaged(0xc000cdb0e0, {0x2ad2b70, 0xc000f4b900}, 0x0?, 0xc000d421a0, {0x2ad1560, 0xc0009fff50}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/operator/deploymentcontroller/deployment_controller.go:138 +0xde 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/operator/deploymentcontroller.(*DeploymentController).sync(0xc000cdb0e0, {0x2ad2b70, 0xc000f4b900}, {0x2ad1560, 0xc0009fff50}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/operator/deploymentcontroller/deployment_controller.go:127 +0x15d 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).reconcile(0xc000cdb560, {0x2ad2b70, 0xc000f4b900}, {0x2ad1560?, 0xc0009fff50?}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:205 +0x47 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).processNextWorkItem(0xc000cdb560, {0x2ad2b70, 0xc000f4b900}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:264 +0x23e 2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker.func1({0x2ad2b70, 0xc000f4b900}) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:196 +0x96 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1() 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:190 +0x25 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x3feedb2?) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000c639e0?, {0x2aac600, 0xc0009ffef0}, 0x1, 0xc000c639e0) 2023-05-26T08:52:06.742816737Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6 2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000f4b900?, 0x3b9aca00, 0x0, 0xf8?, 0x20?) 2023-05-26T08:52:06.742865703Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89 2023-05-26T08:52:06.742865703Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x2ad2b70, 0xc000f4b900}, 0xc000b03f70, 0x1fae5c5?, 0xc001016750?, 0x80?) 2023-05-26T08:52:06.742865703Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:190 +0x99 2023-05-26T08:52:06.742865703Z k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...) 2023-05-26T08:52:06.742865703Z /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:101 2023-05-26T08:52:06.742865703Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker(0xc000b7da70?, {0x2ad2b70?, 0xc000f4b900?}) 2023-05-26T08:52:06.742865703Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:187 +0x53 2023-05-26T08:52:06.742865703Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).Run.func2() 2023-05-26T08:52:06.742865703Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:117 +0x72 2023-05-26T08:52:06.742865703Z created by github.com/openshift/library-go/pkg/controller/factory.(*baseController).Run 2023-05-26T08:52:06.742865703Z /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:112 +0x297 2023-05-26T08:52:06.742865703Z I0526 08:52:06.742658 1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"cert-manager-operator", Name:"cert-manager-operator-controller-manager", UID:"f181ea7a-239f-46f8-8a42-11f23f328f46", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'Cert-Manager-OperatorPanic' Panic observed: runtime error: invalid memory address or nil pointer dereference 2023-05-26T08:52:06.742865703Z I0526 08:52:06.742757 1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"cert-manager-operator", Name:"cert-manager-operator-controller-manager", UID:"f181ea7a-239f-46f8-8a42-11f23f328f46", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'Cert-Manager-OperatorPanic' Panic observed: runtime error: invalid memory address or nil pointer dereference ...
See log attached to https://drive.google.com/file/d/1Iw_zHg1qgHlPKG0WsSm08hmxPXKfnZKj/view?usp=share_link :
Actual results:
cert-manager-operator pod panics
Expected results:
No panics.
Additional info:
- is cloned by
-
OCPBUGS-14343 cherrypick/cert-manager-1.11: cert-manager-operator pod panics after setting logLevel and overrideArgs --v args
- Closed
- is depended on by
-
OCPBUGS-14343 cherrypick/cert-manager-1.11: cert-manager-operator pod panics after setting logLevel and overrideArgs --v args
- Closed
- links to