Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14129

cert-manager-operator pod panics after setting logLevel and overrideArgs --v args

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • 4.13, 4.12
    • cert-manager
    • None
    • Critical
    • No
    • Proposed
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      $ oc patch certmanager/cluster --type=merge -p '
      spec:
        cainjectorConfig:
          overrideArgs: 
          - --v=2
        logLevel: Trace
        webhookConfig:
          overrideArgs:
          - --v=4
      '

      Then observed cert-manager-operator pod panics.

      Version-Release number of selected component (if applicable):

      v1.11.1-6 in 4.13.0-0.nightly-2023-05-25-001936

      How reproducible:

      Tried two clusters, both hit.

      Steps to Reproduce:

      1.
      $ oc patch certmanager/cluster --type=merge -p '
      spec:
        cainjectorConfig:
          overrideArgs: 
          - --v=2
        logLevel: Trace
        webhookConfig:
          overrideArgs:
          - --v=4
      '
      
      2. Check cert-manager 3 pods, only cert-manager pod is renewed:
      $ oc get po -n cert-manager
      NAME                                      READY   STATUS    RESTARTS   AGE
      cert-manager-6884bb8c8-fqj64              1/1     Running   0          16s
      cert-manager-cainjector-5c4c4b59b-rhgsg   1/1     Running   0          10m51s
      cert-manager-webhook-5cfcfdb665-fj4ln     1/1     Running   0          20m54s
      
      3. Check cert-manager-operator pod logs, found panic:
      ...
      2023-05-26T08:52:06.742816737Z E0526 08:52:06.742494       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
      2023-05-26T08:52:06.742816737Z goroutine 666 [running]:
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/runtime.logPanic({0x22b8d80?, 0x3f82000})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0xc0010635f8, 0x1, 0x21978a0?})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
      2023-05-26T08:52:06.742816737Z panic({0x22b8d80, 0x3f82000})
      2023-05-26T08:52:06.742816737Z  /usr/lib/golang/src/runtime/panic.go:884 +0x212
      2023-05-26T08:52:06.742816737Z github.com/openshift/cert-manager-operator/pkg/controller/deployment.withPodLabelsValidateHook.func2(0x3f305e0?, 0xc0013b4500?)
      2023-05-26T08:52:06.742816737Z  /remote-source/app/pkg/controller/deployment/deployment_overrides_validation.go:182 +0x21b
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/operator/deploymentcontroller.(*DeploymentController).getDeployment(0xc000cec000, 0xc00127a3c0?)
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/operator/deploymentcontroller/deployment_controller.go:226 +0x1ce
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/operator/deploymentcontroller.(*DeploymentController).syncManaged(0xc000cec000, {0x2ad2b70, 0xc00127a3c0}, 0x0?, 0xc000d421a0, {0x2ad1560, 0xc000bb98c0})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/operator/deploymentcontroller/deployment_controller.go:138 +0xde
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/operator/deploymentcontroller.(*DeploymentController).sync(0xc000cec000, {0x2ad2b70, 0xc00127a3c0}, {0x2ad1560, 0xc000bb98c0})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/operator/deploymentcontroller/deployment_controller.go:127 +0x15d
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).reconcile(0xc000cec7e0, {0x2ad2b70, 0xc00127a3c0}, {0x2ad1560?, 0xc000bb98c0?})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:205 +0x47
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).processNextWorkItem(0xc000cec7e0, {0x2ad2b70, 0xc00127a3c0})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:264 +0x23e
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker.func1({0x2ad2b70, 0xc00127a3c0})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:196 +0x96
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:190 +0x25
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x3feedb2?)
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0008d0720?, {0x2aac600, 0xc000bb9890}, 0x1, 0xc0008d0720)
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00127a3c0?, 0x3b9aca00, 0x0, 0xb0?, 0xc000065e60?)
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x2ad2b70, 0xc00127a3c0}, 0xc000c9b770, 0xc000c9b7d0?, 0xb7e3f1?, 0xb0?)
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:190 +0x99
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:101
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker(0x1?, {0x2ad2b70?, 0xc00127a3c0?})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:187 +0x53
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).Run.func2()
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:117 +0x72
      2023-05-26T08:52:06.742816737Z created by github.com/openshift/library-go/pkg/controller/factory.(*baseController).Run
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:112 +0x297
      2023-05-26T08:52:06.742816737Z I0526 08:52:06.742586       1 base_controller.go:73] Caches are synced for cert-manager-cainjector-static-resources-
      2023-05-26T08:52:06.742816737Z I0526 08:52:06.742593       1 base_controller.go:110] Starting #1 worker of cert-manager-cainjector-static-resources- controller ...
      2023-05-26T08:52:06.742816737Z E0526 08:52:06.742659       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
      2023-05-26T08:52:06.742816737Z goroutine 677 [running]:
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/runtime.logPanic({0x22b8d80?, 0x3f82000})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0xc0010675f8, 0x1, 0x21978a0?})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
      2023-05-26T08:52:06.742816737Z panic({0x22b8d80, 0x3f82000})
      2023-05-26T08:52:06.742816737Z  /usr/lib/golang/src/runtime/panic.go:884 +0x212
      2023-05-26T08:52:06.742816737Z github.com/openshift/cert-manager-operator/pkg/controller/deployment.withPodLabelsValidateHook.func2(0x3f3df40?, 0xc00015ca00?)
      2023-05-26T08:52:06.742816737Z  /remote-source/app/pkg/controller/deployment/deployment_overrides_validation.go:178 +0x2a5
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/operator/deploymentcontroller.(*DeploymentController).getDeployment(0xc000cdb0e0, 0xc000f4b900?)
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/operator/deploymentcontroller/deployment_controller.go:226 +0x1ce
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/operator/deploymentcontroller.(*DeploymentController).syncManaged(0xc000cdb0e0, {0x2ad2b70, 0xc000f4b900}, 0x0?, 0xc000d421a0, {0x2ad1560, 0xc0009fff50})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/operator/deploymentcontroller/deployment_controller.go:138 +0xde
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/operator/deploymentcontroller.(*DeploymentController).sync(0xc000cdb0e0, {0x2ad2b70, 0xc000f4b900}, {0x2ad1560, 0xc0009fff50})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/operator/deploymentcontroller/deployment_controller.go:127 +0x15d
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).reconcile(0xc000cdb560, {0x2ad2b70, 0xc000f4b900}, {0x2ad1560?, 0xc0009fff50?})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:205 +0x47
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).processNextWorkItem(0xc000cdb560, {0x2ad2b70, 0xc000f4b900})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:264 +0x23e
      2023-05-26T08:52:06.742816737Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker.func1({0x2ad2b70, 0xc000f4b900})
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:196 +0x96
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:190 +0x25
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x3feedb2?)
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000c639e0?, {0x2aac600, 0xc0009ffef0}, 0x1, 0xc000c639e0)
      2023-05-26T08:52:06.742816737Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
      2023-05-26T08:52:06.742816737Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000f4b900?, 0x3b9aca00, 0x0, 0xf8?, 0x20?)
      2023-05-26T08:52:06.742865703Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
      2023-05-26T08:52:06.742865703Z k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x2ad2b70, 0xc000f4b900}, 0xc000b03f70, 0x1fae5c5?, 0xc001016750?, 0x80?)
      2023-05-26T08:52:06.742865703Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:190 +0x99
      2023-05-26T08:52:06.742865703Z k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
      2023-05-26T08:52:06.742865703Z  /remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:101
      2023-05-26T08:52:06.742865703Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker(0xc000b7da70?, {0x2ad2b70?, 0xc000f4b900?})
      2023-05-26T08:52:06.742865703Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:187 +0x53
      2023-05-26T08:52:06.742865703Z github.com/openshift/library-go/pkg/controller/factory.(*baseController).Run.func2()
      2023-05-26T08:52:06.742865703Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:117 +0x72
      2023-05-26T08:52:06.742865703Z created by github.com/openshift/library-go/pkg/controller/factory.(*baseController).Run
      2023-05-26T08:52:06.742865703Z  /remote-source/app/vendor/github.com/openshift/library-go/pkg/controller/factory/base_controller.go:112 +0x297
      2023-05-26T08:52:06.742865703Z I0526 08:52:06.742658       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"cert-manager-operator", Name:"cert-manager-operator-controller-manager", UID:"f181ea7a-239f-46f8-8a42-11f23f328f46", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'Cert-Manager-OperatorPanic' Panic observed: runtime error: invalid memory address or nil pointer dereference
      2023-05-26T08:52:06.742865703Z I0526 08:52:06.742757       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"cert-manager-operator", Name:"cert-manager-operator-controller-manager", UID:"f181ea7a-239f-46f8-8a42-11f23f328f46", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'Cert-Manager-OperatorPanic' Panic observed: runtime error: invalid memory address or nil pointer dereference
      ...

      See log attached to https://drive.google.com/file/d/1Iw_zHg1qgHlPKG0WsSm08hmxPXKfnZKj/view?usp=share_link :

      Actual results:

      cert-manager-operator pod panics

      Expected results:

      No panics.

      Additional info:

       

            swghosh@redhat.com Swarup Ghosh
            xxia-1 Xingxing Xia
            Xingxing Xia Xingxing Xia
            Swarup Ghosh, Thejas N (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: