Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-1273

Network operator pod crash with panic: runtime error

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.12
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 1
    • Important
    • None
    • None
    • None
    • Proposed
    • OCP VE Sprint 228
    • 1
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      
      

      Version-Release number of selected component (if applicable):

      4.12.0-0.nightly-2022-09-12-152748
      

      How reproducible:

      Two vsphere installation jobs failed with same error
      

      Steps to Reproduce:

      Created vshpere sdn clusters by flexy job, installation failed.
      oc get network -o yaml
      ....
      networkType: OpenShiftSDN
      ....
      Checked network operator
      $ oc get pods -n openshift-network-operator
      NAME                                READY   STATUS             RESTARTS        AGE
      network-operator-59b84bb686-fqggh   0/1     CrashLoopBackOff   11 (114s ago)   60m
      
      $ oc logs network-operator-59b84bb686-fqggh -n openshift-network-operator
      W0914 01:14:53.234814       1 cmd.go:213] Using insecure, self-signed certificates
      I0914 01:14:53.634225       1 observer_polling.go:159] Starting file observer
      I0914 01:14:53.658825       1 builder.go:262] network-operator version 4.12.0-202209111137.p0.g263b970.assembly.stream-263b970-263b970d84d2a96c07f6131bfac40a372516e6a8
      W0914 01:14:53.815452       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected.
      W0914 01:14:53.815475       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected.
      I0914 01:14:53.818250       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
      I0914 01:14:53.818273       1 shared_informer.go:255] Waiting for caches to sync for RequestHeaderAuthRequestController
      I0914 01:14:53.818332       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
      I0914 01:14:53.818358       1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
      I0914 01:14:53.818393       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
      I0914 01:14:53.818403       1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
      I0914 01:14:53.818513       1 secure_serving.go:210] Serving securely on [::]:9104
      I0914 01:14:53.818542       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/serving-cert-2212658079/tls.crt::/tmp/serving-cert-2212658079/tls.key"
      I0914 01:14:53.818654       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
      I0914 01:14:53.818843       1 leaderelection.go:248] attempting to acquire leader lease openshift-network-operator/network-operator-lock...
      I0914 01:14:53.918982       1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
      I0914 01:14:53.919087       1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
      I0914 01:14:53.919034       1 shared_informer.go:262] Caches are synced for RequestHeaderAuthRequestController
      I0914 01:17:40.777884       1 leaderelection.go:258] successfully acquired lease openshift-network-operator/network-operator-lock
      I0914 01:17:40.778031       1 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"openshift-network-operator", Name:"network-operator-lock", UID:"a8aa11d2-2417-485d-8fe9-bff31d5e1588", APIVersion:"v1", ResourceVersion:"10731", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' jima-upi-proxy-13a-pdjnf-control-plane-2_b27c7563-8417-409c-bac2-16eb2e08f48b became leader
      I0914 01:17:40.778081       1 event.go:285] Event(v1.ObjectReference{Kind:"Lease", Namespace:"openshift-network-operator", Name:"network-operator-lock", UID:"fcc02024-841e-4d58-bf5f-c27352339a3a", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"10732", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' jima-upi-proxy-13a-pdjnf-control-plane-2_b27c7563-8417-409c-bac2-16eb2e08f48b became leader
      I0914 01:17:40.796741       1 operator.go:71] Adding controller-runtime controllers
      I0914 01:17:40.797451       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-network-operator", Name:"network-operator", UID:"50e6d6ed-8b43-41b4-a71b-39bdf5477fd4", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "LoggingSyncer" resync interval is set to 0s which might lead to client request throttling
      I0914 01:17:40.797549       1 client.go:232] Starting informers...
      I0914 01:17:40.797580       1 client.go:243] Waiting for informers to sync...
      I0914 01:17:40.898329       1 client.go:264] Informers started and synced
      I0914 01:17:40.898355       1 operator.go:95] Starting controller-manager
      I0914 01:17:40.898552       1 base_controller.go:67] Waiting for caches to sync for ManagementStateController
      I0914 01:17:40.898566       1 base_controller.go:73] Caches are synced for ManagementStateController 
      I0914 01:17:40.898571       1 base_controller.go:110] Starting #1 worker of ManagementStateController controller ...
      I0914 01:17:40.898641       1 base_controller.go:67] Waiting for caches to sync for LoggingSyncer
      I0914 01:17:40.898650       1 base_controller.go:73] Caches are synced for LoggingSyncer 
      I0914 01:17:40.898653       1 base_controller.go:110] Starting #1 worker of LoggingSyncer controller ...
      I0914 01:17:40.899046       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-network-operator", Name:"network-operator", UID:"50e6d6ed-8b43-41b4-a71b-39bdf5477fd4", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'StatusNotFound' Unable to determine current operator status for cluster-network-operator
      I0914 01:17:40.899344       1 log.go:195]  openshift-network-operator/openshift-service-ca.crt changed, triggering operconf reconciliation
      I0914 01:17:40.899361       1 log.go:195]  openshift-network-operator/kube-root-ca.crt changed, triggering operconf reconciliation
      I0914 01:17:40.899586       1 log.go:195] Reconciling configmap from  openshift-authentication-operator/trusted-ca-bundle
      I0914 01:17:40.899636       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
      I0914 01:17:40.899682       1 log.go:195] Reconciling configmap from  openshift-machine-api/mao-trusted-ca
      I0914 01:17:40.899705       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
      I0914 01:17:40.899738       1 log.go:195] Reconciling configmap from  openshift-marketplace/marketplace-trusted-ca
      I0914 01:17:40.899756       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
      I0914 01:17:40.899775       1 log.go:195] Reconciling configmap from  openshift-machine-api/cbo-trusted-ca
      I0914 01:17:40.899794       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
      I0914 01:17:40.899837       1 log.go:195] Reconciling configmap from  openshift-ingress-operator/trusted-ca
      I0914 01:17:40.899856       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
      I0914 01:17:40.899879       1 log.go:195] Reconciling configmap from  openshift-apiserver-operator/trusted-ca-bundle
      I0914 01:17:40.899905       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
      I0914 01:17:40.899936       1 log.go:195] Reconciling configmap from  openshift-insights/trusted-ca-bundle
      I0914 01:17:40.899953       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
      I0914 01:17:40.899983       1 log.go:195] Reconciling configmap from  openshift-cloud-credential-operator/cco-trusted-ca
      I0914 01:17:40.900001       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
      I0914 01:17:40.900024       1 log.go:195] Reconciling configmap from  openshift-image-registry/trusted-ca
      I0914 01:17:40.900042       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
      I0914 01:17:40.900080       1 log.go:195] Reconciling configmap from  openshift-cluster-node-tuning-operator/trusted-ca
      I0914 01:17:40.900107       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
      I0914 01:17:40.900243       1 log.go:195] Controller "ConnectivityCheckController" resync interval is set to 0s which might lead to client request throttling
      I0914 01:17:40.900641       1 base_controller.go:67] Waiting for caches to sync for ConnectivityCheckController
      I0914 01:17:41.000362       1 log.go:195]  /jima-upi-proxy-13a-pdjnf-control-plane-0 changed, triggering operconf reconciliation
      I0914 01:17:41.000401       1 log.go:195] Reconciling Network.operator.openshift.io cluster
      I0914 01:17:41.000432       1 log.go:195] Reconciling additional trust bundle configmap 'openshift-config/cloud-provider-config'
      I0914 01:17:41.000449       1 log.go:195]  /jima-upi-proxy-13a-pdjnf-control-plane-1 changed, triggering operconf reconciliation
      I0914 01:17:41.000457       1 log.go:195]  /jima-upi-proxy-13a-pdjnf-control-plane-2 changed, triggering operconf reconciliation
      I0914 01:17:41.000463       1 log.go:195] Reconciling Infrastructure.config.openshift.io cluster
      I0914 01:17:41.000513       1 log.go:195] Reconciling Network.config.openshift.io cluster
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
      	panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1b840e7]
      
      goroutine 1014 [running]:
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
      	sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:118 +0x1f4
      panic({0x21a7ec0, 0x3cd0a30})
      	runtime/panic.go:838 +0x207
      github.com/openshift/cluster-network-operator/pkg/controller/infrastructureconfig.(*apiAndIngressVipsSynchronizer).VipsSynchronize(0x25be4dc, 0xc000d96240?)
      	github.com/openshift/cluster-network-operator/pkg/controller/infrastructureconfig/sync_vips.go:34 +0x107
      github.com/openshift/cluster-network-operator/pkg/controller/infrastructureconfig.(*ReconcileInfrastructureConfig).Reconcile(0xc000c37320, {0x295dd78, 0xc000912300}, {{{0x0?, 0x10?}, {0xc0008a7330?, 0x413f07?}}})
      	github.com/openshift/cluster-network-operator/pkg/controller/infrastructureconfig/infrastructureconfig_controller.go:93 +0x368
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x295dcd0?, {0x295dd78?, 0xc000912300?}, {{{0x0?, 0x2411bc0?}, {0xc0008a7330?, 0x4095d4?}}})
      	sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:121 +0xc8
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000c3bae0, {0x295dcd0, 0xc000b493c0}, {0x228e8a0?, 0xc0000f21a0?})
      	sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:320 +0x33c
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000c3bae0, {0x295dcd0, 0xc000b493c0})
      	sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:273 +0x1d9
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
      	sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:234 +0x85
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
      	sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:230 +0x325
      

      Actual results:

      Network operator pod crashed
      

      Expected results:

      Network operator works
      

      Additional info:

      
      

              cstabler@redhat.com Christoph Stäbler
              huirwang Huiran Wang
              None
              None
              Anurag Saxena Anurag Saxena
              None
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: