-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
4.12
-
None
-
Quality / Stability / Reliability
-
False
-
-
1
-
Important
-
None
-
None
-
None
-
Proposed
-
OCP VE Sprint 228
-
1
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
Version-Release number of selected component (if applicable):
4.12.0-0.nightly-2022-09-12-152748
How reproducible:
Two vsphere installation jobs failed with same error
Steps to Reproduce:
Created vshpere sdn clusters by flexy job, installation failed. oc get network -o yaml .... networkType: OpenShiftSDN .... Checked network operator $ oc get pods -n openshift-network-operator NAME READY STATUS RESTARTS AGE network-operator-59b84bb686-fqggh 0/1 CrashLoopBackOff 11 (114s ago) 60m $ oc logs network-operator-59b84bb686-fqggh -n openshift-network-operator W0914 01:14:53.234814 1 cmd.go:213] Using insecure, self-signed certificates I0914 01:14:53.634225 1 observer_polling.go:159] Starting file observer I0914 01:14:53.658825 1 builder.go:262] network-operator version 4.12.0-202209111137.p0.g263b970.assembly.stream-263b970-263b970d84d2a96c07f6131bfac40a372516e6a8 W0914 01:14:53.815452 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0914 01:14:53.815475 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. I0914 01:14:53.818250 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0914 01:14:53.818273 1 shared_informer.go:255] Waiting for caches to sync for RequestHeaderAuthRequestController I0914 01:14:53.818332 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0914 01:14:53.818358 1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0914 01:14:53.818393 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0914 01:14:53.818403 1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0914 01:14:53.818513 1 secure_serving.go:210] Serving securely on [::]:9104 I0914 01:14:53.818542 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/serving-cert-2212658079/tls.crt::/tmp/serving-cert-2212658079/tls.key" I0914 01:14:53.818654 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0914 01:14:53.818843 1 leaderelection.go:248] attempting to acquire leader lease openshift-network-operator/network-operator-lock... I0914 01:14:53.918982 1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0914 01:14:53.919087 1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0914 01:14:53.919034 1 shared_informer.go:262] Caches are synced for RequestHeaderAuthRequestController I0914 01:17:40.777884 1 leaderelection.go:258] successfully acquired lease openshift-network-operator/network-operator-lock I0914 01:17:40.778031 1 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"openshift-network-operator", Name:"network-operator-lock", UID:"a8aa11d2-2417-485d-8fe9-bff31d5e1588", APIVersion:"v1", ResourceVersion:"10731", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' jima-upi-proxy-13a-pdjnf-control-plane-2_b27c7563-8417-409c-bac2-16eb2e08f48b became leader I0914 01:17:40.778081 1 event.go:285] Event(v1.ObjectReference{Kind:"Lease", Namespace:"openshift-network-operator", Name:"network-operator-lock", UID:"fcc02024-841e-4d58-bf5f-c27352339a3a", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"10732", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' jima-upi-proxy-13a-pdjnf-control-plane-2_b27c7563-8417-409c-bac2-16eb2e08f48b became leader I0914 01:17:40.796741 1 operator.go:71] Adding controller-runtime controllers I0914 01:17:40.797451 1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-network-operator", Name:"network-operator", UID:"50e6d6ed-8b43-41b4-a71b-39bdf5477fd4", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "LoggingSyncer" resync interval is set to 0s which might lead to client request throttling I0914 01:17:40.797549 1 client.go:232] Starting informers... I0914 01:17:40.797580 1 client.go:243] Waiting for informers to sync... I0914 01:17:40.898329 1 client.go:264] Informers started and synced I0914 01:17:40.898355 1 operator.go:95] Starting controller-manager I0914 01:17:40.898552 1 base_controller.go:67] Waiting for caches to sync for ManagementStateController I0914 01:17:40.898566 1 base_controller.go:73] Caches are synced for ManagementStateController I0914 01:17:40.898571 1 base_controller.go:110] Starting #1 worker of ManagementStateController controller ... I0914 01:17:40.898641 1 base_controller.go:67] Waiting for caches to sync for LoggingSyncer I0914 01:17:40.898650 1 base_controller.go:73] Caches are synced for LoggingSyncer I0914 01:17:40.898653 1 base_controller.go:110] Starting #1 worker of LoggingSyncer controller ... I0914 01:17:40.899046 1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-network-operator", Name:"network-operator", UID:"50e6d6ed-8b43-41b4-a71b-39bdf5477fd4", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'StatusNotFound' Unable to determine current operator status for cluster-network-operator I0914 01:17:40.899344 1 log.go:195] openshift-network-operator/openshift-service-ca.crt changed, triggering operconf reconciliation I0914 01:17:40.899361 1 log.go:195] openshift-network-operator/kube-root-ca.crt changed, triggering operconf reconciliation I0914 01:17:40.899586 1 log.go:195] Reconciling configmap from openshift-authentication-operator/trusted-ca-bundle I0914 01:17:40.899636 1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped I0914 01:17:40.899682 1 log.go:195] Reconciling configmap from openshift-machine-api/mao-trusted-ca I0914 01:17:40.899705 1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped I0914 01:17:40.899738 1 log.go:195] Reconciling configmap from openshift-marketplace/marketplace-trusted-ca I0914 01:17:40.899756 1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped I0914 01:17:40.899775 1 log.go:195] Reconciling configmap from openshift-machine-api/cbo-trusted-ca I0914 01:17:40.899794 1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped I0914 01:17:40.899837 1 log.go:195] Reconciling configmap from openshift-ingress-operator/trusted-ca I0914 01:17:40.899856 1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped I0914 01:17:40.899879 1 log.go:195] Reconciling configmap from openshift-apiserver-operator/trusted-ca-bundle I0914 01:17:40.899905 1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped I0914 01:17:40.899936 1 log.go:195] Reconciling configmap from openshift-insights/trusted-ca-bundle I0914 01:17:40.899953 1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped I0914 01:17:40.899983 1 log.go:195] Reconciling configmap from openshift-cloud-credential-operator/cco-trusted-ca I0914 01:17:40.900001 1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped I0914 01:17:40.900024 1 log.go:195] Reconciling configmap from openshift-image-registry/trusted-ca I0914 01:17:40.900042 1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped I0914 01:17:40.900080 1 log.go:195] Reconciling configmap from openshift-cluster-node-tuning-operator/trusted-ca I0914 01:17:40.900107 1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped I0914 01:17:40.900243 1 log.go:195] Controller "ConnectivityCheckController" resync interval is set to 0s which might lead to client request throttling I0914 01:17:40.900641 1 base_controller.go:67] Waiting for caches to sync for ConnectivityCheckController I0914 01:17:41.000362 1 log.go:195] /jima-upi-proxy-13a-pdjnf-control-plane-0 changed, triggering operconf reconciliation I0914 01:17:41.000401 1 log.go:195] Reconciling Network.operator.openshift.io cluster I0914 01:17:41.000432 1 log.go:195] Reconciling additional trust bundle configmap 'openshift-config/cloud-provider-config' I0914 01:17:41.000449 1 log.go:195] /jima-upi-proxy-13a-pdjnf-control-plane-1 changed, triggering operconf reconciliation I0914 01:17:41.000457 1 log.go:195] /jima-upi-proxy-13a-pdjnf-control-plane-2 changed, triggering operconf reconciliation I0914 01:17:41.000463 1 log.go:195] Reconciling Infrastructure.config.openshift.io cluster I0914 01:17:41.000513 1 log.go:195] Reconciling Network.config.openshift.io cluster panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1b840e7] goroutine 1014 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:118 +0x1f4 panic({0x21a7ec0, 0x3cd0a30}) runtime/panic.go:838 +0x207 github.com/openshift/cluster-network-operator/pkg/controller/infrastructureconfig.(*apiAndIngressVipsSynchronizer).VipsSynchronize(0x25be4dc, 0xc000d96240?) github.com/openshift/cluster-network-operator/pkg/controller/infrastructureconfig/sync_vips.go:34 +0x107 github.com/openshift/cluster-network-operator/pkg/controller/infrastructureconfig.(*ReconcileInfrastructureConfig).Reconcile(0xc000c37320, {0x295dd78, 0xc000912300}, {{{0x0?, 0x10?}, {0xc0008a7330?, 0x413f07?}}}) github.com/openshift/cluster-network-operator/pkg/controller/infrastructureconfig/infrastructureconfig_controller.go:93 +0x368 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x295dcd0?, {0x295dd78?, 0xc000912300?}, {{{0x0?, 0x2411bc0?}, {0xc0008a7330?, 0x4095d4?}}}) sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:121 +0xc8 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000c3bae0, {0x295dcd0, 0xc000b493c0}, {0x228e8a0?, 0xc0000f21a0?}) sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:320 +0x33c sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000c3bae0, {0x295dcd0, 0xc000b493c0}) sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:273 +0x1d9 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:234 +0x85 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:230 +0x325
Actual results:
Network operator pod crashed
Expected results:
Network operator works
Additional info: