4.12.0-0.nightly-2022-09-12-152748

Two vsphere installation jobs failed with same error

Created vshpere sdn clusters by flexy job, installation failed.
oc get network -o yaml
....
networkType: OpenShiftSDN
....
Checked network operator
$ oc get pods -n openshift-network-operator
NAME                                READY   STATUS             RESTARTS        AGE
network-operator-59b84bb686-fqggh   0/1     CrashLoopBackOff   11 (114s ago)   60m

$ oc logs network-operator-59b84bb686-fqggh -n openshift-network-operator
W0914 01:14:53.234814       1 cmd.go:213] Using insecure, self-signed certificates
I0914 01:14:53.634225       1 observer_polling.go:159] Starting file observer
I0914 01:14:53.658825       1 builder.go:262] network-operator version 4.12.0-202209111137.p0.g263b970.assembly.stream-263b970-263b970d84d2a96c07f6131bfac40a372516e6a8
W0914 01:14:53.815452       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected.
W0914 01:14:53.815475       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected.
I0914 01:14:53.818250       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0914 01:14:53.818273       1 shared_informer.go:255] Waiting for caches to sync for RequestHeaderAuthRequestController
I0914 01:14:53.818332       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0914 01:14:53.818358       1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0914 01:14:53.818393       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0914 01:14:53.818403       1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0914 01:14:53.818513       1 secure_serving.go:210] Serving securely on [::]:9104
I0914 01:14:53.818542       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/serving-cert-2212658079/tls.crt::/tmp/serving-cert-2212658079/tls.key"
I0914 01:14:53.818654       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0914 01:14:53.818843       1 leaderelection.go:248] attempting to acquire leader lease openshift-network-operator/network-operator-lock...
I0914 01:14:53.918982       1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0914 01:14:53.919087       1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0914 01:14:53.919034       1 shared_informer.go:262] Caches are synced for RequestHeaderAuthRequestController
I0914 01:17:40.777884       1 leaderelection.go:258] successfully acquired lease openshift-network-operator/network-operator-lock
I0914 01:17:40.778031       1 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"openshift-network-operator", Name:"network-operator-lock", UID:"a8aa11d2-2417-485d-8fe9-bff31d5e1588", APIVersion:"v1", ResourceVersion:"10731", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' jima-upi-proxy-13a-pdjnf-control-plane-2_b27c7563-8417-409c-bac2-16eb2e08f48b became leader
I0914 01:17:40.778081       1 event.go:285] Event(v1.ObjectReference{Kind:"Lease", Namespace:"openshift-network-operator", Name:"network-operator-lock", UID:"fcc02024-841e-4d58-bf5f-c27352339a3a", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"10732", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' jima-upi-proxy-13a-pdjnf-control-plane-2_b27c7563-8417-409c-bac2-16eb2e08f48b became leader
I0914 01:17:40.796741       1 operator.go:71] Adding controller-runtime controllers
I0914 01:17:40.797451       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-network-operator", Name:"network-operator", UID:"50e6d6ed-8b43-41b4-a71b-39bdf5477fd4", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "LoggingSyncer" resync interval is set to 0s which might lead to client request throttling
I0914 01:17:40.797549       1 client.go:232] Starting informers...
I0914 01:17:40.797580       1 client.go:243] Waiting for informers to sync...
I0914 01:17:40.898329       1 client.go:264] Informers started and synced
I0914 01:17:40.898355       1 operator.go:95] Starting controller-manager
I0914 01:17:40.898552       1 base_controller.go:67] Waiting for caches to sync for ManagementStateController
I0914 01:17:40.898566       1 base_controller.go:73] Caches are synced for ManagementStateController 
I0914 01:17:40.898571       1 base_controller.go:110] Starting #1 worker of ManagementStateController controller ...
I0914 01:17:40.898641       1 base_controller.go:67] Waiting for caches to sync for LoggingSyncer
I0914 01:17:40.898650       1 base_controller.go:73] Caches are synced for LoggingSyncer 
I0914 01:17:40.898653       1 base_controller.go:110] Starting #1 worker of LoggingSyncer controller ...
I0914 01:17:40.899046       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-network-operator", Name:"network-operator", UID:"50e6d6ed-8b43-41b4-a71b-39bdf5477fd4", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'StatusNotFound' Unable to determine current operator status for cluster-network-operator
I0914 01:17:40.899344       1 log.go:195]  openshift-network-operator/openshift-service-ca.crt changed, triggering operconf reconciliation
I0914 01:17:40.899361       1 log.go:195]  openshift-network-operator/kube-root-ca.crt changed, triggering operconf reconciliation
I0914 01:17:40.899586       1 log.go:195] Reconciling configmap from  openshift-authentication-operator/trusted-ca-bundle
I0914 01:17:40.899636       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
I0914 01:17:40.899682       1 log.go:195] Reconciling configmap from  openshift-machine-api/mao-trusted-ca
I0914 01:17:40.899705       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
I0914 01:17:40.899738       1 log.go:195] Reconciling configmap from  openshift-marketplace/marketplace-trusted-ca
I0914 01:17:40.899756       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
I0914 01:17:40.899775       1 log.go:195] Reconciling configmap from  openshift-machine-api/cbo-trusted-ca
I0914 01:17:40.899794       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
I0914 01:17:40.899837       1 log.go:195] Reconciling configmap from  openshift-ingress-operator/trusted-ca
I0914 01:17:40.899856       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
I0914 01:17:40.899879       1 log.go:195] Reconciling configmap from  openshift-apiserver-operator/trusted-ca-bundle
I0914 01:17:40.899905       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
I0914 01:17:40.899936       1 log.go:195] Reconciling configmap from  openshift-insights/trusted-ca-bundle
I0914 01:17:40.899953       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
I0914 01:17:40.899983       1 log.go:195] Reconciling configmap from  openshift-cloud-credential-operator/cco-trusted-ca
I0914 01:17:40.900001       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
I0914 01:17:40.900024       1 log.go:195] Reconciling configmap from  openshift-image-registry/trusted-ca
I0914 01:17:40.900042       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
I0914 01:17:40.900080       1 log.go:195] Reconciling configmap from  openshift-cluster-node-tuning-operator/trusted-ca
I0914 01:17:40.900107       1 log.go:195] ConfigMap 'openshift-config-managed/trusted-ca-bundle' not found; reconciliation will be skipped
I0914 01:17:40.900243       1 log.go:195] Controller "ConnectivityCheckController" resync interval is set to 0s which might lead to client request throttling
I0914 01:17:40.900641       1 base_controller.go:67] Waiting for caches to sync for ConnectivityCheckController
I0914 01:17:41.000362       1 log.go:195]  /jima-upi-proxy-13a-pdjnf-control-plane-0 changed, triggering operconf reconciliation
I0914 01:17:41.000401       1 log.go:195] Reconciling Network.operator.openshift.io cluster
I0914 01:17:41.000432       1 log.go:195] Reconciling additional trust bundle configmap 'openshift-config/cloud-provider-config'
I0914 01:17:41.000449       1 log.go:195]  /jima-upi-proxy-13a-pdjnf-control-plane-1 changed, triggering operconf reconciliation
I0914 01:17:41.000457       1 log.go:195]  /jima-upi-proxy-13a-pdjnf-control-plane-2 changed, triggering operconf reconciliation
I0914 01:17:41.000463       1 log.go:195] Reconciling Infrastructure.config.openshift.io cluster
I0914 01:17:41.000513       1 log.go:195] Reconciling Network.config.openshift.io cluster
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1b840e7]

goroutine 1014 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:118 +0x1f4
panic({0x21a7ec0, 0x3cd0a30})
	runtime/panic.go:838 +0x207
github.com/openshift/cluster-network-operator/pkg/controller/infrastructureconfig.(*apiAndIngressVipsSynchronizer).VipsSynchronize(0x25be4dc, 0xc000d96240?)
	github.com/openshift/cluster-network-operator/pkg/controller/infrastructureconfig/sync_vips.go:34 +0x107
github.com/openshift/cluster-network-operator/pkg/controller/infrastructureconfig.(*ReconcileInfrastructureConfig).Reconcile(0xc000c37320, {0x295dd78, 0xc000912300}, {{{0x0?, 0x10?}, {0xc0008a7330?, 0x413f07?}}})
	github.com/openshift/cluster-network-operator/pkg/controller/infrastructureconfig/infrastructureconfig_controller.go:93 +0x368
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x295dcd0?, {0x295dd78?, 0xc000912300?}, {{{0x0?, 0x2411bc0?}, {0xc0008a7330?, 0x4095d4?}}})
	sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:121 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000c3bae0, {0x295dcd0, 0xc000b493c0}, {0x228e8a0?, 0xc0000f21a0?})
	sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:320 +0x33c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000c3bae0, {0x295dcd0, 0xc000b493c0})
	sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:273 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:234 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	sigs.k8s.io/controller-runtime@v0.12.0/pkg/internal/controller/controller.go:230 +0x325

Network operator pod crashed

Network operator works