Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-12475

Many SNOs failed to complete install with etcd operator unavailable (4.13.0-rc.5)

XMLWordPrintable

    • No
    • 3
    • ETCD Sprint 235
    • 1
    • Proposed
    • False
    • Hide

      None

      Show
      None
    • 4/25: telco reviewed

      Description of problem:

      While attempting to install 2430 SNOs from ACM via ZTP using the assisted-installer, 188 (and possibly more clusters) failed to complete install because etcd operator is not available due to leader election lock. There are an additional 6 other clusters with this failure but also have an machine-config operator degraded failure was well.

      Version-Release number of selected component (if applicable):

      Hub OCP - 4.12.10
SNO OCP - 4.13.0-rc.5
ACM - 2.8.0-DOWNSTREAM-2023-04-17-13-54-41

      How reproducible:

      ~8% of the installs seem to have failed because of this reason

      Steps to Reproduce:

      1.
2.
3.

      Actual results:

       

      Expected results:

       

      Additional info:

      # oc get clusterversion,co
NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
clusterversion.config.openshift.io/version             False       True          24h     Unable to apply 4.13.0-rc.5: the cluster operator etcd is not availableNAME                                                                           VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
clusteroperator.config.openshift.io/authentication                             4.13.0-rc.5   True        False         False      5h43m   
clusteroperator.config.openshift.io/baremetal                                  4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/cloud-controller-manager                   4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/cloud-credential                           4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/cluster-autoscaler                         4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/config-operator                            4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/console                                    4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/control-plane-machine-set                  4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/csi-snapshot-controller                    4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/dns                                        4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/etcd                                                     False       True          False      24h     StaticPodsAvailable: 0 nodes are active; 1 nodes are at revision 0; 0 nodes have achieved new revision 2
clusteroperator.config.openshift.io/image-registry                             4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/ingress                                    4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/insights                                   4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/kube-apiserver                             4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/kube-controller-manager                    4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/kube-scheduler                             4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/kube-storage-version-migrator              4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/machine-api                                4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/machine-approver                           4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/machine-config                             4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/marketplace                                4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/monitoring                                 4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/network                                    4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/node-tuning                                4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/openshift-apiserver                        4.13.0-rc.5   True        False         False      138m    
clusteroperator.config.openshift.io/openshift-controller-manager               4.13.0-rc.5   True        False         False      5h44m   
clusteroperator.config.openshift.io/openshift-samples                          4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/operator-lifecycle-manager                 4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/operator-lifecycle-manager-catalog         4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/operator-lifecycle-manager-packageserver   4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/service-ca                                 4.13.0-rc.5   True        False         False      24h     
clusteroperator.config.openshift.io/storage                                    4.13.0-rc.5   True        False         False      24h  
# oc get po -n openshift-etcd-operator 
NAME                             READY   STATUS             RESTARTS        AGE
etcd-operator-595cb6d4fc-tv65n   0/1     CrashLoopBackOff   274 (27s ago)   24h
# oc get po -n openshift-etcd-operator
NAME                             READY   STATUS             RESTARTS        AGE
etcd-operator-595cb6d4fc-tv65n   0/1     CrashLoopBackOff   274 (27s ago)   24h
(.ansible) [root@e27-h01-000-r650 install-data]# oc describe co etcd
Name:         etcd
Namespace:
Labels:       <none>
Annotations:  exclude.release.openshift.io/internal-openshift-hosted: true
              include.release.openshift.io/self-managed-high-availability: true
              include.release.openshift.io/single-node-developer: true
API Version:  config.openshift.io/v1
Kind:         ClusterOperator
Metadata:
  Creation Timestamp:  2023-04-23T19:33:57Z
  Generation:          1
  Managed Fields:
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:exclude.release.openshift.io/internal-openshift-hosted:
          f:include.release.openshift.io/self-managed-high-availability:
          f:include.release.openshift.io/single-node-developer:
        f:ownerReferences:
          .:
          k:{"uid":"2d0a03a3-fe6f-4604-8b6a-ea58060d1921"}:
      f:spec:
    Manager:      cluster-version-operator
    Operation:    Update
    Time:         2023-04-23T19:33:57Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:extension:
        f:relatedObjects:
    Manager:      cluster-version-operator
    Operation:    Update
    Subresource:  status
    Time:         2023-04-23T19:33:58Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:versions:
    Manager:      cluster-etcd-operator
    Operation:    Update
    Subresource:  status
    Time:         2023-04-23T19:45:19Z
  Owner References:
    API Version:     config.openshift.io/v1
    Controller:      true
    Kind:            ClusterVersion
    Name:            version
    UID:             2d0a03a3-fe6f-4604-8b6a-ea58060d1921
  Resource Version:  5648
  UID:               cfbd4adf-2a0f-4ca9-9dfe-e7051a02a057
Spec:
Status:
  Conditions:
    Last Transition Time:  2023-04-23T19:44:42Z
    Message:               EtcdMembersControllerDegraded: giving up getting a cached client after 3 tries
NodeControllerDegraded: All master nodes are ready
ClusterMemberControllerDegraded: could not get list of unhealthy members: giving up getting a cached client after 3 tries
EtcdEndpointsDegraded: failed to get member list: giving up getting a cached client after 3 tries
    Reason:                AsExpected
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2023-04-23T19:45:11Z
    Message:               NodeInstallerProgressing: 1 nodes are at revision 0; 0 nodes have achieved new revision 2
    Reason:                NodeInstaller
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2023-04-23T19:44:44Z
    Message:               StaticPodsAvailable: 0 nodes are active; 1 nodes are at revision 0; 0 nodes have achieved new revision 2
    Reason:                StaticPods_ZeroNodesActive
    Status:                False
    Type:                  Available
    Last Transition Time:  2023-04-23T19:44:42Z
    Message:               All is well
    Reason:                AsExpected
    Status:                True
    Type:                  Upgradeable
    Last Transition Time:  2023-04-23T19:44:44Z
    Message:               The etcd backup controller is starting, and will decide if recent backups are available or if a backup is required
    Reason:                ControllerStarted
    Status:                Unknown
    Type:                  RecentBackup
  Extension:               <nil>
  Related Objects:
    Group:     operator.openshift.io
    Name:      cluster
    Resource:  etcds
    Group:
    Name:      openshift-config
    Resource:  namespaces
    Group:
    Name:      openshift-config-managed
    Resource:  namespaces
    Group:
    Name:      openshift-etcd-operator
    Resource:  namespaces
    Group:
    Name:      openshift-etcd
    Resource:  namespaces
  Versions:
    Name:     raw-internal
    Version:  4.13.0-rc.5
Events:       <none>

       

      Looks like a lock is held, maybe a race condition is being hit with this version we haven't seen before:

      # oc logs -n openshift-etcd-operator etcd-operator-595cb6d4fc-tv65n
I0424 20:24:30.442186       1 profiler.go:21] Starting profiling endpoint at http://127.0.0.1:6060/debug/pprof/
I0424 20:24:30.442204       1 observer_polling.go:52] Starting from specified content for file "/var/run/secrets/serving-cert/tls.crt"
I0424 20:24:30.442355       1 observer_polling.go:52] Starting from specified content for file "/var/run/secrets/serving-cert/tls.key"
I0424 20:24:30.442410       1 observer_polling.go:159] Starting file observer
I0424 20:24:30.442518       1 observer_polling.go:135] File observer successfully synced
I0424 20:24:30.442612       1 cmd.go:209] Using service-serving-cert provided certificates
I0424 20:24:30.442652       1 observer_polling.go:74] Adding reactor for file "/var/run/secrets/serving-cert/tls.crt"
I0424 20:24:30.442690       1 observer_polling.go:74] Adding reactor for file "/var/run/secrets/serving-cert/tls.key"
I0424 20:24:30.443215       1 observer_polling.go:52] Starting from specified content for file "/var/run/configmaps/config/config.yaml"
I0424 20:24:30.443662       1 observer_polling.go:159] Starting file observer
I0424 20:24:30.443854       1 observer_polling.go:135] File observer successfully synced
I0424 20:24:30.458491       1 builder.go:262] openshift-cluster-etcd-operator version 4.13.0-202304190216.p0.gf9f2b5d.assembly.stream-f9f2b5d-f9f2b5dc404d866dee4b8c097b4313adddeb2f62
I0424 20:24:30.459022       1 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"
I0424 20:24:30.729586       1 requestheader_controller.go:244] Loaded a new request header values for RequestHeaderAuthRequestController
I0424 20:24:30.734561       1 maxinflight.go:140] "Initialized nonMutatingChan" len=400
I0424 20:24:30.734593       1 maxinflight.go:146] "Initialized mutatingChan" len=200
I0424 20:24:30.734615       1 timing_ratio_histogram.go:202] "TimingRatioHistogramVec.NewForLabelValuesSafe hit the inefficient case" fqName="apiserver_flowcontrol_read_vs_write_current_requests" labelValues=[executing readOnly]        
I0424 20:24:30.734640       1 timing_ratio_histogram.go:202] "TimingRatioHistogramVec.NewForLabelValuesSafe hit the inefficient case" fqName="apiserver_flowcontrol_read_vs_write_current_requests" labelValues=[executing mutating]        
I0424 20:24:30.734654       1 maxinflight.go:117] "Set denominator for readonly requests" limit=400
I0424 20:24:30.734661       1 maxinflight.go:121] "Set denominator for mutating requests" limit=200
I0424 20:24:30.734689       1 config.go:760] Not requested to run hook priority-and-fairness-config-consumer
W0424 20:24:30.738701       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected.
W0424 20:24:30.738735       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected.
I0424 20:24:30.738734       1 genericapiserver.go:484] MuxAndDiscoveryComplete has all endpoints registered and discovery information is complete
I0424 20:24:30.741856       1 builder.go:403] detected SingleReplicaTopologyMode, the original leader election has been altered for the default SingleReplicaToplogy
I0424 20:24:30.742045       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0424 20:24:30.742092       1 shared_informer.go:273] Waiting for caches to sync for RequestHeaderAuthRequestController
I0424 20:24:30.742140       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0424 20:24:30.742158       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0424 20:24:30.742179       1 shared_informer.go:273] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0424 20:24:30.742211       1 reflector.go:221] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169
I0424 20:24:30.742221       1 reflector.go:257] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169
I0424 20:24:30.742167       1 shared_informer.go:273] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0424 20:24:30.742243       1 leaderelection.go:248] attempting to acquire leader lease openshift-etcd-operator/openshift-cluster-etcd-operator-lock...
I0424 20:24:30.742260       1 reflector.go:221] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169
I0424 20:24:30.742267       1 reflector.go:257] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169
I0424 20:24:30.742211       1 reflector.go:221] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169
I0424 20:24:30.742293       1 reflector.go:257] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169
I0424 20:24:30.742417       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key"
I0424 20:24:30.742421       1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" certDetail="\"metrics.openshift-etcd-operator.svc\" [serving] validServingFor=[metrics.openshift-etcd-operator.svc,metrics.openshift-etcd-operator.svc.cluster.local] issuer=\"openshift-service-serving-signer@1682279082\" (2023-04-23 19:44:46 +0000 UTC to 2025-04-22 19:44:47 +0000 UTC (now=2023-04-24 20:24:30.742373973 +0000 UTC))"
I0424 20:24:30.742612       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1682367870\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1682367870\" (2023-04-24 19:24:30 +0000 UTC to 2024-04-23 19:24:30 +0000 UTC (now=2023-04-24 20:24:30.742592419 +0000 UTC))"
I0424 20:24:30.742641       1 secure_serving.go:210] Serving securely on [::]:8443
I0424 20:24:30.742659       1 genericapiserver.go:589] [graceful-termination] waiting for shutdown to be initiated
I0424 20:24:30.742677       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0424 20:24:30.745112       1 leaderelection.go:352] lock is held by etcd-operator-595cb6d4fc-tv65n_39d77f73-d651-45e8-aa25-4c7151669bc1 and has not yet expired
I0424 20:24:30.745142       1 leaderelection.go:253] failed to acquire lease openshift-etcd-operator/openshift-cluster-etcd-operator-lock
I0424 20:24:30.843003       1 shared_informer.go:303] caches populated
I0424 20:24:30.843028       1 shared_informer.go:280] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0424 20:24:30.843003       1 shared_informer.go:303] caches populated
I0424 20:24:30.843094       1 shared_informer.go:280] Caches are synced for RequestHeaderAuthRequestController
I0424 20:24:30.843011       1 shared_informer.go:303] caches populated
I0424 20:24:30.843189       1 shared_informer.go:280] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0424 20:24:30.843222       1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"openshift-kube-apiserver-operator_aggregator-client-signer@1682347375\" [] issuer=\"<self>\" (2023-04-24 14:42:54 +0000 UTC to 2023-05-24 14:42:55 +0000 UTC (now=2023-04-24 20:24:30.843196562 +0000 UTC))"     
I0424 20:24:30.843371       1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" certDetail="\"metrics.openshift-etcd-operator.svc\" [serving] validServingFor=[metrics.openshift-etcd-operator.svc,metrics.openshift-etcd-operator.svc.cluster.local] issuer=\"openshift-service-serving-signer@1682279082\" (2023-04-23 19:44:46 +0000 UTC to 2025-04-22 19:44:47 +0000 UTC (now=2023-04-24 20:24:30.843353322 +0000 UTC))"
I0424 20:24:30.843502       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1682367870\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1682367870\" (2023-04-24 19:24:30 +0000 UTC to 2024-04-23 19:24:30 +0000 UTC (now=2023-04-24 20:24:30.843487048 +0000 UTC))"
I0424 20:24:30.843638       1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"admin-kubeconfig-signer\" [] issuer=\"<self>\" (2023-04-23 19:30:42 +0000 UTC to 2033-04-20 19:30:42 +0000 UTC (now=2023-04-24 20:24:30.843626087 +0000 UTC))"
I0424 20:24:30.843660       1 tlsconfig.go:178] "Loaded client CA" index=1 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kube-control-plane-signer\" [] issuer=\"<self>\" (2023-04-23 19:30:46 +0000 UTC to 2024-04-22 19:30:46 +0000 UTC (now=2023-04-24 20:24:30.843650181 +0000 UTC))"                                                 
I0424 20:24:30.843683       1 tlsconfig.go:178] "Loaded client CA" index=2 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kube-apiserver-to-kubelet-signer\" [] issuer=\"<self>\" (2023-04-23 19:30:46 +0000 UTC to 2024-04-22 19:30:46 +0000 UTC (now=2023-04-24 20:24:30.843667459 +0000 UTC))"                                          
I0424 20:24:30.843703       1 tlsconfig.go:178] "Loaded client CA" index=3 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubelet-bootstrap-kubeconfig-signer\" [] issuer=\"<self>\" (2023-04-23 19:30:43 +0000 UTC to 2033-04-20 19:30:43 +0000 UTC (now=2023-04-24 20:24:30.843692958 +0000 UTC))"                                       
I0424 20:24:30.843722       1 tlsconfig.go:178] "Loaded client CA" index=4 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"openshift-kube-apiserver-operator_node-system-admin-signer@1682279354\" [] issuer=\"<self>\" (2023-04-23 19:49:13 +0000 UTC to 2024-04-22 19:49:14 +0000 UTC (now=2023-04-24 20:24:30.843709439 +0000 UTC))"     
I0424 20:24:30.843742       1 tlsconfig.go:178] "Loaded client CA" index=5 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"openshift-kube-controller-manager-operator_csr-signer-signer@1682347374\" [] issuer=\"<self>\" (2023-04-24 14:42:53 +0000 UTC to 2023-06-23 14:42:54 +0000 UTC (now=2023-04-24 20:24:30.843731795 +0000 UTC))"   
I0424 20:24:30.843762       1 tlsconfig.go:178] "Loaded client CA" index=6 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"openshift-kube-controller-manager-operator_csr-signer-signer@1682347374\" [] issuer=\"<self>\" (2023-04-24 14:42:53 +0000 UTC to 2023-06-23 14:42:54 +0000 UTC (now=2023-04-24 20:24:30.843751697 +0000 UTC))"   
I0424 20:24:30.843782       1 tlsconfig.go:178] "Loaded client CA" index=7 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kube-csr-signer_@1682347554\" [] issuer=\"openshift-kube-controller-manager-operator_csr-signer-signer@1682347374\" (2023-04-24 14:45:53 +0000 UTC to 2023-05-24 14:45:54 +0000 UTC (now=2023-04-24 20:24:30.843771595 +0000 UTC))"
I0424 20:24:30.843810       1 tlsconfig.go:178] "Loaded client CA" index=8 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"openshift-kube-apiserver-operator_aggregator-client-signer@1682347375\" [] issuer=\"<self>\" (2023-04-24 14:42:54 +0000 UTC to 2023-05-24 14:42:55 +0000 UTC (now=2023-04-24 20:24:30.843791757 +0000 UTC))"     
I0424 20:24:30.843982       1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" certDetail="\"metrics.openshift-etcd-operator.svc\" [serving] validServingFor=[metrics.openshift-etcd-operator.svc,metrics.openshift-etcd-operator.svc.cluster.local] issuer=\"openshift-service-serving-signer@1682279082\" (2023-04-23 19:44:46 +0000 UTC to 2025-04-22 19:44:47 +0000 UTC (now=2023-04-24 20:24:30.843947831 +0000 UTC))"
I0424 20:24:30.844089       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1682367870\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1682367870\" (2023-04-24 19:24:30 +0000 UTC to 2024-04-23 19:24:30 +0000 UTC (now=2023-04-24 20:24:30.844077112 +0000 UTC))"
I0424 20:24:42.390021       1 httplog.go:132] "HTTP" verb="GET" URI="/metrics" latency="2.751038ms" userAgent="Prometheus/2.42.0" audit-ID="1834080f-f216-4dc5-ba80-493dce325c02" srcIP="[fd01:0:0:1::58]:60842" resp=200                   
I0424 20:25:12.385498       1 httplog.go:132] "HTTP" verb="GET" URI="/metrics" latency="3.701062ms" userAgent="Prometheus/2.42.0" audit-ID="a674ef49-257f-428b-9335-eae7da18218c" srcIP="[fd01:0:0:1::58]:60842" resp=200                   
I0424 20:25:42.385312       1 httplog.go:132] "HTTP" verb="GET" URI="/metrics" latency="3.762627ms" userAgent="Prometheus/2.42.0" audit-ID="e58a2546-ee6f-430e-bc53-c0db08088374" srcIP="[fd01:0:0:1::58]:60842" resp=200                   
I0424 20:26:01.368045       1 leaderelection.go:352] lock is held by etcd-operator-595cb6d4fc-tv65n_39d77f73-d651-45e8-aa25-4c7151669bc1 and has not yet expired
I0424 20:26:01.368180       1 leaderelection.go:253] failed to acquire lease openshift-etcd-operator/openshift-cluster-etcd-operator-lock
I0424 20:26:12.391336       1 httplog.go:132] "HTTP" verb="GET" URI="/metrics" latency="8.357661ms" userAgent="Prometheus/2.42.0" audit-ID="9087bfbb-0802-473a-b3a7-1d49f0106020" srcIP="[fd01:0:0:1::58]:60842" resp=200                   
I0424 20:26:42.384777       1 httplog.go:132] "HTTP" verb="GET" URI="/metrics" latency="2.6563ms" userAgent="Prometheus/2.42.0" audit-ID="a7e35c0a-4446-4136-80cd-817cbd384b16" srcIP="[fd01:0:0:1::58]:60842" resp=200                     
I0424 20:27:12.384041       1 httplog.go:132] "HTTP" verb="GET" URI="/metrics" latency="2.823607ms" userAgent="Prometheus/2.42.0" audit-ID="23c8a1f8-95fc-4bd5-a95c-7dcbc5a05836" srcIP="[fd01:0:0:1::58]:60842" resp=200                   
I0424 20:27:29.978373       1 cmd.go:97] Received SIGTERM or SIGINT signal, shutting down controller.
W0424 20:27:29.978429       1 leaderelection.go:84] leader election lost

        1. must-gather-vm00046-etcd-degraded.tar.gz
          35.51 MB
        2. must-gather-vm00065-etcd-degraded.tar.gz
          39.62 MB
        3. must-gather-vm00081-etcd-degraded.tar.gz
          38.68 MB
        4. must-gather-vm00105-etcd-degraded.tar.gz
          17.25 MB

              tjungblu@redhat.com Thomas Jungblut
              akrzos@redhat.com Alex Krzos
              Ge Liu Ge Liu
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: