Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-25341

[4.17] The certificate relating to operator-lifecycle-manager-packageserver isn't rotated after expired

XMLWordPrintable

    • Moderate
    • No
    • Nyan Cat, Orion, Phlogiston 250, Quality OLM Sprint 251, Rasputin OLM Sprint 252, Sassy OLM Sprint 253, Toasty OLM Sprint 254, Umbridge OLM Sprint 255
    • 8
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Release Note Not Required
    • Done

      Hi Team,

       Customer is getting Cluster operator status showing Unavailable: 

      ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: APIServiceResourceIssue, message: found the CA cert is not active

      MSFT provided the article:  https://access.redhat.com/solutions/6993953
      Cx refused to apply the AP getting from Redhat official documentation.  Cx need a redhat engineer to execute commands for him ! 
      Below steps have been given to cx:

       

      # Check Cluster Existing Certificates :
              echo -e "NAMESPACE\tNAME\tEXPIRY" && oc get secrets -A -o go-template='{{range .items}}{{if eq .type "kubernetes.io/tls"}}{{.metadata.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}' | while read namespace name cert; do echo -en "$namespace\t$name\t"; echo $cert | base64 -d | openssl x509 -noout -enddate; done | column -t
      # Manually Update Cluster Certificates : 
              az aro update -n xxxx -g xxxx  --refresh-credentials --debug
      # Check again Cluster Existing Certificates :
      
              echo -e "NAMESPACE\tNAME\tEXPIRY" && oc get secrets -A -o go-template='{{range .items}}{{if eq .type "kubernetes.io/tls"}}{{.metadata.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}' | while read namespace name cert; do echo -en "$namespace\t$name\t"; echo $cert | base64 -d | openssl x509 -noout -enddate; done | column -t
      #Renew Secret/Certificate for OLM :
              # Check Secret Expiration :
                      oc get secret packageserver-service-cert -o json -n openshift-operator-lifecycle-manager | jq -r '.data | .["tls.crt"]' | base64 -d | openssl x509 -noout -dates
              # Backup the current secret :
                      oc get secret packageserver-service-cert -o json -n openshift-operator-lifecycle-manager > packageserver-service-cert.yaml
              # Delete the Secret :
                      oc delete secret packageserver-service-cert -n openshift-operator-lifecycle-manager
              # Check Secret Expiration again :
                      oc get secret packageserver-service-cert -o json -n openshift-operator-lifecycle-manager | jq -r '.data | .["tls.crt"]' | base64 -d | openssl x509 -noout -dates
      # Get Cluster Operator :
        oc get co
        oc get co operator-lifecycle-manager
        oc get co operator-lifecycle-manager-catalog
        oc get co operator-lifecycle-manager-packageserver
      # Go to the kube-system namespace and take the backup of extension-apiserver-authentication configmap:
        oc project kube-system 
        oc get cm extension-apiserver-authentication -oyaml >> extcm_backup.yaml
      # delete the extension-apiserver-authentication configmap to :
        oc delete cm extension-apiserver-authentication -n kube-system
        oc get cm -n kube-system |grep extension-apiserver-authentication
        oc get apiservice v1.packages.operators.coreos.com -o jsonpath='{.spec.caBundle}' | base64 -d | openssl x509 -noout -text
       
      

      We have check the certificate details as below :

      $ oc get apiservice 
      v1.packages.operators.coreos.com
      -o jsonpath='{.spec.caBundle}' | base64 -d | openssl x509 -text
      E1213 10:24:41.606151 3802053 memcache.go:255] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
      E1213 10:24:41.639144 3802053 memcache.go:106] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
      E1213 10:24:41.651532 3802053 memcache.go:106] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
      E1213 10:24:41.660851 3802053 memcache.go:106] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
      Certificate:
          Data:
              Version: 3 (0x2)
              Serial Number: 5319897470906267024 (0x49d4129052ddf590)
              Signature Algorithm: ecdsa-with-SHA256
              Issuer: O = "Red Hat, Inc."
              Validity
                  Not Before: Nov 29 18:41:35 2021 GMT
                  Not After : Nov 29 18:41:35 2023 GMT
              Subject: O = "Red Hat, Inc."
              Subject Public Key Info:
                  Public Key Algorithm: id-ecPublicKey
                      Public-Key: (256 bit)
                      pub:
                          04:ea:c0:af:d3:af:e6:0e:61:82:c8:f4:fe:ec:22:
                          8d:c5:c1:08:6f:91:92:8b:09:05:e9:72:ca:d4:68:
                          fb:aa:e1:ec:e2:e8:ca:32:4c:1f:e7:fc:3a:eb:61:
                          0b:df:9c:b4:13:62:f4:67:6c:d2:8f:97:a0:a8:a8:
                          69:08:22:4d:62
                      ASN1 OID: prime256v1
                      NIST CURVE: P-256
              X509v3 extensions:
                  X509v3 Key Usage: critical
                      Digital Signature, Certificate Sign
                  X509v3 Extended Key Usage:
                      TLS Web Client Authentication, TLS Web Server Authentication
                  X509v3 Basic Constraints: critical
                      CA:TRUE
                  X509v3 Subject Key Identifier:
                      53:A4:1D:22:F8:0F:8E:C5:74:8C:C6:F4:90:F0:2D:29:B0:65:89:19
          Signature Algorithm: ecdsa-with-SHA256
               30:45:02:21:00:f5:32:98:3d:34:b6:fd:65:47:3b:31:0d:88:
               fc:fe:35:cd:4f:51:75:a0:89:16:1a:9e:56:d5:f7:49:e6:3a:
               a3:02:20:43:fa:81:78:56:f4:1f:9b:3a:5b:7f:28:7e:a8:5b:
               b7:7a:3e:0a:99:67:88:0e:66:e4:c9:d5:9d:2f:79:80:3e
      ----BEGIN CERTIFICATE----
      MIIBhzCCAS2gAwIBAgIISdQSkFLd9ZAwCgYIKoZIzj0EAwIwGDEWMBQGA1UEChMN
      UmVkIEhhdCwgSW5jLjAeFw0yMTExMjkxODQxMzVaFw0yMzExMjkxODQxMzVaMBgx
      FjAUBgNVBAoTDVJlZCBIYXQsIEluYy4wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNC
      AATqwK/Tr+YOYYLI9P7sIo3FwQhvkZKLCQXpcsrUaPuq4ezi6MoyTB/n/DrrYQvf
      nLQTYvRnbNKPl6CoqGkIIk1io2EwXzAOBgNVHQ8BAf8EBAMCAoQwHQYDVR0lBBYw
      FAYIKwYBBQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYE
      FFOkHSL4D47FdIzG9JDwLSmwZYkZMAoGCCqGSM49BAMCA0gAMEUCIQD1Mpg9NLb9
      ZUc7MQ2I/P41zU9RdaCJFhqeVtX3SeY6owIgQ/qBeFb0H5s6W38ofqhbt3o+Cpln
      iA5m5MnVnS95gD4=
      

       

      Thanks

              ankithom Ankita Thomas
              rhn-support-psuryawa Pramod Suryawanshi
              Jian Zhang Jian Zhang
              Alex Dellapenta Alex Dellapenta
              Ankita Thomas
              Votes:
              5 Vote for this issue
              Watchers:
              37 Start watching this issue

                Created:
                Updated:
                Resolved: