Critical Hi Team,
Customer is getting Cluster operator status showing Unavailable:
ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: APIServiceResourceIssue, message: found the CA cert is not active
MSFT provided the article: https://access.redhat.com/solutions/6993953
Cx refused to apply the AP getting from Redhat official documentation. Cx need a redhat engineer to execute commands for him !
Below steps have been given to cx:
# Check Cluster Existing Certificates : echo -e "NAMESPACE\tNAME\tEXPIRY" && oc get secrets -A -o go-template='{{range .items}}{{if eq .type "kubernetes.io/tls"}}{{.metadata.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}' | while read namespace name cert; do echo -en "$namespace\t$name\t"; echo $cert | base64 -d | openssl x509 -noout -enddate; done | column -t # Manually Update Cluster Certificates : az aro update -n xxxx -g xxxx --refresh-credentials --debug # Check again Cluster Existing Certificates : echo -e "NAMESPACE\tNAME\tEXPIRY" && oc get secrets -A -o go-template='{{range .items}}{{if eq .type "kubernetes.io/tls"}}{{.metadata.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}' | while read namespace name cert; do echo -en "$namespace\t$name\t"; echo $cert | base64 -d | openssl x509 -noout -enddate; done | column -t #Renew Secret/Certificate for OLM : # Check Secret Expiration : oc get secret packageserver-service-cert -o json -n openshift-operator-lifecycle-manager | jq -r '.data | .["tls.crt"]' | base64 -d | openssl x509 -noout -dates # Backup the current secret : oc get secret packageserver-service-cert -o json -n openshift-operator-lifecycle-manager > packageserver-service-cert.yaml # Delete the Secret : oc delete secret packageserver-service-cert -n openshift-operator-lifecycle-manager # Check Secret Expiration again : oc get secret packageserver-service-cert -o json -n openshift-operator-lifecycle-manager | jq -r '.data | .["tls.crt"]' | base64 -d | openssl x509 -noout -dates # Get Cluster Operator : oc get co oc get co operator-lifecycle-manager oc get co operator-lifecycle-manager-catalog oc get co operator-lifecycle-manager-packageserver # Go to the kube-system namespace and take the backup of extension-apiserver-authentication configmap: oc project kube-system oc get cm extension-apiserver-authentication -oyaml >> extcm_backup.yaml # delete the extension-apiserver-authentication configmap to : oc delete cm extension-apiserver-authentication -n kube-system oc get cm -n kube-system |grep extension-apiserver-authentication oc get apiservice v1.packages.operators.coreos.com -o jsonpath='{.spec.caBundle}' | base64 -d | openssl x509 -noout -text
We have check the certificate details as below :
$ oc get apiservice v1.packages.operators.coreos.com -o jsonpath='{.spec.caBundle}' | base64 -d | openssl x509 -text E1213 10:24:41.606151 3802053 memcache.go:255] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request E1213 10:24:41.639144 3802053 memcache.go:106] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request E1213 10:24:41.651532 3802053 memcache.go:106] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request E1213 10:24:41.660851 3802053 memcache.go:106] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request Certificate: Data: Version: 3 (0x2) Serial Number: 5319897470906267024 (0x49d4129052ddf590) Signature Algorithm: ecdsa-with-SHA256 Issuer: O = "Red Hat, Inc." Validity Not Before: Nov 29 18:41:35 2021 GMT Not After : Nov 29 18:41:35 2023 GMT Subject: O = "Red Hat, Inc." Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:ea:c0:af:d3:af:e6:0e:61:82:c8:f4:fe:ec:22: 8d:c5:c1:08:6f:91:92:8b:09:05:e9:72:ca:d4:68: fb:aa:e1:ec:e2:e8:ca:32:4c:1f:e7:fc:3a:eb:61: 0b:df:9c:b4:13:62:f4:67:6c:d2:8f:97:a0:a8:a8: 69:08:22:4d:62 ASN1 OID: prime256v1 NIST CURVE: P-256 X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Certificate Sign X509v3 Extended Key Usage: TLS Web Client Authentication, TLS Web Server Authentication X509v3 Basic Constraints: critical CA:TRUE X509v3 Subject Key Identifier: 53:A4:1D:22:F8:0F:8E:C5:74:8C:C6:F4:90:F0:2D:29:B0:65:89:19 Signature Algorithm: ecdsa-with-SHA256 30:45:02:21:00:f5:32:98:3d:34:b6:fd:65:47:3b:31:0d:88: fc:fe:35:cd:4f:51:75:a0:89:16:1a:9e:56:d5:f7:49:e6:3a: a3:02:20:43:fa:81:78:56:f4:1f:9b:3a:5b:7f:28:7e:a8:5b: b7:7a:3e:0a:99:67:88:0e:66:e4:c9:d5:9d:2f:79:80:3e ----BEGIN CERTIFICATE---- MIIBhzCCAS2gAwIBAgIISdQSkFLd9ZAwCgYIKoZIzj0EAwIwGDEWMBQGA1UEChMN UmVkIEhhdCwgSW5jLjAeFw0yMTExMjkxODQxMzVaFw0yMzExMjkxODQxMzVaMBgx FjAUBgNVBAoTDVJlZCBIYXQsIEluYy4wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNC AATqwK/Tr+YOYYLI9P7sIo3FwQhvkZKLCQXpcsrUaPuq4ezi6MoyTB/n/DrrYQvf nLQTYvRnbNKPl6CoqGkIIk1io2EwXzAOBgNVHQ8BAf8EBAMCAoQwHQYDVR0lBBYw FAYIKwYBBQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYE FFOkHSL4D47FdIzG9JDwLSmwZYkZMAoGCCqGSM49BAMCA0gAMEUCIQD1Mpg9NLb9 ZUc7MQ2I/P41zU9RdaCJFhqeVtX3SeY6owIgQ/qBeFb0H5s6W38ofqhbt3o+Cpln iA5m5MnVnS95gD4=
Thanks
