Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-49343

On some clusters, old "serving-cert-*" Secrets in openshift-kube-controller-manager are not cleaned up

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Normal Normal
    • None
    • 4.15.z
    • service-ca
    • Important
    • None
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Customer reported that on at least two clusters, they are seeing expired certificates in the "openshift-kube-controller-manager" namespace. Specifically, we can see that there are still old "serving-cert-*" certificates that were not cleaned up by the Service CA Controller:

       for i in $(oc get secrets -n openshift-kube-controller-manager | grep serving | awk '{print $1}'); do echo "certName: $i" && oc get secrets -n openshift-kube-controller-manager $i -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -issuer -subject && echo ""; done
      certName: serving-cert
      notBefore=Feb 22 14:08:09 2024 GMT
      notAfter=Feb 21 14:08:10 2026 GMT
      issuer=CN = openshift-service-serving-signer@1674482820
      subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc
      
      certName: serving-cert-2
      notBefore=Jan 23 14:07:17 2023 GMT
      notAfter=Jan 22 14:07:18 2025 GMT
      issuer=CN = openshift-service-serving-signer@1674482820
      subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc
      
      certName: serving-cert-29
      notBefore=Feb 22 14:08:09 2024 GMT
      notAfter=Feb 21 14:08:10 2026 GMT
      issuer=CN = openshift-service-serving-signer@1674482820
      subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc
      
      certName: serving-cert-3
      notBefore=Jan 23 14:07:17 2023 GMT
      notAfter=Jan 22 14:07:18 2025 GMT
      issuer=CN = openshift-service-serving-signer@1674482820
      subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc
      
      certName: serving-cert-30
      notBefore=Feb 22 14:08:09 2024 GMT
      notAfter=Feb 21 14:08:10 2026 GMT
      issuer=CN = openshift-service-serving-signer@1674482820
      subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc
      
      certName: serving-cert-31
      notBefore=Feb 22 14:08:09 2024 GMT
      notAfter=Feb 21 14:08:10 2026 GMT
      issuer=CN = openshift-service-serving-signer@1674482820
      subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc
      
      certName: serving-cert-32
      notBefore=Feb 22 14:08:09 2024 GMT
      notAfter=Feb 21 14:08:10 2026 GMT
      issuer=CN = openshift-service-serving-signer@1674482820
      subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc
      
      certName: serving-cert-33
      notBefore=Feb 22 14:08:09 2024 GMT
      notAfter=Feb 21 14:08:10 2026 GMT
      issuer=CN = openshift-service-serving-signer@1674482820
      subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc
      
      certName: serving-cert-4
      notBefore=Jan 23 14:07:17 2023 GMT
      notAfter=Jan 22 14:07:18 2025 GMT
      issuer=CN = openshift-service-serving-signer@1674482820
      subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc
      
      certName: serving-cert-5
      notBefore=Jan 23 14:07:17 2023 GMT
      notAfter=Jan 22 14:07:18 2025 GMT
      issuer=CN = openshift-service-serving-signer@1674482820
      subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc
      
      certName: serving-cert-6
      notBefore=Jan 23 14:07:17 2023 GMT
      notAfter=Jan 22 14:07:18 2025 GMT
      issuer=CN = openshift-service-serving-signer@1674482820
      subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc

      Note the expired Secrets "serving-cert-2", "serving-cert-3", "serving-cert-4", "serving-cert-5", "serving-cert-6".

      Customer would expect that these are deleted (as they are on most of their clusters, but not on two clusters).

      Workaround is to manually remove the expired certificates.

      Version-Release number of selected component (if applicable):

      OpenShift Container Platform 4.15.42

      Platform: vSphere

      How reproducible:

      Only reproducible on two customer clusters (out of 15 clusters)

      Cluster install dates: 2023-01-23 (4.10.47) and 2024-03-27 (4.12.45)

      Steps to Reproduce:

      1.  ** Install the cluster
      2. Leave the cluster running for at least 1 year
      3. Check the Secrets / certificates in "openshift-kube-controller-manager"

      Actual results:

      Some old (maybe even expired after 24 months) certificates are still present in "openshift-kube-controller-manager".

      Expected results:

      No old certificates are to be found

      Additional info:

              wlewis@redhat.com Wallace Lewis
              rhn-support-skrenger Simon Krenger
              Xingxing Xia Xingxing Xia
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: