-
Bug
-
Resolution: Duplicate
-
Normal
-
None
-
4.15.z
-
Important
-
None
-
False
-
Description of problem:
Customer reported that on at least two clusters, they are seeing expired certificates in the "openshift-kube-controller-manager" namespace. Specifically, we can see that there are still old "serving-cert-*" certificates that were not cleaned up by the Service CA Controller:
for i in $(oc get secrets -n openshift-kube-controller-manager | grep serving | awk '{print $1}'); do echo "certName: $i" && oc get secrets -n openshift-kube-controller-manager $i -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -issuer -subject && echo ""; done certName: serving-cert notBefore=Feb 22 14:08:09 2024 GMT notAfter=Feb 21 14:08:10 2026 GMT issuer=CN = openshift-service-serving-signer@1674482820 subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc certName: serving-cert-2 notBefore=Jan 23 14:07:17 2023 GMT notAfter=Jan 22 14:07:18 2025 GMT issuer=CN = openshift-service-serving-signer@1674482820 subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc certName: serving-cert-29 notBefore=Feb 22 14:08:09 2024 GMT notAfter=Feb 21 14:08:10 2026 GMT issuer=CN = openshift-service-serving-signer@1674482820 subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc certName: serving-cert-3 notBefore=Jan 23 14:07:17 2023 GMT notAfter=Jan 22 14:07:18 2025 GMT issuer=CN = openshift-service-serving-signer@1674482820 subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc certName: serving-cert-30 notBefore=Feb 22 14:08:09 2024 GMT notAfter=Feb 21 14:08:10 2026 GMT issuer=CN = openshift-service-serving-signer@1674482820 subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc certName: serving-cert-31 notBefore=Feb 22 14:08:09 2024 GMT notAfter=Feb 21 14:08:10 2026 GMT issuer=CN = openshift-service-serving-signer@1674482820 subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc certName: serving-cert-32 notBefore=Feb 22 14:08:09 2024 GMT notAfter=Feb 21 14:08:10 2026 GMT issuer=CN = openshift-service-serving-signer@1674482820 subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc certName: serving-cert-33 notBefore=Feb 22 14:08:09 2024 GMT notAfter=Feb 21 14:08:10 2026 GMT issuer=CN = openshift-service-serving-signer@1674482820 subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc certName: serving-cert-4 notBefore=Jan 23 14:07:17 2023 GMT notAfter=Jan 22 14:07:18 2025 GMT issuer=CN = openshift-service-serving-signer@1674482820 subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc certName: serving-cert-5 notBefore=Jan 23 14:07:17 2023 GMT notAfter=Jan 22 14:07:18 2025 GMT issuer=CN = openshift-service-serving-signer@1674482820 subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc certName: serving-cert-6 notBefore=Jan 23 14:07:17 2023 GMT notAfter=Jan 22 14:07:18 2025 GMT issuer=CN = openshift-service-serving-signer@1674482820 subject=CN = kube-controller-manager.openshift-kube-controller-manager.svc
Note the expired Secrets "serving-cert-2", "serving-cert-3", "serving-cert-4", "serving-cert-5", "serving-cert-6".
Customer would expect that these are deleted (as they are on most of their clusters, but not on two clusters).
Workaround is to manually remove the expired certificates.
Version-Release number of selected component (if applicable):
OpenShift Container Platform 4.15.42
Platform: vSphere
How reproducible:
Only reproducible on two customer clusters (out of 15 clusters)
Cluster install dates: 2023-01-23 (4.10.47) and 2024-03-27 (4.12.45)
Steps to Reproduce:
- ** Install the cluster
- Leave the cluster running for at least 1 year
- Check the Secrets / certificates in "openshift-kube-controller-manager"
Actual results:
Some old (maybe even expired after 24 months) certificates are still present in "openshift-kube-controller-manager".
Expected results:
No old certificates are to be found
Additional info:
- Two must-gathers for both affected clusters are available in Support Case 04041046
- This issue was already discussed with vrutkovs@redhat.com here: https://redhat-internal.slack.com/archives/CB48XQ4KZ/p1737641824270049