Details
-
Bug
-
Resolution: Obsolete
-
Normal
-
None
-
4.12
-
No
-
False
-
-
CU is currently in a good state post manual patching. Desire is to prevent this in the future
Description
Description of problem: CU noticed the API CO was degraded. Upon investigation, the kube-apiserver had logged;
oc logs kube-apiserver-mxq1490ry0 -c kube-apiserver | tail
2023-05-16T23:43:13.995169041Z I0516 23:43:13.995152 18 server.go:203] "Golang settings" GOGC="100" GOMAXPROCS="" GOTRACEBACK=""
2023-05-16T23:43:13.995579362Z I0516 23:43:13.995554 18 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key"
2023-05-16T23:43:13.995711508Z I0516 23:43:13.995693 18 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key"
2023-05-16T23:43:13.995933009Z I0516 23:43:13.995910 18 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key"
2023-05-16T23:43:13.996151238Z I0516 23:43:13.996133 18 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key"
2023-05-16T23:43:13.996364354Z I0516 23:43:13.996344 18 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key"
2023-05-16T23:43:13.996548985Z I0516 23:43:13.996532 18 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key"
2023-05-16T23:43:13.996663606Z E0516 23:43:13.996646 18 run.go:74] "command failed" err="failed to load SNI cert and key: tls: failed to find any PEM data in certificate input"
2023-05-16T23:43:13.998750266Z I0516 23:43:13.998718 1 main.go:235] Termination finished with exit code 1
2023-05-16T23:43:13.998765467Z I0516 23:43:13.998744 1 main.go:188] Deleting termination lock file "/var/log/kube-apiserver/.terminating"
Provided https://access.redhat.com/solutions/6988698 to resolve the issue which required manual patch of kube-apiserver;
oc patch kubeapiserver/cluster --type merge -p "{\"spec\":{\"forceRedeploymentReason\":\"Forcing new revision with random number $RANDOM to make message unique\"}}"
–
Version-Release number of selected component (if applicable):{code:none}
How reproducible: Limited/Difficult
Steps to Reproduce:{code:none} 1. Would need a cluster with kube-apiserver certificates about to expire
Actual results: Auto-update of the certificates failed causing the API CO to go in to a degraded state.
Expected results: kube-apiserver certificate auto-rotation completes and is un-noticed
Additional info: related SF ticket: 03514970