Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14037

Auto-rotation of Kube-apiserver cert/key failed, required manual patch

    XMLWordPrintable

Details

    • No
    • False
    • Hide

      None

      Show
      None
    • CU is currently in a good state post manual patching. Desire is to prevent this in the future

    Description

      Description of problem: CU noticed the API CO was degraded. Upon investigation, the kube-apiserver had logged;

      oc logs kube-apiserver-mxq1490ry0 -c kube-apiserver | tail
      2023-05-16T23:43:13.995169041Z I0516 23:43:13.995152 18 server.go:203] "Golang settings" GOGC="100" GOMAXPROCS="" GOTRACEBACK=""
      2023-05-16T23:43:13.995579362Z I0516 23:43:13.995554 18 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key"
      2023-05-16T23:43:13.995711508Z I0516 23:43:13.995693 18 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key"
      2023-05-16T23:43:13.995933009Z I0516 23:43:13.995910 18 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key"
      2023-05-16T23:43:13.996151238Z I0516 23:43:13.996133 18 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key"
      2023-05-16T23:43:13.996364354Z I0516 23:43:13.996344 18 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key"
      2023-05-16T23:43:13.996548985Z I0516 23:43:13.996532 18 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key"
      2023-05-16T23:43:13.996663606Z E0516 23:43:13.996646 18 run.go:74] "command failed" err="failed to load SNI cert and key: tls: failed to find any PEM data in certificate input"
      2023-05-16T23:43:13.998750266Z I0516 23:43:13.998718 1 main.go:235] Termination finished with exit code 1
      2023-05-16T23:43:13.998765467Z I0516 23:43:13.998744 1 main.go:188] Deleting termination lock file "/var/log/kube-apiserver/.terminating"

      Provided https://access.redhat.com/solutions/6988698 to resolve the issue which required manual patch of kube-apiserver;

      oc patch kubeapiserver/cluster --type merge -p "{\"spec\":{\"forceRedeploymentReason\":\"Forcing new revision with random number $RANDOM to make message unique\"}}"

      Version-Release number of selected component (if applicable):{code:none}
      
      

      How reproducible: Limited/Difficult

      Steps to Reproduce:{code:none}
      1. Would need a cluster with kube-apiserver certificates about to expire
      

      Actual results: Auto-update of the certificates failed causing the API CO to go in to a degraded state.

      Expected results: kube-apiserver certificate auto-rotation completes and is un-noticed
      
      

      Additional info: related SF ticket: 03514970

      
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            dacarpen@redhat.com Darren Carpenter
            Ke Wang Ke Wang
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: