Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-4075

kubelet-serving certs not auto aproved by machine-approver in case of libvirt IPI

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Note: I am not sure if `Installer/Single Node Openshift` is correct component because there is nothing for `machine-approver`, please change accordingly.

       

      Description of problem:

      As per documents https://docs.openshift.com/container-platform/4.11/backup_and_restore/control_plane_ba[...]and_restore/disaster_recovery/scenario-3-expired-certs.html (step-4) in case of IPI it should be auto approved but during libvirt IPI I am not seeing it.

      Version-Release number of selected component (if applicable):

      4.11.13

      How reproducible:

      Start a cluster and forcefully rotate the cert and you will see the CSR request for `node-bootstrapper` initially with need to approved manually and some time later `kubelet-serving` request comes which suppose to handle by machine-approver but it always remain in pending state until manually approved.

      Steps to Reproduce:

      1. Provision a cluster
      2. Force rotate certificate or shut down the cluster and start it in later time so cert rotation kicks in.
      3. Watch for CSR requests.
      

      Actual results:

      # oc get csr
      NAME                                             AGE     SIGNERNAME                                    REQUESTOR                                                                         REQUESTEDDURATION   CONDITION
      csr-5xvgq                                        4m51s   kubernetes.io/kubelet-serving                 system:node:crc-8psnr-master-0                                                    <none>              Pending
      csr-94c4v                                        24h     kubernetes.io/kubelet-serving                 system:node:crc-8psnr-master-0                                                    <none>              Approved,Issued
      csr-j6fbm                                        13m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
      csr-l8dtb                                        24h     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
      system:openshift:openshift-authenticator-kjlbq   24h     kubernetes.io/kube-apiserver-client           system:serviceaccount:openshift-authentication-operator:authentication-operator   <none>              Approved,Issued
      system:openshift:openshift-authenticator-mtfnw   3m58s   kubernetes.io/kube-apiserver-client           system:serviceaccount:openshift-authentication-operator:authentication-operator   <none>              Approved,Issued
      
      # oc logs machine-approver-797967c7fd-2pg6r -n openshift-cluster-machine-approver
      Defaulted container "kube-rbac-proxy" out of: kube-rbac-proxy, machine-approver-controller
      Error from server: Get "https://192.168.126.11:10250/containerLogs/openshift-cluster-machine-approver/machine-approver-797967c7fd-2pg6r/kube-rbac-proxy": remote error: tls: internal error

      Expected results:

      CSR should be approved automatically in case of IPI

      Additional info:

      Once csr approved manually I am able to get the logs from machine-approver pod and it looks like not sure why it saying the serving cert is not targeted for the respective node.
      
      ```
      I1124 05:06:22.912572       1 csr_check.go:182] Failed to retrieve current serving cert: remote error: tls: internal error
      I1124 05:06:22.912585       1 csr_check.go:202] Falling back to machine-api authorization for crc-8psnr-master-0
      E1124 05:06:22.912589       1 csr_check.go:360] csr-5xvgq: Serving Cert: No target machine for node "crc-8psnr-master-0"
      I1124 05:06:22.912595       1 csr_check.go:205] Could not use Machine for serving cert authorization: Unable to find machine for node
      I1124 05:06:22.914266       1 controller.go:233] csr-5xvgq: CSR not authorized
      ```

              romfreiman Rom Freiman
              prkumar@redhat.com Praveen Kumar
              Omri Hochman Omri Hochman
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: