Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-44883

Node CSR status remain in pending state

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • Yes
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          During the debugging of Network edge cases in profile `baremetalds-ipi-ovn-dualstack-primaryv6-f7`, it was identified pod are not coming up in a particular node. So curl and accessing the pod fails and test cases are falling.
      
      mjoseph@mjoseph-mac Downloads % oc debug node/worker-2.ostest.test.metalkube.orgStarting pod/worker-2ostesttestmetalkubeorg-debug-n4mvb ...
      To use host binaries, run `chroot /host`
      Pod IP: fd2e:6f44:5dd8:c956::19
      If you don't see a command prompt, try pressing enter.Removing debug pod ...
      Error from server: error dialing backend: remote error: tls: internal error
      
      
      Further checks reveal even though the node is ready it is not accessible and CSR status remaining pending
      
      mjoseph@mjoseph-mac Downloads % oc get csr
      NAME        AGE     SIGNERNAME                      REQUESTOR                                        REQUESTEDDURATION   CONDITION
      csr-472f9   2m33s   kubernetes.io/kubelet-serving   system:node:worker-2.ostest.test.metalkube.org   <none>              Pending
      csr-4hznt   94m     kubernetes.io/kubelet-serving   system:node:worker-2.ostest.test.metalkube.org   <none>              Pending
      csr-4ntwl   109m    kubernetes.io/kubelet-serving   system:node:worker-2.ostest.test.metalkube.org   <none>              Pending
      csr-5wm2r   33m     kubernetes.io/kubelet-serving   system:node:worker-2.ostest.test.metalkube.org   <none>              Pending
      csr-dwbm7   64m     kubernetes.io/kubelet-serving   system:node:worker-2.ostest.test.metalkube.org   <none>              Pending
      csr-jh882   18m     kubernetes.io/kubelet-serving   system:node:worker-2.ostest.test.metalkube.org   <none>              Pending
      csr-ljjbq   125m    kubernetes.io/kubelet-serving   system:node:worker-2.ostest.test.metalkube.org   <none>              Pending
      csr-rqfx9   79m     kubernetes.io/kubelet-serving   system:node:worker-2.ostest.test.metalkube.org   <none>              Pending
      csr-t7cs8   48m     kubernetes.io/kubelet-serving   system:node:worker-2.ostest.test.metalkube.org   <none>              Pending
      
      E1122 07:47:33.783432       1 controller.go:316] "Reconciler error" err="could not reconcile CSR: could not authorize CSR: exhausted all authorization methods: IP address '192.168.111.25' not in machine addresses: fd2e:6f44:5dd8:c956::19 fd00:1101::e789:5ddd:2def:2abd" controller="certificatesigningrequest" controllerGroup="certificates.k8s.io" controllerKind="CertificateSigningRequest" CertificateSigningRequest="csr-5wm2r" namespace="" name="csr-5wm2r" reconcileID="c38ed7df-82b7-42d6-ab76-bba838567ea0"
      I1122 07:48:45.710911       1 controller.go:120] Reconciling CSR: csr-t7cs8
      I1122 07:48:45.728571       1 csr_check.go:163] csr-t7cs8: CSR does not appear to be client csr
      I1122 07:48:45.730826       1 csr_check.go:565] retrieving serving cert from worker-2.ostest.test.metalkube.org ([fd2e:6f44:5dd8:c956::19]:10250)
      I1122 07:48:45.732305       1 csr_check.go:188] Failed to retrieve current serving cert: remote error: tls: internal error
      I1122 07:48:45.732331       1 csr_check.go:208] Falling back to machine-api authorization for worker-2.ostest.test.metalkube.org
      E1122 07:48:45.732343       1 csr_check.go:426] csr-t7cs8: IP address '192.168.111.25' not in machine addresses: fd2e:6f44:5dd8:c956::19 fd00:1101::e789:5ddd:2def:2abd
      I1122 07:48:45.732353       1 csr_check.go:211] Could not use Machine for serving cert authorization: IP address '192.168.111.25' not in machine addresses: fd2e:6f44:5dd8:c956::19 fd00:1101::e789:5ddd:2def:2abd
      
      $ oc get machine ostest-c7r45-worker-0-kbr8r -o yaml
      status:
        addresses:
        - address: fd2e:6f44:5dd8:c956::19
          type: InternalIP
        - address: fd00:1101::e789:5ddd:2def:2abd
          type: InternalIP
        - address: worker-2.ostest.test.metalkube.org
          type: Hostname
        - address: worker-2.ostest.test.metalkube.org
          type: InternalDNS 

      Version-Release number of selected component (if applicable):

          

      How reproducible:

          

      Steps to Reproduce:

          1. Install a cluster with `baremetalds-ipi-ovn-dualstack-primaryv6-f7` profile 
          2. try to debug into every node, one will be not accessible
          3.
          

      Actual results:

       Pod cannot be deploy able on a particular node 

      Expected results:

         All nodes are accessible for deployment

      Additional info:

          

              bnemec@redhat.com Benjamin Nemec
              rhn-support-mjoseph Melvin Joseph
              None
              None
              Ross Brattain Ross Brattain
              None
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated: