Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-45703

Cert Manager http01 challenge fails to reach challenge pod

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 4.16.z
    • cert-manager
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • None
    • None
    • OAPE Sprint 267
    • 1
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      Attempting to get a certificate issued for the API server endpoint of the cluster. The http01 challenge never succeeds because the endpoint routes to the API VIP on control plane nodes but the challenge pod is behind the ingress routers on worker nodes.

      Following the procedure in the documentation.

      Create ClusterIssuer:

      ---
      apiVersion: cert-manager.io/v1
      kind: ClusterIssuer
      metadata:
        name: test-private-acme
      spec:
        acme:
          preferredChain: ""
          privateKeySecretRef:
            name: test-private-acme-key
          server: https://pebble-svc-pebble.apps.ran-vcl04.ptp.eng.rdu2.dc.redhat.com/dir
          # server PEM, base64 encoded
          caBundle: # <trimmed>
          solvers:
          - http01:
              ingress:
                class: openshift-default

      Create Certificate:

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: custom-api-cert
        namespace: openshift-config
      spec:
        isCA: false
        commonName: "api.cnfdf02.telco5gran.eng.rdu2.redhat.com"
        secretName: custom-api-cert
        dnsNames:
        - "api.cnfdf02.telco5gran.eng.rdu2.redhat.com"
        issuerRef:
          group: cert-manager.io
          kind: ClusterIssuer
          name: test-private-acme 

      The challenge pod and ingress/route get created:

      $ oc get pod -n openshift-config
      NAME                        READY   STATUS    RESTARTS   AGE
      cm-acme-http-solver-x2b5p   1/1     Running   0          19m
      
      $ oc get route -n openshift-config
      NAME                              HOST/PORT                                    PATH                                                                      SERVICES                    PORT   TERMINATION   WILDCARD
      cm-acme-http-solver-gdkpp-bgs7c   api.cnfdf02.telco5gran.eng.rdu2.redhat.com   /.well-known/acme-challenge/NO-Tz3_QE7woxJnY7IIktjQ01PaSaEynpArtRp9Cy_8   cm-acme-http-solver-x4dtx   http                 None
       

      Attempts to reach the challenge pod externally fail:

      $ curl -k api.cnfdf02.telco5gran.eng.rdu2.redhat.com/.well-known/acme-challenge/NO-Tz3_QE7woxJnY7IIktjQ01PaSaEynpArtRp9Cy_8
      curl: (7) Failed to connect to api.cnfdf02.telco5gran.eng.rdu2.redhat.com port 80 after 32 ms: Could not connect to server 

      However, if I update my /etc/hosts so that the api endpoint resolves to the cluster ingress VIP, I can reach the pod as expected:

      $ sudo vi /etc/host
      $ curl -k api.cnfdf02.telco5gran.eng.rdu2.redhat.com/.well-known/acme-challenge/NO-Tz3_QE7woxJnY7IIktjQ01PaSaEynpArtRp9Cy_8
      NO-Tz3_QE7woxJnY7IIktjQ01PaSaEynpArtRp9Cy_8.wntMpvkklAJLtbCkY9GJHbY5flyNG-M7NoMX0ySg5tI

       

      Version-Release number of selected component (if applicable):

      Cert manager: 1.14.1
      OCP: 4.16.15 

       

      How reproducible:

          100%

      Steps to Reproduce:

      see above    

      Actual results:

      No secret issued

      Expected results:

      Challenge completes and secret issued

      Additional info:

       

       

              bpalm@redhat.com Brandon Palm
              rhn-support-imiller Ian Miller
              None
              None
              Yuedong Wu Yuedong Wu
              None
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated: