Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-7044

Nodes unready in PublicAndPrivate / Private Hypershift setups behind a proxy

    XMLWordPrintable

Details

    • Important
    • SDN Sprint 231, SDN Sprint 232
    • 2
    • Proposed
    • False
    • Hide

      None

      Show
      None

    Description

      This is a clone of issue OCPBUGS-6651. The following is the description of the original issue:

      Description of problem:

      When running a hypershift HostedCluster with a publicAndPrivate / private setup behind a proxy, Nodes never go ready.
      
      ovn-kubernetes pods fail to run because the init container fails.
      
      [root@ip-10-0-129-223 core]# crictl logs cf142bb9f427d
      + [[ -f /env/ ]]
      ++ date -Iseconds
      2023-01-25T12:18:46+00:00 - checking sbdb
      + echo '2023-01-25T12:18:46+00:00 - checking sbdb'
      + echo 'hosts: dns files'
      + proxypid=15343
      + ovndb_ctl_ssl_opts='-p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt'
      + sbdb_ip=ssl:ovnkube-sbdb.apps.agl-proxy.hypershift.local:9645
      + retries=0
      + ovn-sbctl --no-leader-only --timeout=5 --db=ssl:ovnkube-sbdb.apps.agl-proxy.hypershift.local:9645 -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt get-connection
      + exec socat TCP-LISTEN:9645,reuseaddr,fork PROXY:10.0.140.167:ovnkube-sbdb.apps.agl-proxy.hypershift.local:443,proxyport=3128
      ovn-sbctl: ssl:ovnkube-sbdb.apps.agl-proxy.hypershift.local:9645: database connection failed ()
      + ((  retries += 1  ))
      
      
      

      Version-Release number of selected component (if applicable):

      4.12

      How reproducible:

      Always.

      Steps to Reproduce:

      1. Create a publicAndPrivate hypershift HostedCluster behind a proxy. E.g"
      ➜  hypershift git:(main) ✗ ./bin/hypershift create cluster \
      aws --pull-secret ~/www/pull-secret-ci.txt \
      --ssh-key ~/.ssh/id_ed25519.pub \
      --name agl-proxy \
      --aws-creds ~/www/config/aws-osd-hypershift-creds \
      --node-pool-replicas=3 \
      --region=us-east-1 \
      --base-domain=agl.hypershift.devcluster.openshift.com \
      --zones=us-east-1a \
      --endpoint-access=PublicAndPrivate \
      --external-dns-domain=agl-services.hypershift.devcluster.openshift.com --enable-proxy=true
      
      2. Get the kubeconfig for the guest cluster. E.g
      kubectl get secret -nclusters agl-proxy-admin-kubeconfig  -oyaml
      
      3. Get pods in the guest cluster.
      See ovnkube-node pods init container failing with
      [root@ip-10-0-129-223 core]# crictl logs cf142bb9f427d
      + [[ -f /env/ ]]
      ++ date -Iseconds
      2023-01-25T12:18:46+00:00 - checking sbdb
      + echo '2023-01-25T12:18:46+00:00 - checking sbdb'
      + echo 'hosts: dns files'
      + proxypid=15343
      + ovndb_ctl_ssl_opts='-p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt'
      + sbdb_ip=ssl:ovnkube-sbdb.apps.agl-proxy.hypershift.local:9645
      + retries=0
      + ovn-sbctl --no-leader-only --timeout=5 --db=ssl:ovnkube-sbdb.apps.agl-proxy.hypershift.local:9645 -p /ovn-cert/tls.key -c /ovn-cert/tls.crt -C /ovn-ca/ca-bundle.crt get-connection
      + exec socat TCP-LISTEN:9645,reuseaddr,fork PROXY:10.0.140.167:ovnkube-sbdb.apps.agl-proxy.hypershift.local:443,proxyport=3128
      ovn-sbctl: ssl:ovnkube-sbdb.apps.agl-proxy.hypershift.local:9645: database connection failed ()
      + ((  retries += 1  ))
      
      

      To create a bastion an ssh into the Nodes See https://hypershift-docs.netlify.app/how-to/debug-nodes/

      Actual results:

      Nodes unready

      Expected results:

      Nodes go ready

      Additional info:

       

      Attachments

        Issue Links

          Activity

            People

              pdiak@redhat.com Patryk Diak
              openshift-crt-jira-prow OpenShift Prow Bot
              Jie Zhao Jie Zhao
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: