Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-46465

Cannot access external network via https from the HCP openshift-apiserver component

XMLWordPrintable

    • None
    • False
    • Hide

      None

      Show
      None
    • Hide
      Cause: Create a hosted cluster that uses the no-egress feature in ROSA and uses a container registry accessible via VPC endpoint.
      Consequence: Cluster fails to complete install because imagestreams that use the container registry cannot be resolved.
      Fix: The konnectivity proxy used by the openshift-apiserver in the control plane was resolving registry names with cloud api suffix on the control plane side and then attempting to access them through the data plane. The fix is to resolve and route hostnames with cloud suffixes consistently.
      Result: After the fix the no-egress cluster can resolve imagestreams and complete installation successfully.
      Show
      Cause: Create a hosted cluster that uses the no-egress feature in ROSA and uses a container registry accessible via VPC endpoint. Consequence: Cluster fails to complete install because imagestreams that use the container registry cannot be resolved. Fix: The konnectivity proxy used by the openshift-apiserver in the control plane was resolving registry names with cloud api suffix on the control plane side and then attempting to access them through the data plane. The fix is to resolve and route hostnames with cloud suffixes consistently. Result: After the fix the no-egress cluster can resolve imagestreams and complete installation successfully.
    • Bug Fix
    • In Progress

      This is a clone of issue OCPBUGS-46464. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-44314. The following is the description of the original issue:
      —
      Description of problem:

      Trying to setup a disconnected HCP cluster with self-managed image registry.

After the cluster installed, all the imagestream failed to import images.
With error:
```
Get "https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/": http: server gave HTTP response to HTTPS client
```

The imagestream will talk to openshift-apiserver and get the image target there.

After login to the hcp namespace, figured out that I cannot access any external network with https protocol.

      Version-Release number of selected component (if applicable):

      4.14.35

      How reproducible:

          always

      Steps to Reproduce:

          1. Install the hypershift hosted cluster with above setup
    2. The cluster can be created successfully and all the pods on the cluster can be running with the expected images pulled
    3. Check the internal image-registry
    4. Check the openshift-apiserver pod from management cluster

      Actual results:

      All the imagestreams failed to sync from the remote registry.
$ oc describe is cli -n openshift
Name:            cli
Namespace:        openshift
Created:        6 days ago
Labels:            <none>
Annotations:        include.release.openshift.io/ibm-cloud-managed=true
            include.release.openshift.io/self-managed-high-availability=true
            openshift.io/image.dockerRepositoryCheck=2024-11-06T22:12:32Z
Image Repository:    image-registry.openshift-image-registry.svc:5000/openshift/cli
Image Lookup:        local=false
Unique Images:        0
Tags:            1latest
  updates automatically from registry quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d  ! error: Import failed (InternalError): Internal error occurred: [122610517469.dkr.ecr.us-west-2.amazonaws.com/ocp-mirror-1@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d: Get "https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/": http: server gave HTTP response to HTTPS client, 122610517469.dkr.ecr.us-west-2.amazonaws.com/ocp-mirror-2@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d: Get "https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/": http: server gave HTTP response to HTTPS client, 122610517469.dkr.ecr.us-west-2.amazonaws.com/ocp-mirror-3@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d: Get "https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/": http: server gave HTTP response to HTTPS client, 122610517469.dkr.ecr.us-west-2.amazonaws.com/ocp-mirror-4@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d: Get "https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/": http: server gave HTTP response to HTTPS client, 122610517469.dkr.ecr.us-west-2.amazonaws.com/ocp-mirror-5@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d: Get "https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/": http: server gave HTTP response to HTTPS client, quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d: Get "https://quay.io/v2/": http: server gave HTTP response to HTTPS client]


Access the external network from the openshift-apiserver pod:
sh-5.1$ curl --connect-timeout 5 https://quay.io/v2
curl: (28) Operation timed out after 5001 milliseconds with 0 out of 0 bytes received
sh-5.1$ curl --connect-timeout 5 https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/
curl: (28) Operation timed out after 5001 milliseconds with 0 out of 0 bytes received

sh-5.1$ env | grep -i http.*proxy
HTTPS_PROXY=http://127.0.0.1:8090
HTTP_PROXY=http://127.0.0.1:8090

      Expected results:

      The openshift-apiserver should be able to talk to the remote https services.

      Additional info:

      It is working after set the registry to no_proxy

sh-5.1$ NO_PROXY=122610517469.dkr.ecr.us-west-2.amazonaws.com curl --connect-timeout 5 https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/
Not Authorized

       

              cewong@redhat.com Cesar Wong
              openshift-crt-jira-prow OpenShift Prow Bot
              Jie Zhao Jie Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: