Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-32220

HCP: hypershift-operator on disconnected clusters ignores RegistryOverrides inspecting the control-plane-operator-image (setting hypershift.openshift.io/control-plane-operator-image is a workaround)

XMLWordPrintable

    • No
    • Hypershift Sprint 252, Hypershift Sprint 253
    • 2
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the Hypershift Operator was not using the `RegistryOverrides` mechanism to inspect the image from the internal registry. With this release, the metadata inspector works as expected during the Hypershift Operator reconciliation, and the `OverrideImages` are properly populated. (link:https://issues.redhat.com/browse/OCPBUGS-32220[*OCPBUGS-32220*])
      Show
      * Previously, the Hypershift Operator was not using the `RegistryOverrides` mechanism to inspect the image from the internal registry. With this release, the metadata inspector works as expected during the Hypershift Operator reconciliation, and the `OverrideImages` are properly populated. (link: https://issues.redhat.com/browse/OCPBUGS-32220 [* OCPBUGS-32220 *])
    • Bug Fix
    • Done

      This is a clone of issue OCPBUGS-29494. The following is the description of the original issue:

      Description of problem:

          The hypershift operator ignores RegistryOverrides (from ICSP/IDMS) inspecting the control-plane-operator-image so on disconnected cluster the user should explicitly set hypershift.openshift.io/control-plane-operator-image annotation pointing to the mirrored image on the internal registry.
      
      Example:
      the correct match is in the IDMS:
      # oc get imagedigestmirrorset -oyaml | grep -B2 registry.ci.openshift.org/ocp/4.14-2024-02-14-135111
       ...
          - mirrors:
            - virthost.ostest.test.metalkube.org:5000/localimages/local-release-image
            source: registry.ci.openshift.org/ocp/4.14-2024-02-14-135111
      
      Creating an hosted cluster with:
      hcp create cluster kubevirt --image-content-sources /home/mgmt_iscp.yaml  --additional-trust-bundle /etc/pki/ca-trust/source/anchors/registry.2.crt --name simone3 --node-pool-replicas 2 --memory 16Gi --cores 4 --root-volume-size 64 --namespace local-cluster --release-image virthost.ostest.test.metalkube.org:5000/localimages/local-release-image@sha256:66c6a46013cda0ad4e2291be3da432fdd03b4a47bf13067e0c7b91fb79eb4539 --pull-secret /tmp/.dockerconfigjson --generate-ssh
      
      on the hostedCluster object we see:
      status:
        conditions:
        - lastTransitionTime: "2024-02-14T22:01:30Z"
          message: 'failed to look up image metadata for registry.ci.openshift.org/ocp/4.14-2024-02-14-135111@sha256:84c74cc05250d0e51fe115274cc67ffcf0a4ac86c831b7fea97e484e646072a6:
            failed to obtain root manifest for registry.ci.openshift.org/ocp/4.14-2024-02-14-135111@sha256:84c74cc05250d0e51fe115274cc67ffcf0a4ac86c831b7fea97e484e646072a6:
            unauthorized: authentication required'
          observedGeneration: 3
          reason: ReconciliationError
          status: "False"
          type: ReconciliationSucceeded
      
      
      and in the logs of the hypershift operator:
      {"level":"info","ts":"2024-02-14T22:18:11Z","msg":"registry override coincidence not found","controller":"hostedcluster","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedCluster","HostedCluster":{"name":"simone3","namespace":"local-cluster"},"namespace":"local-cluster","name":"simone3","reconcileID":"6d6a2479-3d54-42e3-9204-8d0ab1013745","image":"4.14-2024-02-14-135111"}
      {"level":"error","ts":"2024-02-14T22:18:12Z","msg":"Reconciler error","controller":"hostedcluster","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedCluster","HostedCluster":{"name":"simone3","namespace":"local-cluster"},"namespace":"local-cluster","name":"simone3","reconcileID":"6d6a2479-3d54-42e3-9204-8d0ab1013745","error":"failed to look up image metadata for registry.ci.openshift.org/ocp/4.14-2024-02-14-135111@sha256:84c74cc05250d0e51fe115274cc67ffcf0a4ac86c831b7fea97e484e646072a6: failed to obtain root manifest for registry.ci.openshift.org/ocp/4.14-2024-02-14-135111@sha256:84c74cc05250d0e51fe115274cc67ffcf0a4ac86c831b7fea97e484e646072a6: unauthorized: authentication required","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
      
      
      so the hypershift-operator is not using the RegistryOverrides mechanism to inspect the image from the internal registry (virthost.ostest.test.metalkube.org:5000/localimages/local-release-image in this example).
      
      Explicitly setting annotation:
      hypershift.openshift.io/control-plane-operator-image: virthost.ostest.test.metalkube.org:5000/localimages/local-release-image@sha256:84c74cc05250d0e51fe115274cc67ffcf0a4ac86c831b7fea97e484e646072a6
      on the hosted-cluster directly pointing to the mirrored control-plane-operator image is required to proceed on disconnected environments.
      
      

      Version-Release number of selected component (if applicable):

          4.14, 4.15, 4.16

      How reproducible:

          100%

      Steps to Reproduce:

          1. try to deploy an hostedCluster on a disconnected environment without explicitly set hypershift.openshift.io/control-plane-operator-image annotation.
          2.
          3.
          

      Actual results:

          A reconciliation error reported on the hostedCluster object:
      status:
        conditions:
        - lastTransitionTime: "2024-02-14T22:01:30Z"
          message: 'failed to look up image metadata for registry.ci.openshift.org/ocp/4.14-2024-02-14-135111@sha256:84c74cc05250d0e51fe115274cc67ffcf0a4ac86c831b7fea97e484e646072a6:
            failed to obtain root manifest for registry.ci.openshift.org/ocp/4.14-2024-02-14-135111@sha256:84c74cc05250d0e51fe115274cc67ffcf0a4ac86c831b7fea97e484e646072a6:
            unauthorized: authentication required'
          observedGeneration: 3
          reason: ReconciliationError
          status: "False"
          type: ReconciliationSucceeded
      
      The hostedCluster is not spawn.

      Expected results:

          The hypershift operator uses the RegistryOverrides mechanism also for the control-plane-operator image.
          Explicitly setting hypershift.openshift.io/control-plane-operator-image annotation is not required.

      Additional info:

          - Maybe related to OCPBUGS-29110
          - Explicitly setting hypershift.openshift.io/control-plane-operator-image annotation pointing to the mirrored image on the internal registry is a valid workaround.

            jparrill@redhat.com Juan Manuel Parrilla Madrid
            openshift-crt-jira-prow OpenShift Prow Bot
            Liangquan Li Liangquan Li
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: