Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-23579

[MCE 2.9.1/2.10] catalog source have images with all zeros digest

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • Critical
    • None

      Description of the problem:
      MCE catalog source have images with all zeros digest
       

      How reproducible:
      5/5
       
      Steps to reproduce:

      1. Deploy Hub cluster

      2. Try to deploy MCE oprator build-101,v2.9.1,latest-2.9,2.9.1-DOWNSTREAM-2025-08-22-01-13-22

      Actual results:

       MCE pods are in ImagePullBackOff status.

      Expected results:

      MCE pods are installed successfully.

      Additional Info:
      I see the MCE pods are in ImagePullBackOff:

      [kni@ocp-edge72 ~]$ oc -n multicluster-engine get pods
NAME                                            READY   STATUS             RESTARTS   AGE
multicluster-engine-operator-795647b9fd-7scks   0/1     ImagePullBackOff   0          34m
multicluster-engine-operator-795647b9fd-888d6   0/1     ImagePullBackOff   0          34m

      In their logs I see this:

        Type     Reason          Age                 From               Message
  ----     ------          ----                ----               -------
  Normal   Scheduled       35m                 default-scheduler  Successfully assigned multicluster-engine/multicluster-engine-operator-795647b9fd-7scks to master-2
  Normal   AddedInterface  35m                 multus             Add eth0 [10.130.0.24/23] from ovn-kubernetes
  Normal   Pulling         32m (x5 over 35m)   kubelet            Pulling image "registry.redhat.io/multicluster-engine/backplane-rhel9-operator@sha256:0000000000000000000000000000000000000000000000000000000000000000"
  Warning  Failed          32m (x5 over 35m)   kubelet            Failed to pull image "registry.redhat.io/multicluster-engine/backplane-rhel9-operator@sha256:0000000000000000000000000000000000000000000000000000000000000000": initializing source docker://registry.redhat.io/multicluster-engine/backplane-rhel9-operator@sha256:0000000000000000000000000000000000000000000000000000000000000000: (Mirrors also failed: [quay.io:443/acm-d/backplane-rhel9-operator@sha256:0000000000000000000000000000000000000000000000000000000000000000: reading manifest sha256:0000000000000000000000000000000000000000000000000000000000000000 in quay.io:443/acm-d/backplane-rhel9-operator: manifest unknown]): registry.redhat.io/multicluster-engine/backplane-rhel9-operator@sha256:0000000000000000000000000000000000000000000000000000000000000000: reading manifest sha256:0000000000000000000000000000000000000000000000000000000000000000 in registry.redhat.io/multicluster-engine/backplane-rhel9-operator: manifest unknown
  Warning  Failed          32m (x5 over 35m)   kubelet            Error: ErrImagePull
  Normal   BackOff         0s (x153 over 35m)  kubelet            Back-off pulling image "registry.redhat.io/multicluster-engine/backplane-rhel9-operator@sha256:0000000000000000000000000000000000000000000000000000000000000000"
  Warning  Failed          0s (x153 over 35m)  kubelet            Error: ImagePullBackOff

      I see many images have digest of all zero's when I run:

      [kni@ocp-edge72 ~]$ oc -n multicluster-engine describe csv multicluster-engine.v2.9.1
                 Field Path:  metadata.namespace
                  Name:            OPERAND_IMAGE_ADDON_MANAGER
                  Value:           registry.redhat.io/multicluster-engine/addon-manager-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_BACKPLANE_MUST_GATHER
                  Value:           registry.redhat.io/multicluster-engine/must-gather-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_BACKPLANE_OPERATOR
                  Value:           registry.redhat.io/multicluster-engine/backplane-rhel9-operator@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_CLUSTER_API_PROVIDER_AGENT
                  Value:           registry.redhat.io/multicluster-engine/cluster-api-provider-agent-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_CLUSTER_API_PROVIDER_KUBEVIRT
                  Value:           registry.redhat.io/multicluster-engine/cluster-api-provider-kubevirt-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_CLUSTER_API_PROVIDER_OPENSHIFT_ASSISTED_BOOTSTRAP
                  Value:           registry.redhat.io/multicluster-engine/capoa-bootstrap-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_CLUSTER_API_PROVIDER_OPENSHIFT_ASSISTED_CONTROL_PLANE
                  Value:           registry.redhat.io/multicluster-engine/capoa-control-plane-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_CLUSTER_API_WEBHOOK_CONFIG_MCE_29
                  Value:           registry.redhat.io/multicluster-engine/mce-capi-webhook-config-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_CLUSTERCLAIMS_CONTROLLER
                  Value:           registry.redhat.io/multicluster-engine/clusterclaims-controller-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_CLUSTER_CURATOR_CONTROLLER
                  Value:           registry.redhat.io/multicluster-engine/cluster-curator-controller-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_CLUSTER_IMAGE_SET_CONTROLLER
                  Value:           registry.redhat.io/multicluster-engine/cluster-image-set-controller-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_CLUSTERLIFECYCLE_STATE_METRICS
                  Value:           registry.redhat.io/multicluster-engine/clusterlifecycle-state-metrics-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_CLUSTER_PROXY_ADDON
                  Value:           registry.redhat.io/multicluster-engine/cluster-proxy-addon-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_CLUSTER_PROXY
                  Value:           registry.redhat.io/multicluster-engine/cluster-proxy-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_CONSOLE_MCE
                  Value:           registry.redhat.io/multicluster-engine/console-mce-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_DISCOVERY_OPERATOR
                  Value:           registry.redhat.io/multicluster-engine/discovery-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_OPENSHIFT_HIVE
                  Value:           registry.redhat.io/multicluster-engine/hive-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_HYPERSHIFT_ADDON_OPERATOR
                  Value:           registry.redhat.io/multicluster-engine/hypershift-addon-rhel9-operator@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_HYPERSHIFT_CLI
                  Value:           registry.redhat.io/multicluster-engine/hypershift-cli-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_HYPERSHIFT_OPERATOR
                  Value:           registry.redhat.io/multicluster-engine/hypershift-rhel9-operator@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_IMAGE_BASED_INSTALL_OPERATOR
                  Value:           registry.redhat.io/multicluster-engine/image-based-install-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_KUBE_RBAC_PROXY_MCE
                  Value:           registry.redhat.io/multicluster-engine/kube-rbac-proxy-mce-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_MANAGEDCLUSTER_IMPORT_CONTROLLER
                  Value:           registry.redhat.io/multicluster-engine/managedcluster-import-controller-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_MANAGED_SERVICEACCOUNT
                  Value:           registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_MULTICLOUD_MANAGER
                  Value:           registry.redhat.io/multicluster-engine/multicloud-manager-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_PROVIDER_CREDENTIAL_CONTROLLER
                  Value:           registry.redhat.io/multicluster-engine/provider-credential-controller-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_PLACEMENT
                  Value:           registry.redhat.io/multicluster-engine/placement-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_REGISTRATION
                  Value:           registry.redhat.io/multicluster-engine/registration-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_REGISTRATION_OPERATOR
                  Value:           registry.redhat.io/multicluster-engine/registration-operator-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_WORK
                  Value:           registry.redhat.io/multicluster-engine/work-rhel9@sha256:0000000000000000000000000000000000000000000000000000000000000000
                  Name:            OPERAND_IMAGE_ASSISTED_IMAGE_SERVICE
                  Value:           registry.redhat.io/multicluster-engine/assisted-image-service-rhel9@sha256:6b95560a6a6dc1ddd41c76f275f83baec52ec9d79de04bfe54feaed02db123a3
                  Name:            OPERAND_IMAGE_ASSISTED_INSTALLER
                  Value:           registry.redhat.io/multicluster-engine/assisted-installer-rhel9@sha256:0a9c0ccb6a75273474fb04b785aa8fc58aec2f9ffbff8c232d359e14892bbae2
                  Name:            OPERAND_IMAGE_ASSISTED_INSTALLER_AGENT
                  Value:           registry.redhat.io/multicluster-engine/assisted-installer-agent-rhel9@sha256:1cc6fddaf9e55637e90d5c3e598467df8967b8a362f8b8d46286b256bad6dd3f
                  Name:            OPERAND_IMAGE_ASSISTED_INSTALLER_CONTROLLER
                  Value:           registry.redhat.io/multicluster-engine/assisted-installer-controller-rhel9@sha256:ac43feaf079fa0d250d53cd9685fb68677d888dd091730faa32757dcdf2c95e4
                  Name:            OPERAND_IMAGE_ASSISTED_SERVICE_8
                  Value:           registry.redhat.io/multicluster-engine/assisted-service-8-rhel8@sha256:3c450cb6a3ba6f2e3028856f14967803c1a691d95b8f54bd9d29d93691d6e795
                  Name:            OPERAND_IMAGE_ASSISTED_SERVICE_9
                  Value:           registry.redhat.io/multicluster-engine/assisted-service-9-rhel9@sha256:57699ecb529cb46843de847deef3e3757ca252910249d8deac16fcb00c01b636
                  Name:            OPERAND_IMAGE_OSE_CLUSTER_API_RHEL9
                  Value:           registry.redhat.io/openshift4/ose-cluster-api-rhel9@sha256:9a72b8081a57846d942981ced07da86ad8e197309a1bc3c9d53876528b58f8ec
                  Name:            OPERAND_IMAGE_OSE_AWS_CLUSTER_API_CONTROLLERS_RHEL9
                  Value:           registry.redhat.io/openshift4/ose-aws-cluster-api-controllers-rhel9@sha256:cf13b0658271cd0dbd1ce6dcc350e64c5459ef02ba4dc88a0031de9d6f26ffd1
                  Name:            OPERAND_IMAGE_OSE_BAREMETAL_CLUSTER_API_CONTROLLERS_RHEL9
                  Value:           registry.redhat.io/openshift4/ose-baremetal-cluster-api-controllers-rhel9@sha256:f855d5ce17d3ac4560afc73c35d03f045f84551fdccc00312c890288a889b9ce
                  Name:            OPERAND_IMAGE_POSTGRESQL_12
                  Value:           registry.redhat.io/rhel8/postgresql-12@sha256:4773a29755ea826c0d3b58a8b8297c1d7388019516195cae9bd815f28ec4bca0
                  Name:            OPERATOR_VERSION
                  Value:           2.9.1
                  Name:            OPERATOR_PACKAGE
                  Value:           multicluster-engine
                Image:             registry.redhat.io/multicluster-engine/backplane-rhel9-operator@sha256:0000000000000000000000000000000000000000000000000000000000000000

      must-gather: http://rhos-compute-node-10.lab.eng.rdu2.redhat.com/logs/MGMT-21596.tar.gz

              gparvin-redhat Gus Parvin
              rh-ee-ugreener Udi Greener
              Vladislav Kolodny Vladislav Kolodny
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: