Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-59656

LSO diskmaker-manager Pods Stuck in ImagePullBackOff After Creating LocalVolumeDiscovery

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • s390x
    • None
    • None
    • Rejected
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      After installing the Local Storage Operator (LSO) using a custom CatalogSource with the image
      quay.io/openshift-release-dev/ocp-release-nightly:iib-int-index-art-operators-4.20, the diskmaker-manager pods created fail to start and enter the ImagePullBackOff state after a LocalVolumeDiscovery resource is created.    

      Version-Release number of selected component (if applicable):

      LSO CSV version is 4.20.0-202507220613    

      How reproducible:

      Everytime    

      Steps to Reproduce:

          1. Bring up the catalogsource with image - quay.io/openshift-release-dev/ocp-release-nightly:iib-int-index-art-operators-4.20.
      
          2. Label the fist three worker nodes to be used as storage nodes, use below command
      
      echo "Label first three worker nodes as storage nodes."
      for NODE in `oc get no -l node-role.kubernetes.io/worker= -o name | cut -c 6- | head -3`;do
        oc label node $NODE cluster.ocs.openshift.io/openshift-storage= --overwrite
      done
      
          3. Install the LSO operator using the created CatalogSource.
      
          4. Create a LocalVolumeDiscovery CustomResource, using below command
      oc apply -f - <<EOF
      ---
      apiVersion: local.storage.openshift.io/v1alpha1
      kind: LocalVolumeDiscovery
      metadata:
        name: auto-discover-devices
        namespace: openshift-local-storage
      spec:
        nodeSelector:
          nodeSelectorTerms:
          - matchExpressions:
            - key: cluster.ocs.openshift.io/openshift-storage
              operator: In
              values:
              - ''
      EOF     
      
          5. The diskmaker-manager pods will get stuck in imagePullBackoff state.
      
      Error seen in pod description,
      Failed to pull image "registry.redhat.io/openshift4/ose-local-storage-diskmaker-rhel9@sha256:a44fa15e5641630a73680637b853188197f16bcfe0afbf4bb2606b5f193a70c7": unable to try pulling possible OCI artifact: get manifest: build image source: (Mirrors also failed: [brew.registry.redhat.io/openshift4/ose-local-storage-diskmaker-rhel9@sha256:a44fa15e5641630a73680637b853188197f16bcfe0afbf4bb2606b5f193a70c7: reading manifest sha256:a44fa15e5641630a73680637b853188197f16bcfe0afbf4bb2606b5f193a70c7 in brew.registry.redhat.io/openshift4/ose-local-storage-diskmaker-rhel9: name unknown: Digest not found]): registry.redhat.io/openshift4/ose-local-storage-diskmaker-rhel9@sha256:a44fa15e5641630a73680637b853188197f16bcfe0afbf4bb2606b5f193a70c7: reading manifest sha256:a44fa15e5641630a73680637b853188197f16bcfe0afbf4bb2606b5f193a70c7 in registry.redhat.io/openshift4/ose-local-storage-diskmaker-rhel9: manifest unknown

      Actual results:

      The diskmaker-manager pods are stuck in ImagePullBackOff state
      # oc get pods -n openshift-local-storage
      NAME                                      READY   STATUS             RESTARTS   AGE
      diskmaker-discovery-cs454                 1/2     ImagePullBackOff   0          6s
      diskmaker-discovery-pp6xl                 1/2     ImagePullBackOff   0          6s
      diskmaker-discovery-ss86j                 1/2     ErrImagePull       0          6s
      local-storage-operator-7d58f588c5-2ztcw   1/1     Running            0          36s

       

      Expected results:

      The diskmaker-manager pods should be up and running. Tested with LSO version 4.19.0-202507081507. The diskmaker-manager pods are up and running.
      
      # oc get pods -n openshift-local-storage 
      NAME                                      READY   STATUS    RESTARTS   AGE 
      diskmaker-discovery-7g5j5                 2/2     Running   0          36m 
      diskmaker-discovery-s5f4n                 2/2     Running   0          36m 
      diskmaker-discovery-xvnp7                 2/2     Running   0          36m 
      diskmaker-manager-4l8lf                   2/2     Running   0          36m 
      diskmaker-manager-6lhwl                   2/2     Running   0          36m 
      diskmaker-manager-fhcvp                   2/2     Running   0          36m 
      local-storage-operator-69d6b8c458-r2hln   1/1     Running   0          37m

      Additional info:

          

              sjainker@redhat.com Suman Jainkeri
              sjainker@redhat.com Suman Jainkeri
              None
              None
              Wei Duan Wei Duan
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 hour
                  1h