Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-56005

A pod with a volume attached isn't ready when there is a mismatch between Compute AZ and Volume AZ

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • ShiftStack Sprint 272
    • 1
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      Having the following Openstack AZs
      [stack@undercloud-0 ~]$ openstack availability zone list --volume
      +-----------+-------------+
      | Zone Name | Zone Status |
      +-----------+-------------+
      | nova      | available   |
      | AZ-0      | available   |
      | AZ-1      | available   |
      | AZ-2      | available   |
      +-----------+-------------+
      [stack@undercloud-0 ~]$ openstack availability zone list --compute
      +-----------+-------------+
      | Zone Name | Zone Status |
      +-----------+-------------+
      | AZ-0      | available   |
      | AZ-2      | available   |
      | internal  | available   |
      | my-AZ     | available   |
      +-----------+-------------+
      
      Pod is unable to get scheduled
      Name:             foo
      Namespace:        foo
      Priority:         0
      Service Account:  default
      Node:             <none>
      Labels:           <none>
      Annotations:      openshift.io/scc: insights-runtime-extractor-scc
                        seccomp.security.alpha.kubernetes.io/pod: runtime/default
      Status:           Pending
      SeccompProfile:   RuntimeDefault
      IP:
      IPs:              <none>
      Containers:
        test:
          Image:           registry.k8s.io/e2e-test-images/agnhost:2.53
          Port:            8080/TCP
          Host Port:       0/TCP
          SeccompProfile:  RuntimeDefault
          Args:
            netexec
            --http-port=8080
          Environment:  <none>
          Mounts:
            /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-44lp5 (ro)
            data from data-volume (rw)
      Conditions:
        Type           Status
        PodScheduled   False
      Volumes:
        data-volume:
          Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
          ClaimName:  foo
          ReadOnly:   false
        kube-api-access-44lp5:
          Type:                    Projected (a volume that contains injected data from multiple sources)
          TokenExpirationSeconds:  3607
          ConfigMapName:           kube-root-ca.crt
          ConfigMapOptional:       <nil>
          DownwardAPI:             true
          ConfigMapName:           openshift-service-ca.crt
          ConfigMapOptional:       <nil>
      QoS Class:                   BestEffort
      Node-Selectors:              <none>
      Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
      Events:
        Type     Reason            Age                    From               Message
        ----     ------            ----                   ----               -------
        Warning  FailedScheduling  12m                    default-scheduler  running PreBind plugin "VolumeBinding": binding volumes: pv "pvc-1c86a922-4f41-4578-8e12-6f3b56f75f42" node affinity doesn't match node "ostest-f8h2w-worker-0-b9kkt": no matching NodeSelectorTerms
        Warning  FailedScheduling  12m                    default-scheduler  0/6 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 3 node(s) had volume node affinity conflict. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.
        Warning  FailedScheduling  2m25s (x2 over 7m25s)  default-scheduler  0/6 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 3 node(s) had volume node affinity conflict. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.
      
      The PV:
      apiVersion: v1
      items:
      - apiVersion: v1
        kind: PersistentVolume
        metadata:
          annotations:
            pv.kubernetes.io/provisioned-by: cinder.csi.openstack.org
            volume.kubernetes.io/provisioner-deletion-secret-name: ""
            volume.kubernetes.io/provisioner-deletion-secret-namespace: ""
          creationTimestamp: "2025-05-07T07:27:58Z"
          finalizers:
          - external-provisioner.volume.kubernetes.io/finalizer
          - kubernetes.io/pv-protection
          name: pvc-1c86a922-4f41-4578-8e12-6f3b56f75f42
          resourceVersion: "301583"
          uid: 441c6a0f-28e8-4fbe-9b26-da714b35c700
        spec:
          accessModes:
          - ReadWriteOnce
          capacity:
            storage: 1Gi
          claimRef:
            apiVersion: v1
            kind: PersistentVolumeClaim
            name: foo
            namespace: foo
            resourceVersion: "301576"
            uid: 1c86a922-4f41-4578-8e12-6f3b56f75f42
          csi:
            driver: cinder.csi.openstack.org
            fsType: ext4
            volumeAttributes:
              storage.kubernetes.io/csiProvisionerIdentity: 1746539712338-8093-cinder.csi.openstack.org
            volumeHandle: 4fc9c57a-8c06-4873-bda6-de4440e7b17e
          nodeAffinity:
            required:
              nodeSelectorTerms:
              - matchExpressions:
                - key: topology.cinder.csi.openstack.org/zone
                  operator: In
                  values:
                  - nova
          persistentVolumeReclaimPolicy: Delete
          storageClassName: standard-csi
          volumeMode: Filesystem
        status:
          lastPhaseTransitionTime: "2025-05-07T07:27:58Z"
          phase: Bound
      kind: List
      metadata:
      
      There is a conflict between the zone set in the PV and the one set in the nodes(s)
      
      

      Version-Release number of selected component (if applicable):

          4.19.0-0.nightly-2025-04-24-005837

      How reproducible:

          

      Steps to Reproduce:

          1. Install Openshift on Openstack with a mismatch between Cinder and Nova AZ
          2. Create a pod with a volume attached
          3.
          

      Actual results:

          Pod is not scheduled

      Expected results:

       Pod is scheduled and running   

      Additional info:

          

              sfinucan@redhat.com Stephen Finucane
              itbrown@redhat.com Itshak Brown
              None
              None
              Itshak Brown Itshak Brown
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: