Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-67165

Azure CSI Stray Volume Attachment prevents pod starts

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.18.z
    • Storage
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      $ oc describe pod -n argos prometheus-argos-1 | tail -5
      Events:
        Type     Reason              Age   From                     Message
        ----     ------              ----  ----                     -------
        Normal   Scheduled           85m   default-scheduler        Successfully assigned argos/prometheus-argos-1 to xx-xx-xx-w6n7r-worker-hb27h
        Warning  FailedAttachVolume  85m   attachdetach-controller  Multi-Attach error for volume "pvc-230bbeda-df2e-48d4-8b99-dc9d9bc9e3c1" Volume is already exclusively attached to one node and can't be attached to another
      
      however
      pvc-230bbeda-df2e-48d4-8b99-dc9d9bc9e3c1
      is unattached.
      
      We find that there are 2 volume attachment resources pointing at the same node, same lun:
      $ oc get volumeattachment csi-9826d6cbe313fe82f2a8bc00be1bb25aa3167319eded23802cce46f6c42e5d06 csi-8d9cc02b65e4800f6973f514f6c0cad32f893697c07173331fb0c7da62e9094c -o yaml
      apiVersion: v1
      items:
      - apiVersion: storage.k8s.io/v1
        kind: VolumeAttachment
        metadata:
          annotations:
            csi.alpha.kubernetes.io/node-id: xx-xx-xx-worker-e16as-westeurope1-qqnfm
          creationTimestamp: "2025-11-05T23:30:10Z"
          finalizers:
          - external-attacher/disk-csi-azure-com
          name: csi-9826d6cbe313fe82f2a8bc00be1bb25aa3167319eded23802cce46f6c42e5d06
          resourceVersion: "354371758"
          uid: 11e93a17-8873-4b68-9168-83ffa25dbe29
        spec:
          attacher: disk.csi.azure.com
          nodeName: xx-xx-xx-worker-e16as-westeurope1-qqnfm
          source:
            persistentVolumeName: pvc-94aa38d1-f83e-48f2-80e8-6c03839c756c
        status:
          attached: true
          attachmentMetadata:
            LUN: "0"
      - apiVersion: storage.k8s.io/v1
        kind: VolumeAttachment
        metadata:
          annotations:
            csi.alpha.kubernetes.io/node-id: xx-xx-xx-worker-e16as-westeurope1-qqnfm
          creationTimestamp: "2025-11-05T23:02:38Z"
          finalizers:
          - external-attacher/disk-csi-azure-com
          name: csi-8d9cc02b65e4800f6973f514f6c0cad32f893697c07173331fb0c7da62e9094c
          resourceVersion: "354327328"
          uid: 4b1ce05a-2a39-4502-9385-e305530533ad
        spec:
          attacher: disk.csi.azure.com
          nodeName: tst-we-int08a-w6n7r-worker-e16as-westeurope1-qqnfm
          source:
            persistentVolumeName: pvc-230bbeda-df2e-48d4-8b99-dc9d9bc9e3c1
        status:
          attached: true
          attachmentMetadata:
            LUN: "0"
      kind: List
      metadata:
        resourceVersion: ""
          

      Version-Release number of selected component (if applicable):

      OCP 4.18.24
          

      How reproducible:

      Reccurence but not predictable
          

      Steps to Reproduce:

          1. Install OCP 4.18 IPI cluster on Azure with azuredisk CSI driver
          2. Create a pod with persistent volume claim using azuredisk
          3. Reboot the node where the pod is scheduled
          4. Try to start the pod again
          

      Actual results:

          Pod nay fail to start with Multi-Attach error, even though the volume is not attached to any node.
          

      Expected results:

          Pod should start successfully after node reboot.
          

      Additional info:

          

              hekumar@redhat.com Hemant Kumar
              rh-support-fgrosjea Franck Grosjean
              None
              None
              Wei Duan Wei Duan
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: