The customer is having issues with multiple nodes where the PVs that have been attached by a pod at some point keep attached to the node even though it shows as "Released". This hampers the ability to attach new PVs to the nodes as new pods are deployed.

4.14.38

Currently ongoing

N/A

N/A

PVs are deleted once released.

summary
ARO SRE informed they think there is a bug to investigate on this scenario.

"At this point I cannot find any issue with the cluster or any functions that the SRE team can help with at this point since the customer does not want us to take corrective action and needs RCA.  It appears to be some sort of upstream bug that OCP would need to investigate.  I am going to mitigate this ticket at this point.  Feel free to reactive if more is needed from the SRE Team."

SRE Notes:FYI, here is one of the PV with a status of "Released", that is not being deleted
# oc get  persistentvolume/pvc-e785e15e-91cf-4f74-8be9-94ae2f6b1be1 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: disk.csi.azure.com
    volume.kubernetes.io/provisioner-deletion-secret-name: ""
    volume.kubernetes.io/provisioner-deletion-secret-namespace: ""
  creationTimestamp: "2025-04-10T03:59:57Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/disk-csi-azure-com
  name: pvc-e785e15e-91cf-4f74-8be9-94ae2f6b1be1
  resourceVersion: "871138589"
  uid: 8e46919c-8ae0-4c31-9bd4-4108cc97a100
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: rabbit-log-59c03fb35ed19951a40ba993420e3648
    namespace: atmes-ats-tsqa
    resourceVersion: "871108208"
    uid: e785e15e-91cf-4f74-8be9-94ae2f6b1be1
  csi:
    driver: disk.csi.azure.com
    fsType: xfs
    volumeAttributes:
      csi.storage.k8s.io/pv/name: pvc-e785e15e-91cf-4f74-8be9-94ae2f6b1be1
      csi.storage.k8s.io/pvc/name: rabbit-log-59c03fb35ed19951a40ba993420e3648
      csi.storage.k8s.io/pvc/namespace: atmes-ats-tsqa
      fsType: xfs
      kind: Managed
      requestedsizegib: "1"
      storage.kubernetes.io/csiProvisionerIdentity: 1731531988827-8656-disk.csi.azure.com
      storageaccounttype: Premium_LRS
    volumeHandle: /subscriptions/d7584753-660e-4684-9cbf-33411d2e76df/resourceGroups/rg-prod-sea-mgmt-aro/providers/Microsoft.Compute/disks/pvc-e785e15e-91cf-4f74-8be9-94ae2f6b1be1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.disk.csi.azure.com/zone
          operator: In
          values:
          - southeastasia-2
  persistentVolumeReclaimPolicy: Delete
  storageClassName: blocksc-xfs-test
  volumeMode: Filesystem
status:
  phase: Released