Pods randomly failing to create with error "Multi-Attach error for volume "pvc-xxx" Volume is already exclusively attached to one node and can't be attached to another node

When pod is scheduled on another node, It's failing to create with error "Multi-Attach error for volume "pvc-xxx" Volume is already exclusively attached to one node and can't be attached to another"

Pod should be scheduled on another node without any error

-- Volume access mode is set to RWO

(ReadWriteOnce : the volume can be mounted as read-write by a single node. ReadWriteOnce access mode still can allow multiple pods to access the volume when the pods are running on the same node)


-- Also they are using STS

(When a StatefulSet's .spec.updateStrategy.type is set to RollingUpdate, the StatefulSet controller will delete and recreate each Pod in the StatefulSet)


-- application creation workflow

the helm chart of this pvc is as shown below.
kind: PersistentVolumeClaim
  name: fm-storage
  namespace: cran1
spec:
  accessModes:
  - ReadWriteOnce
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: fm-storage-snapshot
  dataSourceRef:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: fm-storage-snapshot
  resources:
    requests:
      storage: 1Gi
  storageClassName: ocs-storagecluster-cephfs
  volumeMode: Filesystem
  volumeName: pvc-*******

the general application creation process is as below.

step 1: create PVC helm release a
step 2: create CNF helm release b including all PODs

When deleting release a, the volume snapshots of PVCs are created accordingly. When create release a, the PVCs are restored from volumesnapshot if they exist.

the issue recently happened for pod  "po-cran6-rcpfm-0". It is assigned to "worker8" and the pv associated with the pod is "pvc-d5f2c46d-49b5-4ce2-af92-d183377fea50" in this latest issue only CNF was restarted, it means PODs were recreated and PVC should not be recreated.


Events:
  Type     Reason              Age                   From                     Message
  ----     ------              ----                  ----                     -------
  Normal   Scheduled           164m                  default-scheduler        Successfully assigned cran6/po-cran6-rcpfm-0 to worker8.hzdc-pz-10-110-10-98.ocp.hz.nsn-rdnet.net
  Warning  FailedAttachVolume  164m                  attachdetach-controller  Multi-Attach error for volume "pvc-d5f2c46d-49b5-4ce2-af92-d183377fea50" Volume is already exclusively attached to one node and can't be attached to another

PVC :- 

pvc-d5f2c46d-49b5-4ce2-af92-d183377fea50  1Gi       RWO           Delete          Bound   cran6/fm-storage                                              ocs-storagecluster-cephfs            6h17m


This issue may have occurred as a result of a previous instance of a pod not having been successfully terminated prior to the new pod attempting to attach.