Description of problem (please be detailed as possible and provide log
snippests):

Testing disaster recover with stretch cluster for OpenShift Data Foundation as defined here : https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.16/html/configuring_openshift_data_foundation_disaster_recovery_for_openshift_workloads/introduction-to-stretch-cluster-disaster-recovery_stretch-cluster

Installed a product with HA configuration and simulated disaster by shutting down ALL nodes for a Zone-2. After shutdown applied non-graceful node shutdown taint as defined here : https://kubernetes.io/blog/2023/08/16/kubernetes-1-28-non-graceful-node-shutdown-ga/ which triggered creation of a networkfence. The networkfence impacted recovery of application on Zone-1.

Network fence :
---------------
NAME DRIVER CIDRS FENCESTATE ...
xxxx openshift-storage.rbd.csi.ceph.com ["100.64.0.9/32"] Fenced ...

Datastore pods on surviving zone had issues mounting PVCs (read-only file system):

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 6m9s (x4 over 7m21s) kubelet Error: relabel failed /var/lib/kubelet/pods/8820f331-64bc-47d1-9140-05d963f43634/volumes/kubernetes.io~csi/pvc-f892de38-e7c8-418f-9d64-cc1e958ca209/mount: lsetxattr /var/lib/kubelet/pods/8820f331-64bc-47d1-9140-05d963f43634/volumes/kubernetes.io~csi/pvc-f892de38-e7c8-418f-9d64-cc1e958ca209/mount: read-only file system
Warning Failed 4m52s kubelet Error: relabel failed /var/lib/kubelet/pods/8820f331-64bc-47d1-9140-05d963f43634/volumes/kubernetes.io~csi/pvc-f892de38-e7c8-418f-9d64-cc1e958ca209/mount: lsetxattr /var/lib/kubelet/pods/8820f331-64bc-47d1-9140-05d963f43634/volumes/kubernetes.io~csi/pvc-f892de38-e7c8-418f-9d64-cc1e958ca209/mount/conf: read-only file system

Deleted datastore pod on surviving zone and when recreated saw following :

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
...
Warning FailedMount 21s (x9 over 2m29s) kubelet MountVolume.MountDevice failed for volume "pvc-5438fe65-a911-4226-9d1f-d3402af17cc9" : rpc error: code = Internal desc = error generating volume 0001-0011-openshift-storage-0000000000000001-71f3c614-f564-4814-8cbb-4a0247804fb7: rados: ret=-108, Cannot send after transport endpoint shutdown

Version of all relevant components (if applicable):
OCP 4.16.10
ODF Client : 4.17.0-98.stable
ODF Foundation : 4.170-98.stable

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)? yes - unable to recover in disaster scenario

Is there any workaround available to the best of your knowledge? no

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)? 3 (due to custom product install steps)

Can this issue reproducible? likely

Can this issue reproduce from the UI? no

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
1. Establish cluster in Stretch configuration
2. Install Product w/ HA configuration with datastores using RWO to access ocs-storagecluster-ceph-rbd storage
3. Simulate Zone outage for Zone-2 by shutting down all nodes
4. Add taint for non-graceful shutdown to Zone-2 nodes
`kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute`
5. Monitor kube resources on surviving zone - all should be able to access storage

Actual results:
Pods on surviving zone, unable to mount select PVCs :

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
...
Warning FailedMount 21s (x9 over 2m29s) kubelet MountVolume.MountDevice failed for volume "pvc-5438fe65-a911-4226-9d1f-d3402af17cc9" : rpc error: code = Internal desc = error generating volume 0001-0011-openshift-storage-0000000000000001-71f3c614-f564-4814-8cbb-4a0247804fb7: rados: ret=-108, Cannot send after transport endpoint shutdown

Expected results:
Pods on surviving zone are able to mount PVCs

Additional info:

When Zone-2 was restored and the non-graceful taint removed, the mount error was resolved. All pods running successfully on surviving zone.