Description of problem (please be detailed as possible and provide log snippets):

This case originally opened up due to the customer hitting slow ops as a result of SELinux relabeling. We fixed this and were in the process of monitoring before case closure. The customer reported everything was working great and when we were about to close the case the customer hit the "HEALTH_WARN" with "1 clients failing to respond to capability release" issue.

Upon further research, the customer should not be hitting this issue since they're on ceph version 17.2.6-246.el9cp, or 6.1z7 - 6.1.7 and from what I am tracking the fix was implemented in 6.1z4.

The good news is that the customer states that storage is working fine, and we have this tracked down to one specific workload causing this issue (This is a very high file count workload):

Name: pvc-d272cca3-9f94-423c-ad37-f4993caf2949
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: openshift-storage.cephfs.csi.ceph.com
volume.kubernetes.io/provisioner-deletion-secret-name: rook-csi-cephfs-provisioner
volume.kubernetes.io/provisioner-deletion-secret-namespace: openshift-storage
Finalizers: [kubernetes.io/pv-protection]
StorageClass: vivo-cephfs-selinux-relabel
Status: Bound
Claim: osb-fscustomer-esteira2/osb-fscustomer-esteira2-pvc <------ WORKLOAD
Reclaim Policy: Delete
Access Modes: RWX
VolumeMode: Filesystem
Capacity: 30Gi
Node Affinity: <none>
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: openshift-storage.cephfs.csi.ceph.com
FSType:
VolumeHandle: 0001-0011-openshift-storage-0000000000000001-aab3519a-5c94-4874-abb1-6cb90ececb6a
ReadOnly: false
VolumeAttributes: clusterID=openshift-storage
fsName=ocs-storagecluster-cephfilesystem
kernelMountOptions=context="system_u:object_r:container_file_t:s0"
storage.kubernetes.io/csiProvisionerIdentity=1726082784973-9392-openshift-storage.cephfs.csi.ceph.com
subvolumeName=csi-vol-aab3519a-5c94-4874-abb1-6cb90ececb6a
subvolumePath=/volumes/csi/csi-vol-aab3519a-5c94-4874-abb1-6cb90ececb6a/05ed12dd-b991-4bf0-9b9e-35611555e642
Events: <none>

Specifically two particular operations cause this issue:

Issue A and Issue B:

A. PVC clone operation or PVC content copy from pod towards an internal Node folder.

B. Trying to create 5 Pods (osb-fscustomer-server1, osb-fscustomer-server2, osb-fscustomer-server3, osb-fscustomer-server4, osb-fscustomer-server5) pointing out to osb-fscustomer-esteira2-pvc.

Since we're able to successfully reproduce this issue, the customer has been given a data collection process that should give Engineering all the logs/data needed.

Version of all relevant components (if applicable):

OCP:
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.14.33 True False 49d Cluster version is 4.14.33

ODF:
NAME DISPLAY VERSION REPLACES PHASE
argocd-operator.v0.12.0 Argo CD 0.12.0 argocd-operator.v0.11.0 Succeeded
cluster-logging.v5.9.6 Red Hat OpenShift Logging 5.9.6 cluster-logging.v5.9.5 Succeeded
loki-operator.v5.9.6 Loki Operator 5.9.6 loki-operator.v5.9.5 Succeeded
mcg-operator.v4.14.10-rhodf NooBaa Operator 4.14.10-rhodf mcg-operator.v4.14.9-rhodf Succeeded
ocs-operator.v4.14.10-rhodf OpenShift Container Storage 4.14.10-rhodf ocs-operator.v4.14.9-rhodf Succeeded
odf-csi-addons-operator.v4.14.10-rhodf CSI Addons 4.14.10-rhodf odf-csi-addons-operator.v4.14.9-rhodf Succeeded
odf-operator.v4.14.10-rhodf OpenShift Data Foundation 4.14.10-rhodf odf-operator.v4.14.9-rhodf Succeeded

Ceph:

{
"mon":

{ "ceph version 17.2.6-246.el9cp (0f65af2d95ce0936640f6ccd6a4825dce6237e4f) quincy (stable)": 3 }

,
"mgr":

{ "ceph version 17.2.6-246.el9cp (0f65af2d95ce0936640f6ccd6a4825dce6237e4f) quincy (stable)": 1 }

,
"osd":

{ "ceph version 17.2.6-246.el9cp (0f65af2d95ce0936640f6ccd6a4825dce6237e4f) quincy (stable)": 3 }

,
"mds":

{ "ceph version 17.2.6-246.el9cp (0f65af2d95ce0936640f6ccd6a4825dce6237e4f) quincy (stable)": 2 }

,
"rgw":

{ "ceph version 17.2.6-246.el9cp (0f65af2d95ce0936640f6ccd6a4825dce6237e4f) quincy (stable)": 1 }

,
"overall":

{ "ceph version 17.2.6-246.el9cp (0f65af2d95ce0936640f6ccd6a4825dce6237e4f) quincy (stable)": 10 }

}

Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)?

Yes, the customer wants to roll this in production and they have raised this issue stating this is crucial to fix due to current timelines. Below is the following statement from the customer:

"This issue is impacting a major project with Telefonica Brazil. If this is a bug, we urgently need a Bugzilla report created as soon as possible to facilitate a timely resolution."

Is there any workaround available to the best of your knowledge?

Yes, the following process will clear the issue, but the operations A and B do not succeed:

1. Run the following command to capture client ID and MDS name holding on to the caps:

$ oc exec -n openshift-storage deployment/rook-ceph-tools – ceph health detail

2. Run the following command to capture the session:

$ oc exec -n openshift-storage deployment/rook-ceph-tools – ceph tell mds.ocs-storagecluster-cephfilesystem:0 session ls > active-mds-session-ls.txt

3. Search for "<client.ID>" in that active-mds-session-ls.txt file. When you find that client session, scroll down until you see the csi-vol information. Copy the csi-vol-xxx-xxx all the way until the /. Then run the following command:

$ oc get pv | awk 'NR>1

{print $1}

' | while read it; do oc describe pv ${it}; echo " "; done > pv.out

4. Search for that csi-vol that you found in that client session in that pv.out file. That's your problematic workload. Scale it down.

5. Once all pods for that workload have terminated, delete the `rook-ceph-mds-ocs-storagecluster-cephfilesystem-a` pod:

~~~
$ oc delete pod n openshift-storage rook-ceph-mds-ocs-storagecluster-cephfilesystem-a<pod-name>
~~~

6. After a few minutes Ceph will get back to healthy.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

Can this issue reproducible?

Yes

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
1. PVC clone operation or PVC content copy from pod towards an internal Node folder.

or

2. Trying to create 5 Pods (osb-fscustomer-server1, osb-fscustomer-server2, osb-fscustomer-server3, osb-fscustomer-server4, osb-fscustomer-server5) pointing out to osb-fscustomer-esteira2-pvc.

Additional info:

I will put the data collection steps given to the customer in the private comment. Once collected, we'll set needinfo.