Description of problem (please be detailed as possible and provide log
snippests):
ceph-mds process has generated a following crash on 4.17.0-114 cluster, and cephcluster showing HEALTH_WARN state.
Since the crash happened in the libc.so.6 and backtrace doesn’t provide detailed function names or locations beyond the generic memory addresses.

sh-5.1$ ceph crash info 2024-10-05T06:10:52.621730Z_c7f22b45-d236-43ea-86a4-aa19b31c380a
{
"backtrace": [
"/lib64/libc.so.6(+0x3e6f0) [0x7f66b5bea6f0]",
"[0x5579e7405330]"
],
"ceph_version": "18.2.1-229.el9cp",
"crash_id": "2024-10-05T06:10:52.621730Z_c7f22b45-d236-43ea-86a4-aa19b31c380a",
"entity_name": "mds.ocs-storagecluster-cephfilesystem-b",
"os_id": "rhel",
"os_name": "Red Hat Enterprise Linux",
"os_version": "9.4 (Plow)",
"os_version_id": "9.4",
"process_name": "ceph-mds",
"stack_sig": "12c4f060cf8b59a0ebac25da63a7f5b2a2cf5b99f12a288248409824102b5615",
"timestamp": "2024-10-05T06:10:52.621730Z",
"utsname_hostname": "rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-6d77bb68ccw6s",
"utsname_machine": "x86_64",
"utsname_release": "5.14.0-427.37.1.el9_4.x86_64",
"utsname_sysname": "Linux",
"utsname_version": "#1 SMP PREEMPT_DYNAMIC Fri Sep 13 12:41:50 EDT 2024"
}

rook-ceph-mds logs
==========
❯ ocs logs rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-6d77bb68ccw6s
Defaulted container "mds" out of: mds, log-collector, chown-container-data-dir (init)
debug 2024-10-05T06:10:52.959+0000 7f528f474ac0 0 set uid:gid to 167:167 (ceph:ceph)
debug 2024-10-05T06:10:52.959+0000 7f528f474ac0 0 ceph version 18.2.1-229.el9cp (ef652b206f2487adfc86613646a4cac946f6b4e0) reef (stable), process ceph-mds, pid 151
debug 2024-10-05T06:10:52.959+0000 7f528f474ac0 1 main not setting numa affinity
debug 2024-10-05T06:10:52.959+0000 7f528f474ac0 0 pidfile_write: ignore empty --pid-file
starting mds.ocs-storagecluster-cephfilesystem-b at
debug 2024-10-05T06:10:52.970+0000 7f528ac08640 1 mds.ocs-storagecluster-cephfilesystem-b Updating MDS map to version 35 from mon.0
debug 2024-10-05T06:10:52.991+0000 7f528ac08640 1 mds.ocs-storagecluster-cephfilesystem-b Updating MDS map to version 36 from mon.0
debug 2024-10-05T06:10:52.991+0000 7f528ac08640 1 mds.ocs-storagecluster-cephfilesystem-b Monitors have assigned me to become a standby.
debug 2024-10-05T06:11:42.402+0000 7f528ac08640 1 mds.ocs-storagecluster-cephfilesystem-b Updating MDS map to version 41 from mon.0
debug 2024-10-05T06:11:42.403+0000 7f528ac08640 1 mds.0.0 handle_mds_map i am now mds.74241.0 replaying mds.0.0
debug 2024-10-05T06:11:42.403+0000 7f528ac08640 1 mds.0.0 handle_mds_map state change up:standby --> up:standby-replay
debug 2024-10-05T06:11:42.403+0000 7f528ac08640 1 mds.0.0 replay_start
debug 2024-10-05T06:11:42.403+0000 7f528ac08640 1 mds.0.0 waiting for osdmap 127 (which blocklists prior instance)
debug 2024-10-05T06:11:42.451+0000 7f5284bfc640 0 mds.0.cache creating system inode with ino:0x100
debug 2024-10-05T06:11:42.451+0000 7f5284bfc640 0 mds.0.cache creating system inode with ino:0x1
debug 2024-10-06T00:07:31.076+0000 7f528c40b640 -1 received signal: Hangup from (PID: 39092) UID: 0
debug 2024-10-06T00:07:31.080+0000 7f528c40b640 -1 received signal: Hangup from (PID: 39093) UID: 0
debug 2024-10-07T00:07:31.508+0000 7f528c40b640 -1 Fail to open '/proc/91114/cmdline' error = (2) No such file or directory
debug 2024-10-07T00:07:31.508+0000 7f528c40b640 -1 received signal: Hangup from <unknown> (PID: 91114) UID: 0
debug 2024-10-07T00:07:31.511+0000 7f528c40b640 -1 received signal: Hangup from (PID: 91115) UID: 0

Cephcluster is in HEALTH_WARN state
=========-=
❯ ocs get cephclusters.ceph.rook.io
NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL FSID
ocs-storagecluster-cephcluster /var/lib/rook 3 3d21h Ready Cluster created successfully HEALTH_WARN 6b3f9622-7cbd-44b0-9991-4c75c6f9cf39

Version of all relevant components (if applicable): 4.17

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)? Y

Is there any workaround available to the best of your knowledge? N

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

Can this issue reproducible? N

Can this issue reproduce from the UI? N

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
1. Deploy ODF 4.17.0-114 cluster
2. Create a 4 PVC with cephFS interface
3. Attach a PVC to pods and start FIO workload from each pod
4. wait till 3-4 minuters
5. POwerOff one Worker node from vcenter and wait 120 seconds minute.
6. POwerOn same worker node and wait till node join to the cluster

Actual results: After PowerOn worker node has joined but the cephcluster showing as HEALTH_WARN state and ceph-mds process has generated a crash

Expected results: When the node rejoins the cluster, all operations are expected to work

Additional info:

Must Gather logs : https://ibm.box.com/s/vxanlqhr461m82gafl3984a3awtsrlso