This bug was initially created as a copy of Bug #2249844

I am copying this bug because:

Description of problem (please be detailed as possible and provide log
snippests):
After upgrade execution to 4.13.5-6 from 4.12 - (both OCP and ODF upgrade)
we see ceph health warn issue:

sh-5.1$ ceph status
cluster:
id: 68dc565f-f700-4312-93be-265b7ed15941
health: HEALTH_WARN
1 daemons have recently crashed

services:
mon: 3 daemons, quorum a,b,c (age 78m)
mgr: a(active, since 77m)
mds: 1/1 daemons up, 1 hot standby
osd: 3 osds: 3 up (since 77m), 3 in (since 2h)
rgw: 1 daemon active (1 hosts, 1 zones)

data:
volumes: 1/1 healthy
pools: 12 pools, 185 pgs
objects: 1.05k objects, 2.0 GiB
usage: 5.9 GiB used, 1.5 TiB / 1.5 TiB avail
pgs: 185 active+clean

io:
client: 1.4 KiB/s rd, 134 KiB/s wr, 2 op/s rd, 2 op/s wr

sh-5.1$ ceph crash ls
ID ENTITY NEW
2023-11-15T08:10:44.427601Z_b4fd4568-7eb7-4508-ab38-58e561dc809a mgr.a *
sh-5.1$ ceph crash info 2023-11-15T08:10:44.427601Z_b4fd4568-7eb7-4508-ab38-58e561dc809a
{
"backtrace": [
"/lib64/libc.so.6(+0x54df0) [0x7f7c91f2bdf0]",
"/lib64/libc.so.6(+0xa154c) [0x7f7c91f7854c]",
"raise()",
"abort()",
"/lib64/libstdc++.so.6(+0xa1a01) [0x7f7c92279a01]",
"/lib64/libstdc++.so.6(+0xad37c) [0x7f7c9228537c]",
"/lib64/libstdc++.so.6(+0xad3e7) [0x7f7c922853e7]",
"/lib64/libstdc++.so.6(+0xad649) [0x7f7c92285649]",
"/usr/lib64/ceph/libceph-common.so.2(+0x170d39) [0x7f7c9256fd39]",
"(SnapRealmInfo::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x3b) [0x7f7c926a7f4b]",
"/lib64/libcephfs.so.2(+0xaaec7) [0x7f7c86c43ec7]",
"/lib64/libcephfs.so.2(+0xacc59) [0x7f7c86c45c59]",
"/lib64/libcephfs.so.2(+0xadf10) [0x7f7c86c46f10]",
"/lib64/libcephfs.so.2(+0x929e8) [0x7f7c86c2b9e8]",
"(DispatchQueue::entry()+0x53a) [0x7f7c9272defa]",
"/usr/lib64/ceph/libceph-common.so.2(+0x3bab31) [0x7f7c927b9b31]",
"/lib64/libc.so.6(+0x9f802) [0x7f7c91f76802]",
"/lib64/libc.so.6(+0x3f450) [0x7f7c91f16450]"
],
"ceph_version": "17.2.6-148.el9cp",
"crash_id": "2023-11-15T08:10:44.427601Z_b4fd4568-7eb7-4508-ab38-58e561dc809a",
"entity_name": "mgr.a",
"os_id": "rhel",
"os_name": "Red Hat Enterprise Linux",
"os_version": "9.2 (Plow)",
"os_version_id": "9.2",
"process_name": "ceph-mgr",
"stack_sig": "4cb0911c06087a31d9752535de90ba18fd7aab25c037945b2c61f584dcf6a6db",
"timestamp": "2023-11-15T08:10:44.427601Z",
"utsname_hostname": "rook-ceph-mgr-a-5d475468dd-wzhmt",
"utsname_machine": "x86_64",
"utsname_release": "5.14.0-284.40.1.el9_2.x86_64",
"utsname_sysname": "Linux",
"utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed Nov 1 10:30:09 EDT 2023"
}

Discussed here:
https://chat.google.com/room/AAAAREGEba8/fZvCCW1MQfU

Venky pointed out that it smells like this issue:
https://tracker.ceph.com/issues/63188
BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=2247174

Venky cloned the 7.0 BZ to 6.1z4 target - https://bugzilla.redhat.com/show_bug.cgi?id=2249814

Version of all relevant components (if applicable):
ODF 4.13.5-6

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Is there any workaround available to the best of your knowledge?

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

Can this issue reproducible?
Trying to reproduce here:
https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-trigger-vsphere-upi-encryption-1az-rhcos-vsan-lso-vmdk-3m-3w-upgrade-ocp-ocs-auto/32/

Can this issue reproduce from the UI?

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
1. Install ODF 4.12 and OCP 4.12
2. Upgrade OCP to 4.13
3. Upgrade ODF to 4.13.5-6 build
4. After some time we see the health warn

Actual results:
Do not have health warn

Expected results:

Additional info:
Must gather:
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-031vue1cslv33-uba/j-031vue1cslv33-uba_20231115T053551/logs/testcases_1700036781/j-031vue1cslv33-u/
Job:
https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-trigger-vsphere-upi-encryption-1az-rhcos-vsan-lso-vmdk-3m-3w-upgrade-ocp-ocs-auto/31/