$ ceph health detail HEALTH_WARN 2 failed cephadm daemon(s); 3 osds down; 1 host (3 osds) down; Reduced data availability: 105 pgs inactive, 34 pgs peering; 34 pgs not deep-scrubbed in time; 34 pgs not scrubbed in time; 18 slow ops, oldest one blocked for 59 sec, mon.serverc.lab.example.com has slow ops [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon rgw.realm.zone.serverc.bqwjcv on serverc.lab.example.com is in error state daemon rgw.realm.zone.serverd.kfmflx on serverd.lab.example.com is in error state [WRN] OSD_DOWN: 3 osds down osd.0 (root=default,host=serverc) is down osd.1 (root=default,host=serverc) is down osd.2 (root=default,host=serverc) is down [WRN] OSD_HOST_DOWN: 1 host (3 osds) down host serverc (root=default) (3 osds) is down [WRN] PG_AVAILABILITY: Reduced data availability: 105 pgs inactive, 34 pgs peering pg 2.f is stuck inactive for 7m, current state unknown, last acting [] pg 2.10 is stuck inactive for 7m, current state unknown, last acting [] pg 2.11 is stuck inactive for 7m, current state unknown, last acting [] pg 2.12 is stuck inactive for 7m, current state unknown, last acting [] pg 2.13 is stuck inactive for 7m, current state unknown, last acting [] pg 2.14 is stuck inactive for 7m, current state unknown, last acting [] pg 2.15 is stuck inactive for 7m, current state unknown, last acting [] pg 2.16 is stuck inactive for 7m, current state unknown, last acting [] pg 2.17 is stuck inactive for 7m, current state unknown, last acting [] pg 2.18 is stuck inactive for 7m, current state unknown, last acting [] pg 2.19 is stuck peering for 3y, current state peering, last acting [0,3,8] pg 2.1a is stuck inactive for 7m, current state unknown, last acting [] pg 2.1b is stuck inactive for 7m, current state unknown, last acting [] pg 2.1c is stuck inactive for 7m, current state unknown, last acting [] pg 2.1d is stuck inactive for 7m, current state unknown, last acting [] pg 2.1e is stuck peering for 3y, current state peering, last acting [2,6,5] pg 2.1f is stuck peering for 3y, current state peering, last acting [0,3,8] pg 3.e is stuck inactive for 7m, current state unknown, last acting [] pg 3.10 is stuck inactive for 7m, current state unknown, last acting [] pg 3.11 is stuck inactive for 7m, current state unknown, last acting [] pg 3.12 is stuck peering for 3y, current state peering, last acting [0,7,4] pg 3.13 is stuck inactive for 7m, current state unknown, last acting [] pg 3.14 is stuck inactive for 7m, current state unknown, last acting [] pg 3.15 is stuck inactive for 7m, current state unknown, last acting [] pg 3.16 is stuck inactive for 7m, current state unknown, last acting [] pg 3.17 is stuck peering for 3y, current state peering, last acting [0,5,8] pg 3.18 is stuck peering for 3y, current state peering, last acting [2,6,7] pg 3.19 is stuck peering for 3y, current state peering, last acting [1,5,6] pg 3.1a is stuck inactive for 7m, current state unknown, last acting [] pg 3.1b is stuck inactive for 7m, current state unknown, last acting [] pg 3.1c is stuck inactive for 7m, current state unknown, last acting [] pg 3.1d is stuck inactive for 7m, current state unknown, last acting [] pg 3.1e is stuck peering for 3y, current state peering, last acting [2,6,3] pg 3.1f is stuck peering for 3y, current state peering, last acting [0,3,4] pg 4.9 is stuck inactive for 7m, current state unknown, last acting [] pg 4.10 is stuck peering for 3y, current state peering, last acting [1,6,3] pg 4.11 is stuck peering for 3y, current state peering, last acting [2,7,8] pg 4.12 is stuck peering for 3y, current state peering, last acting [1,6,7] pg 4.13 is stuck inactive for 7m, current state unknown, last acting [] pg 4.14 is stuck inactive for 7m, current state unknown, last acting [] pg 4.15 is stuck inactive for 7m, current state unknown, last acting [] pg 4.16 is stuck peering for 3y, current state peering, last acting [0,7,6] pg 4.17 is stuck peering for 3y, current state peering, last acting [0,7,8] pg 4.18 is stuck inactive for 7m, current state unknown, last acting [] pg 4.19 is stuck inactive for 7m, current state unknown, last acting [] pg 4.1a is stuck inactive for 7m, current state unknown, last acting [] pg 4.1b is stuck inactive for 7m, current state unknown, last acting [] pg 4.1c is stuck inactive for 7m, current state unknown, last acting [] pg 4.1d is stuck inactive for 7m, current state unknown, last acting [] pg 4.1e is stuck peering for 3y, current state peering, last acting [0,7,6] pg 4.1f is stuck inactive for 7m, current state unknown, last acting [] [WRN] PG_NOT_DEEP_SCRUBBED: 34 pgs not deep-scrubbed in time pg 2.1f not deep-scrubbed since 2021-10-29T07:07:56.404931+0000 pg 3.1e not deep-scrubbed since 2021-10-29T07:08:00.507051+0000 pg 2.1e not deep-scrubbed since 2021-10-29T07:07:56.404931+0000 pg 3.1f not deep-scrubbed since 2021-10-29T07:08:00.507051+0000 pg 2.19 not deep-scrubbed since 2021-10-29T07:07:56.404931+0000 pg 3.18 not deep-scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.1e not deep-scrubbed since 2021-10-29T07:08:02.544747+0000 pg 3.19 not deep-scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.11 not deep-scrubbed since 2021-10-29T07:08:02.544747+0000 pg 4.10 not deep-scrubbed since 2021-10-29T07:08:02.544747+0000 pg 3.17 not deep-scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.12 not deep-scrubbed since 2021-10-29T07:08:02.544747+0000 pg 3.12 not deep-scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.17 not deep-scrubbed since 2021-10-29T07:08:02.544747+0000 pg 4.16 not deep-scrubbed since 2021-10-29T07:08:02.544747+0000 pg 5.3 not deep-scrubbed since 2021-10-29T07:08:04.610579+0000 pg 2.4 not deep-scrubbed since 2021-10-29T07:07:56.404931+0000 pg 5.5 not deep-scrubbed since 2021-10-29T07:08:04.610579+0000 pg 4.4 not deep-scrubbed since 2021-10-29T07:08:02.544747+0000 pg 5.6 not deep-scrubbed since 2021-10-29T07:08:04.610579+0000 pg 2.1 not deep-scrubbed since 2021-10-29T07:07:56.404931+0000 pg 4.7 not deep-scrubbed since 2021-10-29T07:08:02.544747+0000 pg 3.2 not deep-scrubbed since 2021-10-29T07:08:00.507051+0000 pg 3.1 not deep-scrubbed since 2021-10-29T07:08:00.507051+0000 pg 3.4 not deep-scrubbed since 2021-10-29T07:08:00.507051+0000 pg 2.6 not deep-scrubbed since 2021-10-29T07:07:56.404931+0000 pg 3.7 not deep-scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.0 not deep-scrubbed since 2021-10-29T07:08:02.544747+0000 pg 3.6 not deep-scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.f not deep-scrubbed since 2021-10-29T07:08:02.544747+0000 pg 2.9 not deep-scrubbed since 2021-10-29T07:07:56.404931+0000 pg 3.b not deep-scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.b not deep-scrubbed since 2021-10-29T07:08:02.544747+0000 pg 2.e not deep-scrubbed since 2021-10-29T07:07:56.404931+0000 [WRN] PG_NOT_SCRUBBED: 34 pgs not scrubbed in time pg 2.1f not scrubbed since 2021-10-29T07:07:56.404931+0000 pg 3.1e not scrubbed since 2021-10-29T07:08:00.507051+0000 pg 2.1e not scrubbed since 2021-10-29T07:07:56.404931+0000 pg 3.1f not scrubbed since 2021-10-29T07:08:00.507051+0000 pg 2.19 not scrubbed since 2021-10-29T07:07:56.404931+0000 pg 3.18 not scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.1e not scrubbed since 2021-10-29T07:08:02.544747+0000 pg 3.19 not scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.11 not scrubbed since 2021-10-29T07:08:02.544747+0000 pg 4.10 not scrubbed since 2021-10-29T07:08:02.544747+0000 pg 3.17 not scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.12 not scrubbed since 2021-10-29T07:08:02.544747+0000 pg 3.12 not scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.17 not scrubbed since 2021-10-29T07:08:02.544747+0000 pg 4.16 not scrubbed since 2021-10-29T07:08:02.544747+0000 pg 5.3 not scrubbed since 2021-10-29T07:08:04.610579+0000 pg 2.4 not scrubbed since 2021-10-29T07:07:56.404931+0000 pg 5.5 not scrubbed since 2021-10-29T07:08:04.610579+0000 pg 4.4 not scrubbed since 2021-10-29T07:08:02.544747+0000 pg 5.6 not scrubbed since 2021-10-29T07:08:04.610579+0000 pg 2.1 not scrubbed since 2021-10-29T07:07:56.404931+0000 pg 4.7 not scrubbed since 2021-10-29T07:08:02.544747+0000 pg 3.2 not scrubbed since 2021-10-29T07:08:00.507051+0000 pg 3.1 not scrubbed since 2021-10-29T07:08:00.507051+0000 pg 3.4 not scrubbed since 2021-10-29T07:08:00.507051+0000 pg 2.6 not scrubbed since 2021-10-29T07:07:56.404931+0000 pg 3.7 not scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.0 not scrubbed since 2021-10-29T07:08:02.544747+0000 pg 3.6 not scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.f not scrubbed since 2021-10-29T07:08:02.544747+0000 pg 2.9 not scrubbed since 2021-10-29T07:07:56.404931+0000 pg 3.b not scrubbed since 2021-10-29T07:08:00.507051+0000 pg 4.b not scrubbed since 2021-10-29T07:08:02.544747+0000 pg 2.e not scrubbed since 2021-10-29T07:07:56.404931+0000 [WRN] SLOW_OPS: 18 slow ops, oldest one blocked for 59 sec, mon.serverc.lab.example.com has slow ops .1 debug 2025-08-28T15:08:02.852+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.7 [v2:172.25.250.13:6800/4152519844,v1:172.25.250.13:6801/4152519844] from osd.1 is reporting failure:1 debug 2025-08-28T15:08:02.852+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.7 reported failed by osd.1 debug 2025-08-28T15:08:02.852+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.8 [v2:172.25.250.14:6816/3649522219,v1:172.25.250.14:6817/3649522219] from osd.1 is reporting failure:1 debug 2025-08-28T15:08:02.852+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.8 reported failed by osd.1 debug 2025-08-28T15:08:02.928+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.0 [v2:172.25.250.12:6816/2932330964,v1:172.25.250.12:6817/2932330964] from osd.5 is reporting failure:1 debug 2025-08-28T15:08:02.928+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.0 reported failed by osd.5 debug 2025-08-28T15:08:02.928+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.1 [v2:172.25.250.12:6808/3316974618,v1:172.25.250.12:6809/3316974618] from osd.5 is reporting failure:1 debug 2025-08-28T15:08:02.928+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.1 reported failed by osd.5 debug 2025-08-28T15:08:02.928+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.2 [v2:172.25.250.12:6800/4065841302,v1:172.25.250.12:6801/4065841302] from osd.5 is reporting failure:1 debug 2025-08-28T15:08:02.928+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.2 reported failed by osd.5 debug 2025-08-28T15:08:03.401+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.3 [v2:172.25.250.13:6816/2126803717,v1:172.25.250.13:6817/2126803717] from osd.2 is reporting failure:1 debug 2025-08-28T15:08:03.401+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.3 reported failed by osd.2 debug 2025-08-28T15:08:03.401+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.4 [v2:172.25.250.14:6800/4141236412,v1:172.25.250.14:6801/4141236412] from osd.2 is reporting failure:1 debug 2025-08-28T15:08:03.401+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.4 reported failed by osd.2 debug 2025-08-28T15:08:03.401+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.5 [v2:172.25.250.13:6808/3655064894,v1:172.25.250.13:6809/3655064894] from osd.2 is reporting failure:1 debug 2025-08-28T15:08:03.401+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.5 reported failed by osd.2 debug 2025-08-28T15:08:03.401+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.6 [v2:172.25.250.14:6808/4094980740,v1:172.25.250.14:6809/4094980740] from osd.2 is reporting failure:1 debug 2025-08-28T15:08:03.401+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.6 reported failed by osd.2 debug 2025-08-28T15:08:03.401+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.7 [v2:172.25.250.13:6800/4152519844,v1:172.25.250.13:6801/4152519844] from osd.2 is reporting failure:1 debug 2025-08-28T15:08:03.401+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.7 reported failed by osd.2 debug 2025-08-28T15:08:03.401+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.8 [v2:172.25.250.14:6816/3649522219,v1:172.25.250.14:6817/3649522219] from osd.2 is reporting failure:1 debug 2025-08-28T15:08:03.401+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.8 reported failed by osd.2 cluster 2025-08-28T15:08:02.738097+0000 mon.serverc.lab.example.com (mon.0) 169 : cluster [DBG] osd.0 reported failed by osd.7 cluster 2025-08-28T15:08:02.738259+0000 mon.serverc.lab.example.com (mon.0) 170 : cluster [DBG] osd.1 reported failed by osd.7 cluster 2025-08-28T15:08:02.738320+0000 mon.serverc.lab.example.com (mon.0) 171 : cluster [DBG] osd.2 reported failed by osd.7 cluster 2025-08-28T15:08:02.788027+0000 mgr.clienta.nncugs (mgr.54395) 191 : cluster [DBG] pgmap v197: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:02.806587+0000 mon.serverc.lab.example.com (mon.0) 172 : cluster [DBG] osd.3 reported failed by osd.0 cluster 2025-08-28T15:08:02.806683+0000 mon.serverc.lab.example.com (mon.0) 173 : cluster [DBG] osd.4 reported failed by osd.0 cluster 2025-08-28T15:08:02.806737+0000 mon.serverc.lab.example.com (mon.0) 174 : cluster [DBG] osd.5 reported failed by osd.0 cluster 2025-08-28T15:08:02.806772+0000 mon.serverc.lab.example.com (mon.0) 175 : cluster [DBG] osd.6 reported failed by osd.0 cluster 2025-08-28T15:08:02.806810+0000 mon.serverc.lab.example.com (mon.0) 176 : cluster [DBG] osd.7 reported failed by osd.0 cluster 2025-08-28T15:08:02.806849+0000 mon.serverc.lab.example.com (mon.0) 177 : cluster [DBG] osd.8 reported failed by osd.0 cluster 2025-08-28T15:08:02.852874+0000 mon.serverc.lab.example.com (mon.0) 178 : cluster [DBG] osd.3 reported failed by osd.1 cluster 2025-08-28T15:08:02.852983+0000 mon.serverc.lab.example.com (mon.0) 179 : cluster [DBG] osd.4 reported failed by osd.1 cluster 2025-08-28T15:08:02.853045+0000 mon.serverc.lab.example.com (mon.0) 180 : cluster [DBG] osd.5 reported failed by osd.1 cluster 2025-08-28T15:08:02.853120+0000 mon.serverc.lab.example.com (mon.0) 181 : cluster [DBG] osd.6 reported failed by osd.1 cluster 2025-08-28T15:08:02.853171+0000 mon.serverc.lab.example.com (mon.0) 182 : cluster [DBG] osd.7 reported failed by osd.1 cluster 2025-08-28T15:08:02.853217+0000 mon.serverc.lab.example.com (mon.0) 183 : cluster [DBG] osd.8 reported failed by osd.1 cluster 2025-08-28T15:08:02.928929+0000 mon.serverc.lab.example.com (mon.0) 184 : cluster [DBG] osd.0 reported failed by osd.5 cluster 2025-08-28T15:08:02.929023+0000 mon.serverc.lab.example.com (mon.0) 185 : cluster [DBG] osd.1 reported failed by osd.5 cluster 2025-08-28T15:08:02.929066+0000 mon.serverc.lab.example.com (mon.0) 186 : cluster [DBG] osd.2 reported failed by osd.5 audit 2025-08-28T15:08:03.307978+0000 mon.serverd (mon.2) 44 : audit [DBG] from='client.? 172.25.250.10:0/518545405' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch cluster 2025-08-28T15:08:03.402013+0000 mon.serverc.lab.example.com (mon.0) 187 : cluster [DBG] osd.3 reported failed by osd.2 cluster 2025-08-28T15:08:03.402098+0000 mon.serverc.lab.example.com (mon.0) 188 : cluster [DBG] osd.4 reported failed by osd.2 cluster 2025-08-28T15:08:03.402149+0000 mon.serverc.lab.example.com (mon.0) 189 : cluster [DBG] osd.5 reported failed by osd.2 cluster 2025-08-28T15:08:03.402201+0000 mon.serverc.lab.example.com (mon.0) 190 : cluster [DBG] osd.6 reported failed by osd.2 cluster 2025-08-28T15:08:03.402232+0000 mon.serverc.lab.example.com (mon.0) 191 : cluster [DBG] osd.7 reported failed by osd.2 cluster 2025-08-28T15:08:03.402263+0000 mon.serverc.lab.example.com (mon.0) 192 : cluster [DBG] osd.8 reported failed by osd.2 debug 2025-08-28T15:08:03.533+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.0 [v2:172.25.250.12:6816/2932330964,v1:172.25.250.12:6817/2932330964] from osd.3 is reporting failure:1 debug 2025-08-28T15:08:03.533+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.0 reported failed by osd.3 debug 2025-08-28T15:08:03.533+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.1 [v2:172.25.250.12:6808/3316974618,v1:172.25.250.12:6809/3316974618] from osd.3 is reporting failure:1 debug 2025-08-28T15:08:03.533+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.1 reported failed by osd.3 debug 2025-08-28T15:08:03.533+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.2 [v2:172.25.250.12:6800/4065841302,v1:172.25.250.12:6801/4065841302] from osd.3 is reporting failure:1 debug 2025-08-28T15:08:03.533+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.2 reported failed by osd.3 debug 2025-08-28T15:08:03.533+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.0 [v2:172.25.250.12:6816/2932330964,v1:172.25.250.12:6817/2932330964] from osd.8 is reporting failure:1 debug 2025-08-28T15:08:03.533+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.0 reported failed by osd.8 debug 2025-08-28T15:08:03.533+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.1 [v2:172.25.250.12:6808/3316974618,v1:172.25.250.12:6809/3316974618] from osd.8 is reporting failure:1 debug 2025-08-28T15:08:03.533+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.1 reported failed by osd.8 debug 2025-08-28T15:08:03.533+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.2 [v2:172.25.250.12:6800/4065841302,v1:172.25.250.12:6801/4065841302] from osd.8 is reporting failure:1 debug 2025-08-28T15:08:03.533+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.2 reported failed by osd.8 cluster 2025-08-28T15:08:03.533845+0000 mon.serverc.lab.example.com (mon.0) 193 : cluster [DBG] osd.0 reported failed by osd.3 cluster 2025-08-28T15:08:03.533913+0000 mon.serverc.lab.example.com (mon.0) 194 : cluster [DBG] osd.1 reported failed by osd.3 cluster 2025-08-28T15:08:03.533974+0000 mon.serverc.lab.example.com (mon.0) 195 : cluster [DBG] osd.2 reported failed by osd.3 cluster 2025-08-28T15:08:03.534025+0000 mon.serverc.lab.example.com (mon.0) 196 : cluster [DBG] osd.0 reported failed by osd.8 cluster 2025-08-28T15:08:03.534060+0000 mon.serverc.lab.example.com (mon.0) 197 : cluster [DBG] osd.1 reported failed by osd.8 cluster 2025-08-28T15:08:03.534113+0000 mon.serverc.lab.example.com (mon.0) 198 : cluster [DBG] osd.2 reported failed by osd.8 cluster 2025-08-28T15:08:04.788992+0000 mgr.clienta.nncugs (mgr.54395) 192 : cluster [DBG] pgmap v198: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:08:05.851+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.0 [v2:172.25.250.12:6816/2932330964,v1:172.25.250.12:6817/2932330964] from osd.4 is reporting failure:1 debug 2025-08-28T15:08:05.851+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.0 reported failed by osd.4 debug 2025-08-28T15:08:05.851+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.1 [v2:172.25.250.12:6808/3316974618,v1:172.25.250.12:6809/3316974618] from osd.4 is reporting failure:1 debug 2025-08-28T15:08:05.851+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.1 reported failed by osd.4 debug 2025-08-28T15:08:05.851+0000 7fc11c351700 1 mon.serverc.lab.example.com@0(leader).osd e251 prepare_failure osd.2 [v2:172.25.250.12:6800/4065841302,v1:172.25.250.12:6801/4065841302] from osd.4 is reporting failure:1 debug 2025-08-28T15:08:05.851+0000 7fc11c351700 0 log_channel(cluster) log [DBG] : osd.2 reported failed by osd.4 cluster 2025-08-28T15:08:05.851998+0000 mon.serverc.lab.example.com (mon.0) 199 : cluster [DBG] osd.0 reported failed by osd.4 cluster 2025-08-28T15:08:05.852101+0000 mon.serverc.lab.example.com (mon.0) 200 : cluster [DBG] osd.1 reported failed by osd.4 cluster 2025-08-28T15:08:05.852164+0000 mon.serverc.lab.example.com (mon.0) 201 : cluster [DBG] osd.2 reported failed by osd.4 debug 2025-08-28T15:08:07.440+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e251 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 cluster 2025-08-28T15:08:06.790298 +0000 mgr.clienta.nncugs (mgr.54395) 193 : cluster [DBG] pgmap v199: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:08.791266+0000 mgr.clienta.nncugs (mgr.54395) 194 : cluster [DBG] pgmap v200: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail audit 2025-08-28T15:08:09.838765+0000 mon.servere (mon.3) 10 : audit [DBG] from='client.? 172.25.250.10:0/4232062615' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch cluster 2025-08-28T15:08:10.792258+0000 mgr.clienta.nncugs (mgr.54395) 195 : cluster [DBG] pgmap v201: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:08:12.449+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e251 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 cluster 2025-08-28T15:08:12.793210+0000 mgr.clienta.nncugs (mgr.54395) 196 : cluster [DBG] pgmap v202: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:14.794038+0000 mgr.clienta.nncugs (mgr.54395) 197 : cluster [DBG] pgmap v203: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:08:17.454+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e251 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 cluster 2025-08-28T15:08:16.795010+0000 mgr.clienta.nncugs (mgr.54395) 198 : cluster [DBG] pgmap v204: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:18.795783+0000 mgr.clienta.nncugs (mgr.54395) 199 : cluster [DBG] pgmap v205: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:20.796571+0000 mgr.clienta.nncugs (mgr.54395) 200 : cluster [DBG] pgmap v206: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:08:22.459+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e251 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 cluster 2025-08-28T15:08:22.797461+0000 mgr.clienta.nncugs (mgr.54395) 201 : cluster [DBG] pgmap v207: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:24.798267+0000 mgr.clienta.nncugs (mgr.54395) 202 : cluster [DBG] pgmap v208: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:08:27.470+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e251 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 cluster 2025-08-28T15:08:26.799204+0000 mgr.clienta.nncugs (mgr.54395) 203 : cluster [DBG] pgmap v209: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:28.800044+0000 mgr.clienta.nncugs (mgr.54395) 204 : cluster [DBG] pgmap v210: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:30.800808+0000 mgr.clienta.nncugs (mgr.54395) 205 : cluster [DBG] pgmap v211: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:08:32.479+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e251 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:08:32.495+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.25.250.12:6816/2932330964,v1:172.25.250.12:6817/2932330964] for 21sec e251 v251) debug 2025-08-28T15:08:32.524+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #2022. Immutable memtables: 0. debug 2025-08-28T15:08:32.524+0000 7fc121b5c700 4 rocksdb: (Original Log Time 2025/08/28-15:08:32.525189) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 debug 2025-08-28T15:08:32.524+0000 7fc121b5c700 4 rocksdb: [flush_job.cc:321] [default] [JOB 5] Flushing memtable with next log file: 2022 debug 2025-08-28T15:08:32.524+0000 7fc121b5c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756393712525276, "job": 5, "event": "flush_started", "num_memtables": 1, "num_entries": 712, "num_deletes": 250, "total_data_size": 2546727, "memory_usage": 2560200, "flush_reason": "Manual Compaction"} debug 2025-08-28T15:08:32.524+0000 7fc121b5c700 4 rocksdb: [flush_job.cc:350] [default] [JOB 5] Level-0 flush table #2023: started debug 2025-08-28T15:08:32.546+0000 7fc121b5c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756393712547113, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 2023, "file_size": 2248599, "table_properties": {"data_size": 2243960, "index_size": 2828, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 965, "raw_key_size": 8010, "raw_average_key_size": 20, "raw_value_size": 2236607, "raw_average_value_size": 5824, "num_data_blocks": 86, "num_entries": 384, "num_deletions": 250, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1756393668, "oldest_key_time": 1756393668, "file_creation_time": 1756393712}} debug 2025-08-28T15:08:32.546+0000 7fc121b5c700 4 rocksdb: [flush_job.cc:401] [default] [JOB 5] Level-0 flush table #2023: 2248599 bytes OK debug 2025-08-28T15:08:32.565+0000 7fc121b5c700 4 rocksdb: (Original Log Time 2025/08/28-15:08:32.547184) [memtable_list.cc:447] [default] Level-0 commit table #2023 started debug 2025-08-28T15:08:32.565+0000 7fc121b5c700 4 rocksdb: (Original Log Time 2025/08/28-15:08:32.560656) [memtable_list.cc:503] [default] Level-0 commit table #2023: memtable #1 done debug 2025-08-28T15:08:32.565+0000 7fc121b5c700 4 rocksdb: (Original Log Time 2025/08/28-15:08:32.560703) EVENT_LOG_v1 {"time_micros": 1756393712560693, "job": 5, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} debug 2025-08-28T15:08:32.565+0000 7fc121b5c700 4 rocksdb: (Original Log Time 2025/08/28-15:08:32.560744) [db_impl/db_impl_compaction_flush.cc:205] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 debug 2025-08-28T15:08:32.565+0000 7fc121b5c700 4 rocksdb: [db_impl/db_impl_files.cc:353] [JOB 5] Try to delete WAL files size 2543053, prev total WAL file size 2543219, number of live WAL files 2. debug 2025-08-28T15:08:32.567+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2025-08-28T15:08:32.567+0000 7fc12235d700 4 rocksdb: (Original Log Time 2025/08/28-15:08:32.567669) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-6 from 'mgrstat .. 'mgrstat; will stop at (end) debug 2025-08-28T15:08:32.567+0000 7fc12235d700 4 rocksdb: [compaction/compaction_job.cc:1676] [default] [JOB 6] Compacting 1@0 + 1@6 files to L6, score -1.00 debug 2025-08-28T15:08:32.567+0000 7fc12235d700 4 rocksdb: [compaction/compaction_job.cc:1680] [default] Compaction start summary: Base version 5 Base level 0, inputs: [2023(2195KB)], [2021(37MB)] debug 2025-08-28T15:08:32.567+0000 7fc12235d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756393712567727, "job": 6, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [2023], "files_L6": [2021], "score": -1, "input_data_size": 42090931} debug 2025-08-28T15:08:32.754+0000 7fc12235d700 4 rocksdb: [compaction/compaction_job.cc:1349] [default] [JOB 6] Generated table #2024: 2689 keys, 40293093 bytes debug 2025-08-28T15:08:32.754+0000 7fc12235d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756393712755219, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 2024, "file_size": 40293093, "table_properties": {"data_size": 40232288, "index_size": 53108, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 6853, "raw_key_size": 55778, "raw_average_key_size": 20, "raw_value_size": 40155898, "raw_average_value_size": 14933, "num_data_blocks": 1566, "num_entries": 2689, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1633080631, "oldest_key_time": 0, "file_creation_time": 1756393712}} debug 2025-08-28T15:08:32.755+0000 7fc12235d700 4 rocksdb: [compaction/compaction_job.cc:1415] [default] [JOB 6] Compacted 1@0 + 1@6 files to L6 => 40293093 bytes debug 2025-08-28T15:08:32.763+0000 7fc12235d700 4 rocksdb: (Original Log Time 2025/08/28-15:08:32.764212) [compaction/compaction_job.cc:760] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 224.4 rd, 214.8 wr, level 6, files in(1, 1) out(1) MB in(2.1, 38.0) out(38.4), read-write-amplify(36.6) write-amplify(17.9) OK, records in: 3189, records dropped: 500 output_compression: NoCompression debug 2025-08-28T15:08:32.763+0000 7fc12235d700 4 rocksdb: (Original Log Time 2025/08/28-15:08:32.764248) EVENT_LOG_v1 {"time_micros": 1756393712764239, "job": 6, "event": "compaction_finished", "compaction_time_micros": 187573, "compaction_time_cpu_micros": 52385, "output_level": 6, "num_output_files": 1, "total_output_size": 40293093, "num_input_records": 3189, "num_output_records": 2689, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} debug 2025-08-28T15:08:32.764+0000 7fc12235d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756393712764634, "job": 6, "event": "table_file_deletion", "file_number": 2023} debug 2025-08-28T15:08:32.766+0000 7fc12235d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756393712767257, "job": 6, "event": "table_file_deletion", "file_number": 2021} debug 2025-08-28T15:08:32.766+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2025-08-28T15:08:32.766+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2025-08-28T15:08:32.766+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2025-08-28T15:08:32.766+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2025-08-28T15:08:32.766+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2025-08-28T15:08:32.766+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2025-08-28T15:08:32.766+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2025-08-28T15:08:32.766+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2025-08-28T15:08:32.766+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2025-08-28T15:08:32.766+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2025-08-28T15:08:32.766+0000 7fc117b48700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting cluster 2025-08-28T15:08:32.801697+0000 mgr.clienta.nncugs (mgr.54395) 206 : cluster [DBG] pgmap v212: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:34.802596+0000 mgr.clienta.nncugs (mgr.54395) 207 : cluster [DBG] pgmap v213: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:08:36.729+0000 7fc11c351700 0 mon.serverc.lab.example.com@0(leader) e4 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/mirror_snapshot_schedule"} v 0) v1 debug 2025-08-28T15:08:36.729+0000 7fc11c351700 0 log_channel(audit) log [INF] : from='mgr.54395 ' entity='mgr.clienta.nncugs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/mirror_snapshot_schedule"}]: dispatch debug 2025-08-28T15:08:36.757+0000 7fc11c351700 0 mon.serverc.lab.example.com@0(leader) e4 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/trash_purge_schedule"} v 0) v1 debug 2025-08-28T15:08:36.757+0000 7fc11c351700 0 log_channel(audit) log [INF] : from='mgr.54395 ' entity='mgr.clienta.nncugs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/trash_purge_schedule"}]: dispatch audit 2025-08-28T15:08:36.728899+0000 mon.clienta (mon.1) 123 : audit [INF] from='mgr.54395 172.25.250.10:0/876639683' entity='mgr.clienta.nncugs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/mirror_snapshot_schedule"}]: dispatch audit 2025-08-28T15:08:36.729909+0000 mon.serverc.lab.example.com (mon.0) 202 : audit [INF] from='mgr.54395 ' entity='mgr.clienta.nncugs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/mirror_snapshot_schedule"}]: dispatch audit 2025-08-28T15:08:36.755514+0000 mon.clienta (mon.1) 124 : audit [INF] from='mgr.54395 172.25.250.10:0/876639683' entity='mgr.clienta.nncugs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/trash_purge_schedule"}]: dispatch audit 2025-08-28T15:08:36.758085+0000 mon.serverc.lab.example.com (mon.0) 203 : audit [INF] from='mgr.54395 ' entity='mgr.clienta.nncugs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/trash_purge_schedule"}]: dispatch debug 2025-08-28T15:08:37.495+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e251 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:08:37.499+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 36 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.25.250.12:6816/2932330964,v1:172.25.250.12:6817/2932330964] for 21sec e251 v251) cluster 2025-08-28T15:08:36.803560+0000 mgr.clienta.nncugs (mgr.54395) 208 : cluster [DBG] pgmap v214: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:08:38.972+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 105 pgs inactive, 34 pgs peering (PG_AVAILABILITY) debug 2025-08-28T15:08:38.973+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check failed: 3 slow ops, oldest one blocked for 31 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) cluster 2025-08-28T15:08:38.804483+0000 mgr.clienta.nncugs (mgr.54395) 209 : cluster [DBG] pgmap v215: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:38.973473+0000 mon.serverc.lab.example.com (mon.0) 204 : cluster [WRN] Health check update: Reduced data availability: 105 pgs inactive, 34 pgs peering (PG_AVAILABILITY) cluster 2025-08-28T15:08:38.973614+0000 mon.serverc.lab.example.com (mon.0) 205 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 31 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) cluster 2025-08-28T15:08:40.805350+0000 mgr.clienta.nncugs (mgr.54395) 210 : cluster [DBG] pgmap v216: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:08:42.501+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e251 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:08:42.501+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e251 we have enough reporters to mark osd.0 down debug 2025-08-28T15:08:42.501+0000 7fc11eb56700 0 log_channel(cluster) log [INF] : osd.0 failed (root=default,host=serverc) (2 reporters from different host after 60.650152 >= grace 57.755519) debug 2025-08-28T15:08:42.501+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e251 we have enough reporters to mark osd.1 down debug 2025-08-28T15:08:42.501+0000 7fc11eb56700 0 log_channel(cluster) log [INF] : osd.1 failed (root=default,host=serverc) (2 reporters from different host after 60.650062 >= grace 57.755520) debug 2025-08-28T15:08:42.501+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e251 we have enough reporters to mark osd.2 down debug 2025-08-28T15:08:42.501+0000 7fc11eb56700 0 log_channel(cluster) log [INF] : osd.2 failed (root=default,host=serverc) (2 reporters from different host after 60.650021 >= grace 57.755520) debug 2025-08-28T15:08:42.501+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e251 do_prune osdmap full prune enabled debug 2025-08-28T15:08:42.502+0000 7fc11cb52700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7fc11cb52700' had timed out after 0.000000000s debug 2025-08-28T15:08:42.502+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check failed: 3 osds down (OSD_DOWN) debug 2025-08-28T15:08:42.502+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check failed: 1 host (3 osds) down (OSD_HOST_DOWN) debug 2025-08-28T15:08:42.514+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 36 slow ops, oldest is osd_failure(failed timeout osd.0 [v2:172.25.250.12:6816/2932330964,v1:172.25.250.12:6817/2932330964] for 21sec e251 v251) debug 2025-08-28T15:08:42.525+0000 7fc11ab4e700 1 mon.serverc.lab.example.com@0(leader).osd e252 e252: 9 total, 6 up, 9 in debug 2025-08-28T15:08:42.534+0000 7fc11ab4e700 0 log_channel(cluster) log [DBG] : osdmap e252: 9 total, 6 up, 9 in cluster 2025-08-28T15:08:42.501909+0000 mon.serverc.lab.example.com (mon.0) 206 : cluster [INF] osd.0 failed (root=default,host=serverc) (2 reporters from different host after 60.650152 >= grace 57.755519) cluster 2025-08-28T15:08:42.501953+0000 mon.serverc.lab.example.com (mon.0) 207 : cluster [INF] osd.1 failed (root=default,host=serverc) (2 reporters from different host after 60.650062 >= grace 57.755520) cluster 2025- 08-28T15:08:42.501993+0000 mon.serverc.lab.example.com (mon.0) 208 : cluster [INF] osd.2 failed (root=default,host=serverc) (2 reporters from different host after 60.650021 >= grace 57.755520) cluster 2025-08-28T15:08:42.503062+0000 mon.serverc.lab.example.com (mon.0) 209 : cluster [WRN] Health check failed: 3 osds down (OSD_DOWN) cluster 2025-08-28T15:08:42.503083+0000 mon.serverc.lab.example.com (mon.0) 210 : cluster [WRN] Health check failed: 1 host (3 osds) down (OSD_HOST_DOWN) cluster 2025-08-28T15:08:42 .534674+0000 mon.serverc.lab.example.com (mon.0) 211 : cluster [DBG] osdmap e252: 9 total, 6 up, 9 in debug 2025-08-28T15:08:43.563+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e252 do_prune osdmap full prune enabled debug 2025-08-28T15:08:43.588+0000 7fc11ab4e700 1 mon.serverc.lab.example.com@0(leader).osd e253 e253: 9 total, 6 up, 9 in debug 2025-08-28T15:08:43.597+0000 7fc11ab4e700 0 log_channel(cluster) log [DBG] : osdmap e253: 9 total, 6 up, 9 in cluster 2025-08-28T15:08:42.806237+0000 mgr.clienta.nncugs (mgr.54395) 211 : cluster [DBG] pgmap v218: 105 pgs: 34 stale+peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:43.598008+0000 mon.serverc.lab.example.com (mon.0) 212 : cluster [DBG] osdmap e253: 9 total, 6 up, 9 in cluster 2025-08-28T15:08:44.807054+0000 mgr.clienta.nncugs (mgr.54395) 212 : cluster [DBG] pgmap v220: 105 pgs: 34 stale+peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:08:47.576+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check update: 36 slow ops, oldest one blocked for 36 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) debug 2025-08-28T15:08:47.576+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e253 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:08:47.583+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 18 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.25.250.13:6816/2126803717,v1:172.25.250.13:6817/2126803717] for 23sec e251 v251) cluster 2025-08-28T15:08:46.808084+0000 mgr.clienta.nncugs (mgr.54395) 213 : cluster [DBG] pgmap v221: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:47.576729+0000 mon.serverc.lab.example.com (mon.0) 213 : cluster [WRN] Health check update: 36 slow ops, oldest one blocked for 36 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) cluster 2025-08-28T15:08:48.809222+0000 mgr.clienta.nncugs (mgr.54395) 214 : cluster [DBG] pgmap v222: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:50.810158+0000 mgr.clienta.nncugs (mgr.54395) 215 : cluster [DBG] pgmap v223: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:08:52.585+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check update: 36 slow ops, oldest one blocked for 41 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) debug 2025-08-28T15:08:52.585+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e253 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:08:52.594+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 18 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.25.250.13:6816/2126803717,v1:172.25.250.13:6817/2126803717] for 23sec e251 v251) cluster 2025-08-28T15:08:52.585674+0000 mon.serverc.lab.example.com (mon.0) 214 : cluster [WRN] Health check update: 36 slow ops, oldest one blocked for 41 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) cluster 2025-08-28T15:08:52.811632+0000 mgr.clienta.nncugs (mgr.54395) 216 : cluster [DBG] pgmap v224: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail audit 2025-08-28T15:08:54.103558+0000 mon.clienta (mon.1) 125 : audit [DBG] from='client.? 172.25.250.10:0/2525595697' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch cluster 2025-08-28T15:08:54.812511+0000 mgr.clienta.nncugs (mgr.54395) 217 : cluster [DBG] pgmap v225: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:08:57.595+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check update: 18 slow ops, oldest one blocked for 44 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) debug 2025-08-28T15:08:57.596+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e253 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:08:57.609+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 18 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.25.250.13:6816/2126803717,v1:172.25.250.13:6817/2126803717] for 23sec e251 v251) cluster 2025-08-28T15:08:56.813560+0000 mgr.clienta.nncugs (mgr.54395) 218 : cluster [DBG] pgmap v226: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:08:57.596372+0000 mon.serverc.lab.example.com (mon.0) 215 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 44 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) debug 2025-08-28T15:08:58.491+0000 7fc11c351700 0 mon.serverc.lab.example.com@0(leader) e4 handle_command mon_command([{prefix=config-key set, key=mgr/cephadm/osd_remove_queue}] v 0) v1 debug 2025-08-28T15:08:58.557+0000 7fc11ab4e700 0 log_channel(audit) log [INF] : from='mgr.54395 ' entity='mgr.clienta.nncugs' audit 2025-08-28T15:08:58.081604+0000 mon.clienta (mon.1) 126 : audit [DBG] from='mgr.54395 172.25.250.10:0/876639683' entity='mgr.clienta.nncugs' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch audit 2025-08-28T15:08:58.558371+0000 mon.serverc.lab.example.com (mon.0) 216 : audit [INF] from='mgr.54395 ' entity='mgr.clienta.nncugs' audit 2025-08-28T15:08:58.567635+0000 mon.clienta (mon.1) 127 : audit [DBG] from='mgr.54395 172.25.250.10:0/876639683' entity='mgr.clienta.nncugs' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch cluster 2025-08-28T15:08:58.814338+0000 mgr.clienta.nncugs (mgr.54395) 219 : cluster [DBG] pgmap v227: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:09:00.815073+0000 mgr.clienta.nncugs (mgr.54395) 220 : cluster [DBG] pgmap v228: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail audit 2025-08-28T15:09:01.333845+0000 mon.clienta (mon.1) 128 : audit [DBG] from='client.? 172.25.250.10:0/269031960' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch debug 2025-08-28T15:09:02.609+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check update: 18 slow ops, oldest one blocked for 49 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) debug 2025-08-28T15:09:02.609+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e253 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:09:02.620+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 18 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.25.250.13:6816/2126803717,v1:172.25.250.13:6817/2126803717] for 23sec e251 v251) cluster 2025-08-28T15:09:02.609921+0000 mon.serverc.lab.example.com (mon.0) 217 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 49 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) cluster 2025-08-28T15:09:02.815968+0000 mgr.clienta.nncugs (mgr.54395) 221 : cluster [DBG] pgmap v229: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:09:04.816963+0000 mgr.clienta.nncugs (mgr.54395) 222 : cluster [DBG] pgmap v230: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:09:07.623+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check update: 18 slow ops, oldest one blocked for 54 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) debug 2025-08-28T15:09:07.623+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e253 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:09:07.633+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 18 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.25.250.13:6816/2126803717,v1:172.25.250.13:6817/2126803717] for 23sec e251 v251) cluster 2025-08-28T15:09:06.817933+0000 mgr.clienta.nncugs (mgr.54395) 223 : cluster [DBG] pgmap v231: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:09:07.623721+0000 mon.serverc.lab.example.com (mon.0) 218 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 54 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) cluster 2025-08-28T15:09:08.818724+0000 mgr.clienta.nncugs (mgr.54395) 224 : cluster [DBG] pgmap v232: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:09:10.819478+0000 mgr.clienta.nncugs (mgr.54395) 225 : cluster [DBG] pgmap v233: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:09:12.634+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check update: 18 slow ops, oldest one blocked for 59 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) debug 2025-08-28T15:09:12.634+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e253 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:09:12.643+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 18 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.25.250.13:6816/2126803717,v1:172.25.250.13:6817/2126803717] for 23sec e251 v251) cluster 2025-08-28T15:09:12.634693+0000 mon.serverc.lab.example.com (mon.0) 219 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 59 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) cluster 2025-08-28T15:09:12.820424+0000 mgr.clienta.nncugs (mgr.54395) 226 : cluster [DBG] pgmap v234: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail audit 2025-08-28T15:09:13.660041+0000 mon.clienta (mon.1) 129 : audit [DBG] from='client.? 172.25.250.10:0/8200623' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch cluster 2025-08-28T15:09:14.826870+0000 mgr.clienta.nncugs (mgr.54395) 227 : cluster [DBG] pgmap v235: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:09:17.643+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check update: 18 slow ops, oldest one blocked for 64 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) debug 2025-08-28T15:09:17.643+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e253 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:09:17.659+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 18 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.25.250.13:6816/2126803717,v1:172.25.250.13:6817/2126803717] for 23sec e251 v251) cluster 2025-08-28T15:09:16.828069+0000 mgr.clienta.nncugs (mgr.54395) 228 : cluster [DBG] pgmap v236: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:09:17.644348+0000 mon.serverc.lab.example.com (mon.0) 220 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 64 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) cluster 2025-08-28T15:09:18.828924+0000 mgr.clienta.nncugs (mgr.54395) 229 : cluster [DBG] pgmap v237: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:09:20.829930+0000 mgr.clienta.nncugs (mgr.54395) 230 : cluster [DBG] pgmap v238: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:09:22.660+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check update: 18 slow ops, oldest one blocked for 69 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) debug 2025-08-28T15:09:22.660+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e253 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:09:22.671+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 18 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.25.250.13:6816/2126803717,v1:172.25.250.13:6817/2126803717] for 23sec e251 v251) cluster 2025-08-28T15:09:22.660967+0000 mon.serverc.lab.example.com (mon.0) 221 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 69 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) cluster 2025-08-28T15:09:22.830987+0000 mgr.clienta.nncugs (mgr.54395) 231 : cluster [DBG] pgmap v239: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:09:24.831942+0000 mgr.clienta.nncugs (mgr.54395) 232 : cluster [DBG] pgmap v240: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025- 08-28T15:09:26.833010+0000 mgr.clienta.nncugs (mgr.54395) 233 : cluster [DBG] pgmap v241: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:09:27.674+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check update: 18 slow ops, oldest one blocked for 74 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) debug 2025-08-28T15:09:27.674+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e253 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:09:27.678+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 18 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.25.250.13:6816/2126803717,v1:172.25.250.13:6817/2126803717] for 23sec e251 v251) cluster 2025-08-28T15:09:27.674697+0000 mon.serverc.lab.example.com (mon.0) 222 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 74 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) cluster 2025-08-28T15:09:28.833726+0000 mgr.clienta.nncugs (mgr.54395) 234 : cluster [DBG] pgmap v242: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:09:30.834529+0000 mgr.clienta.nncugs (mgr.54395) 235 : cluster [DBG] pgmap v243: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:09:32.682+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check update: 18 slow ops, oldest one blocked for 79 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) debug 2025-08-28T15:09:32.682+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e253 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:09:32.689+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 18 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.25.250.13:6816/2126803717,v1:172.25.250.13:6817/2126803717] for 23sec e251 v251) cluster 2025-08-28T15:09:32.682668+0000 mon.serverc.lab.example.com (mon.0) 223 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 79 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) cluster 2025-08-28T15:09:32.835446+0000 mgr.clienta.nncugs (mgr.54395) 236 : cluster [DBG] pgmap v244: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail cluster 2025-08-28T15:09:34.836262+0000 mgr.clienta.nncugs (mgr.54395) 237 : cluster [DBG] pgmap v245: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:09:36.733+0000 7fc11c351700 0 mon.serverc.lab.example.com@0(leader) e4 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/mirror_snapshot_schedule"} v 0) v1 debug 2025-08-28T15:09:36.733+0000 7fc11c351700 0 log_channel(audit) log [INF] : from='mgr.54395 ' entity='mgr.clienta.nncugs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/mirror_snapshot_schedule"}]: dispatch debug 2025-08-28T15:09:36.763+0000 7fc11c351700 0 mon.serverc.lab.example.com@0(leader) e4 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/trash_purge_schedule"} v 0) v1 debug 2025-08-28T15:09:36.763+0000 7fc11c351700 0 log_channel(audit) log [INF] : from='mgr.54395 ' entity='mgr.clienta.nncugs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/trash_purge_schedule"}]: dispatch audit 2025-08-28T15:09:36.733032+0000 mon.clienta (mon.1) 130 : audit [INF] from='mgr.54395 172.25.250.10:0/876639683' entity='mgr.clienta.nncugs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/mirror_snapshot_schedule"}]: dispatch audit 2025-08-28T15:09:36.734456+0000 mon.serverc.lab.example.com (mon.0) 224 : audit [INF] from='mgr.54395 ' entity='mgr.clienta.nncugs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/mirror_snapshot_schedule"}]: dispatch audit 2025-08-28T15:09:36.763593+0000 mon.clienta (mon.1) 131 : audit [INF] from='mgr.54395 172.25.250.10:0/876639683' entity='mgr.clienta.nncugs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/trash_purge_schedule"}]: dispatch audit 2025-08-28T15:09:36.764466+0000 mon.serverc.lab.example.com (mon.0) 225 : audit [INF] from='mgr.54395 ' entity='mgr.clienta.nncugs' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/clienta.nncugs/trash_purge_schedule"}]: dispatch cluster 2025-08-28T15:09:36.837211+0000 mgr.clienta.nncugs (mgr.54395) 238 : cluster [DBG] pgmap v246: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail debug 2025-08-28T15:09:37.689+0000 7fc11eb56700 0 log_channel(cluster) log [WRN] : Health check update: 18 slow ops, oldest one blocked for 84 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) debug 2025-08-28T15:09:37.689+0000 7fc11eb56700 1 mon.serverc.lab.example.com@0(leader).osd e253 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 369098752 full_alloc: 369098752 kv_alloc: 272629760 debug 2025-08-28T15:09:37.696+0000 7fc11eb56700 -1 mon.serverc.lab.example.com@0(leader) e4 get_health_metrics reporting 18 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.25.250.13:6816/2126803717,v1:172.25.250.13:6817/2126803717] for 23sec e251 v251) cluster 2025-08-28T15:09:37.689904+0000 mon.serverc.lab.example.com (mon.0) 226 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 84 sec, mon.serverc.lab.example.com has slow ops (SLOW_OPS) cluster 2025-08-28T15:09:38.837930+0000 mgr.clienta.nncugs (mgr.54395) 239 : cluster [DBG] pgmap v247: 105 pgs: 34 peering, 71 unknown; 1.6 KiB data, 73 MiB used, 30 GiB / 30 GiB avail