Prometheus reporting no space left on device - but the alert for KubePersistentVolumeFillingUp is not firing

The following alerts were firing

RULE                         SEVERITY   STATE    AGE   ALERTS   ACTIVE SINCE 
ClusterVersionOperatorDown   critical   firing   28s   1        
KubeControllerManagerDown    critical   firing   23s   1        
KubeSchedulerDown            critical   firing   24s   1        
Watchdog                     none       firing   1s    1        
AppKubeAPIDown               critical   firing   17s   1        
AppKubeletDown               critical   firing   11s   1        
KubeAPIDown                  critical   firing   16s   1        
KubeletDown                  critical   firing   5s    1        
NoRunningOvnMaster           critical   firing   0s    1        

In the must-gather provided I can see the KubePersistentVolumeFillingUp alerts reporting `inactive` but only for seconds...the detail in those alerts do show that prometheus was out of space: 

RULE                            SEVERITY   STATE      AGE   ALERTS
KubePersistentVolumeFillingUp   critical   inactive   21s   0        
KubePersistentVolumeFillingUp   warning    inactive   21s   0     

health: err
  labels:
    severity: warning
  lastError: 'write to WAL: log samples: write /prometheus/wal/00005989: no space
      

metrics are not avaliable

an alert indiacting an issue with prometheus....after the kubeletdown alert has fired 

must-gather is attached to the linked case