The Loki compactor in the openshift-network-observability namespace fails to reclaim disk space after compaction and retention operations.
Although compaction completes, deleted temporary files under /tmp/loki remain held open by the Loki process.
This prevents the OS from freeing space and results in persistent 98 % disk usage until the pod is restarted.This behavior matches upstream Grafana Loki issue:
https://github.com/grafana/loki/issues/19514     

Loki operator 6.2.6
Network-observability 1.9.2

1.Deploy LokiStack with openshift-network-observability operator and default 10 Gi compactor PVC (/tmp/loki).
2.Let ingestion and compaction run for several days.
3.Observe PVC utilization:
  df -h /tmp/loki <<--- showing near-full usage.
4.Check actual files:
  du -sch /tmp/loki/* <<--- showing only a few kilobytes. 
5.Run the command to see deleted files still open:
  lsof -p <compactor PID> | grep DEL
6.Delete/restart compactor pod → disk space is released temporarily.

Workaround: Manual restart of the compactor pod reclaims space temporarily, but the leak recurs after subsequent compaction. 

~~~
sh-5.1$ df -h /tmp/loki
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        9.8G  9.6G  236M  98% /tmp/loki
~~~
~~~
$ du -sch /tmp/loki/*
56K     compactor
16K     lost+found
72K     total
~~~
Despite only ~72 KB of visible files, /tmp/loki shows 9.6 GB used (98 % utilization).lsof confirms multiple deleted files still held open by the Loki process:
~~~
loki  24xxxx8  DEL-W  REG  /tmp/loki/compactor/index_20367/compactor-17xxxxxxxx
loki  24xxxx8  DEL-W  REG  /tmp/loki/compactor/index_20393/compactor-17xxxxxxxx
loki  24xxxx8  DEL-W  REG  /tmp/loki/compactor/index_20395/17xxxxxxxx
~~~
Compactor logs show repeated write failures due to exhausted space:
~~~
2025-11-02T06:17:29Z level=error caller=compactor.go:571 msg="failed to apply retention" err="write /tmp/loki/...: no space left on device"
~~~
Disk space is freed only after restarting the loki-compactor-0 pod, but the issue reappears in the next compaction cycle.

After compaction, Loki should close file descriptors for deleted temporary files so the filesystem can reclaim space automatically.
No manual intervention pod restarts should be required.