In https://github.com/simonpasquier/sandbox/blob/master/cmd/flushwal/main.go we have some code that flushes the wal. Since the code is fairly straight forward we should get an idea how long it takes to flush the WAL back to disk.

The expectation is that this doesn't take too long (say 10s of minutes for a large WAL would be acceptable). If it does the whole approach might be invalid, as the goal is to unblock clusters that run into prometheus OOM during upgrades.

DoD

Measure runtimes of WAL flush for reasonably big WAL instances (say a WAL size that causes prometheus to consume > 100GB RAM).