prometheus pods are crashing after some time and cleanup the WAL fixes it temporarily. We tried to replace the storage and tried initialize the prometheus-k8s pods but the issue re-occurs. I notice below errors as well:
2023-06-28T16:56:20.873596352Z ts=2023-06-28T16:56:20.872Z caller=stdlib.go:105 level=error component=web caller="http: superfluous response.WriteHeader call from github.com/opentracing-contrib/go-stdlib/nethttp.(*statusCodeTracker).WriteHeader (status-code-tracker.go:17" msg=)
2023-06-28T16:56:41.817611918Z ts=2023-06-28T16:56:41.817Z caller=stdlib.go:105 level=error component=web caller="http: superfluous response.WriteHeader call from github.com/opentracing-contrib/go-stdlib/nethttp.(*statusCodeTracker).WriteHeader (status-code-tracker.go:17" msg=)
2023-06-28T16:56:42.931037383Z ts=2023-06-28T16:56:42.930Z caller=stdlib.go:105 level=error component=web caller="http: superfluous response.WriteHeader call from github.com/opentracing-contrib/go-stdlib/nethttp.(*statusCodeTracker).WriteHeader (status-code-tracker.go:17" msg=)

4.10

1.
2.
3.

Issue was discussed in https://redhat-internal.slack.com/archives/C0VMT03S5/p1689194595993769 and was discussed if could be related to  https://github.com/prometheus/prometheus/issues/6139