Description of problem:

Issue started with empty dashboards in Grafana, along the way customer deployed a new S3 bucket, verified AWS creds, and updated to 2.9.4. The dashboards are still empty, but now we are seeing "database is locked" and "invalid character" with "no such host" errors.

Version-Release number of selected component (if applicable):

2.9.4

How reproducible:

Consistent in customer environment

Steps to Reproduce:

1. Case was open with issue being empty dashboards
2. Deployed new S3 bucket and verified AWS credentials
3. Updated to ACM 2.9.4
4. Seeing database locked and invalid character errors also "no such host"

Actual results:

Expected results:

Additional info:

Errors:

2024-10-26T17:58:53.890406206Z logger=infra.usagestats.collector t=2024-10-26T17:58:53.89+0000 lvl=warn msg="Failed to read Prometheus build info JSON" error="invalid character '
x1f' looking for beginning of value"
2024-10-26T17:58:53.890406206Z logger=infra.usagestats t=2024-10-26T17:58:53.89+0000 lvl=eror msg="Failed to fetch external metrics" error="invalid character '
x1f' looking for beginning of value"
2024-10-26T17:58:53.893550050Z logger=infra.usagestats t=2024-10-26T17:58:53.89+0000 lvl=eror msg="Failed to send usage stats" err="Post \"https://stats.grafana.org/grafana-usage-report\": dial tcp: lookup stats.grafana.org on 10.249.0.10:53: no such host"
2024-10-27T17:58:53.973595167Z logger=infra.usagestats.collector t=2024-10-27T17:58:53.97+0000 lvl=warn msg="Failed to read Prometheus build info JSON" error="invalid character '
x1f' looking for beginning of value"
2024-10-27T17:58:53.973595167Z logger=infra.usagestats t=2024-10-27T17:58:53.97+0000 lvl=eror msg="Failed to fetch external metrics" error="invalid character '
x1f' looking for beginning of value"
2024-10-27T17:58:54.021560752Z logger=infra.usagestats t=2024-10-27T17:58:54.02+0000 lvl=eror msg="Failed to send usage stats" err="Post \"https://stats.grafana.org/grafana-usage-report\": dial tcp: lookup stats.grafana.org on 10.249.0.10:53: no such host". These errors look similar to https://issues.redhat.com/browse/ACM-14296 so we are attempting to determine if another update to 2.9.5 will fix or if we are just chasing updates and this is unrelated.