Description of problem:

We have observed at least two occurrences of an issue where the "search-postgres" database is growing very large:

• https://redhat-internal.slack.com/archives/CUU609ZQC/p1713191360916569
• https://redhat-internal.slack.com/archives/CTDEY6EEA/p1715781632532969

Customer is reporting that in one case, the database for 13 managed clusters grew to more than 50GiB in size. However the database is only roughly 400MB:

>>> POSTGRESQL Database size:

  datname  | pg_size_pretty 
-----------+----------------
 search    | 403 MB
 postgres  | 7753 kB
 template1 | 7753 kB
 template0 | 7753 kB
(4 rows)

The root cause is unclear at the moment.

Version-Release number of selected component (if applicable):

Red Hat Advanced Cluster Management for Kubernetes 2.9

How reproducible:

Always on customer side

Steps to Reproduce:

1. Unknown how to reproduce the issue, we are seeing two occurrences at customers

Actual results:

For a small amount of clusters (<15), the database grows to around 50GB - 90GB.

Expected results:

For example, 20GiB might be sufficient for about 200 managed clusters.

Additional info: