Description

At 1.5k Collections and 3k CollectionVersions in a system, an approval (i.e. moving a CollectionVersion from `staging` to `published` repo) takes from 23-120 seconds, see slack https://redhat-internal.slack.com/archives/C0342L9JAEB/p1684244208417789

In CRC stage, with 17k CollectionVersions in `published` and 18k CollectionVersions in `staging`, a single approval task takes from 3-5min. (infrastructure causes were ruled out).

We are seeing a problem with one approval, but if many moves happen quickly, the problem compounds.

We may need an architectural change, to ensure move/copy tasks are quick, and possibly delay the readiness of search results and not rebuild the x-repo search index on every change to a repo. Maybe rebuild on a schedule (every minute or so) for every repo, if the repo timestamp has changed… or other approaches.

Steps to Reproduce

Approve a collection in the approval-dashboard when many CollectionVersions are present

how reproducible: always

Actual Behavior

Task duration is long, can be 3-5min

Expected Behavior

Task duration is a few seconds