Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-4840

backfill replication script inefficient for larger databases

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • quay
    • False
    • None
    • False

      Running the `python -m util.backfillreplication` script against a Quay instance with ~6 million images is estimated to take well over 30 days to complete.

      A version of the script modified to do nothing but fetch the manifest blobs, ran at a pace that would take those 30 days just to iterate through the 16 million results. Since there are only about half a million layers that are under-replicated, a maximally efficient script could be expected to queue the correct items within 24 hours.

      We have a sample query that inverts the logic to avoid the full iteration.

              Unassigned Unassigned
              dvanduze@redhat.com David Van Duzer
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: