Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-4840

backfill replication script inefficient for larger databases

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • quay
    • False
    • None
    • False
    • 0

      Running the `python -m util.backfillreplication` script against a Quay instance with ~6 million images is estimated to take well over 30 days to complete.

      A version of the script modified to do nothing but fetch the manifest blobs, ran at a pace that would take those 30 days just to iterate through the 16 million results. Since there are only about half a million layers that are under-replicated, a maximally efficient script could be expected to queue the correct items within 24 hours.

      We have a sample query that inverts the logic to avoid the full iteration.

            Unassigned Unassigned
            dvanduze@redhat.com David Van Duzer
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: