Garbage Collection Bug Remediation

Due to a bug in the garbage collection feature between Quay 3.4.x and 3.7.x several storage objects were left in the object store, after they had already been purged from the Quay database.

We need a process to verify that these orphaned storage objects are no longer in use by Quay, and remove them from the object store. The sketch of the proposed process:

1. Make a copy of Quay imagestorage table into a temporary postgres database.
2. Generate list of all objects in site1, site2, and site3 storage buckets.
3. Parse output files with CLI tools (sed, cut, perl, etc) into a single CSV file of the form [sha256:blob,blob_size]
4. Import each CSV into site1,site2,site3 tables in temporary postgres database.
5. Execute the following three queries:
   • postgres=# \o site1_blobs.txt
   • postgres=# select a.content_checksum from site1 AS a LEFT OUTER JOIN imagestorage AS istore ON a.content_checksum = istore.content_checksum WHERE istore.content_checksum IS NULL;
   • postgres=# \o site2_blobs.txt
   • postgres=# select a.content_checksum from site2 AS a LEFT OUTER JOIN imagestorage AS istore ON a.content_checksum = istore.content_checksum WHERE istore.content_checksum IS NULL;
   • postgres=# \o site3_blobs.txt
   • postgres=# select a.content_checksum from site3 AS a LEFT OUTER JOIN imagestorage AS istore ON a.content_checksum = istore.content_checksum WHERE istore.content_checksum IS NULL;
6. Create a backup of the identified objects.
7. Delete the listed objects from the storage buckets.