Currently, mirroring behaviour is that we mirror everything or nothing, i.e. we create a perfect replica of the upstream repository. This creates significant hurdles when complex expressions for tag discovery are used, which create a list of many tags to be mirrored. If any tag at any point fails to be mirrored, Quay will revert the repository to its previous state. If the repo was empty, this would mean deleting all tags that were correctly mirrored.

To demonstrate what I mean: I created a new repository in my Quay instance deployed on AWS. I mirroed the python repo from Docker Hub with the following regular expression:

latest,3.[789].?,3.[789].??,3.??.?,3.??.??,3.??.??-*,3.??.?-*,3-*,3.[789].?-*,3.[789].??-*,3.6-*,3.9.??-slim-*,3.10.?-slim*'

Out of 612 tags that were being mirrored, 600 were mirrored successfully but two failed, so all tags were then deleted from the repository in question. The same thing happened 2 times more, for a total of 3 retries. Each time the end result was a failure and all successfully mirroed tags were deleted by the mirror worker.

This is a very bad user experience, there's no justification in deleting 600+ successfully mirrored tags based on 2 that failed to be mirrored. Mirroring should be best effort, if all tags fail to be mirrored, we should log a PARTIAL_MIRROR error and list tags that failed to be mirrored so that clients can check for themselves what went wrong with this small subset of tags.

How to test

• Attempt to mirror a tag list containing both images that can be successfully mirrored and images that cannot. eg. python - latest, 3.8, 3.4.9-alpine(fails). The tags should not be reverted.
• When REPO_MIRROR_ROLLBACK
   is set to true the same tag list should be reverted
• Switching between REPO_MIRROR_ROLLBACK on and off should still produce the correct behavior