Quay as a cache proxy / pull-through cache for other registries

Customer Problem: Developers require access to some upstream registries, e.g. the DockerHub library project or GitHub packages, to use open source software. Administrators and SecOps personell do not want to to allow generic access to these registries however. Some of these registries are also applying limits to unauthenticated pulls or pull using an account with a lower service tier which can become quickly exhausted in a shared environment. In other environments connection to the the upstream registry might be very slow and bandwidth constraint.

Goal: Quay acts a cache proxy / pull-through cache for arbitrary upstream registries

Problem:

• as of today clusters behind Quay can only pull images which are already in Quay (either pushed directly or added via repo mirroring)
• customers without governance requirements want to allow developers to reference / pull arbitrary images even if they're not synced into Quay yet, Quay should act as a proxy cache then
• customers want to circumvent pull rate limitation / throttling on upstream registries (e.g. DockerHub)
• customers want to accelerate pull performance of upstream registries (e.g. in edge deployment or remote locations)

Why is this important: 

• requested by several customers
• perceived as a threshold feature
• to achieve feature parity with the OCP internal registry
• DockerHub introduced severe rate limiting in Nov 2020
• developers don't want to rewrite all the pull specs in their manifest / helm chart / operators to benefit from a caching registry

Dependencies (internal and external):

• soft-dependency for OCP users: OCPNODE-521
• storage quota PROJQUAY-302

Prioritized epics + deliverables (in scope / not in scope):

• As a Quay user I want to be able to define an organization in Quay that acts a cache for a specific upstream registry
• As a Quay user I want to be able to supply credentials to the upstream registry when defining a cache organization so that I can circumvent / extend possible pull-rate limits or access private repositories
• As a Quay admin I want to be able to leverage the storage quota of an organization to limit the cache size so that backend storage consumption remains predictable by discarding images from the cache according to least recently used or pull frequency
• As a user I can pull an image from Quay from above organization even if it doesn't exist there yet. Once I tried to pull and the image isn't locally cached yet, Quay will fetch it from explicitly defined upstream / source registries, caches it locally and stream it to the client transparently
• As a user I want to rely on Quay's cache to stay coherent with the upstream source so that I transparently get newer images from the cache when tags have been overwritten in the upstream registry immediately or after a certain period of time (see next story)
• As a user I want to be able to configure a staleness period in order to control when the cache checks for upstream image changes so that I can lower the amount of upstream registry dependency (e.g. in case of pull limits being reached or temporary connectivity issues)
• As a user I want to be able to use the cache explicitly by referring to the upstream images by their original path components that are simply appended to URL of the quay cache location e.g. docker.io/library/postgres:latest -> quay.corp/cache/library/postgres:latest
• As a user I want to be able to use the cache transparently by adjusting my registries.conf file which uses the Quay cache organization for a mirror of a specific (namespace in a) upstream registry
• As a administrator I want to be able to select from caching an entire upstream registry (e.g. cache all of docker.io, i.e. docker.io/library/postgres:latest -> quay.corp/cache/library/postgres:latest) or just a selected namespaces (e.g. just cache docker.io/library, i.e. docker.io/library/postgres:latest -> quay.corp/docker-cache/postgres:latest) so that I can constrain access to potentially untrusted upstream registries
• Given the sensitive nature of accessing potentially untrusted content from upstream registry all cache pulls needs to be logged (audit log)
• As a user I want to be able to flush the cache so that I can eliminate excess storage consumption
• Cached images are subject to image scanning via Clair
• Robot accounts can be created in the cache organization as usual, their RBAC can be managed for all existing cached repositories at a given point in time
• Assuming that many existing customers with strong governance requirements explicitly do not want to offer this capability but use the explicit whitelisting via repo mirroring instead the feature need to be configurable (globally on or off) via config file / app.
• Metrics should exist to track cache activity and efficiency (hit rate, size, evictions)

Estimate (XS, S, M, L, XL, XXL):  L

Previous Work: 

• OpenShift internal registry pullthrough capability
• JFrog Artifactory implementation
• VMware Harbor Mirror Cache

Open questions:

• What happens when an upstream image is intermittently not accessible but still cached?