Bring back builds on virtualized platforms

Customer Problem:

As a Quay administrator I want to to provide build infrastructure to my clients. The current requirements for these are very high - the only supported option is to provide bare-metal clusters which are considered precious. Ideally I can use a set of dedicated servers to run builds or let them run on my Kubernetes / Openshift cluster.

Goal: Design around how to bring back container builds on virtual platforms and Investigate opportunities for deeper Quay build integration with DevExperience and Tekton teams

Problem:
Quay builds regressed when moving to Python v3. Currently Quay builds run buildah commands in virtual machines launched via Pods. This requires OCP bare-metal clusters which are rare with customers. There is a high bar to get Quay to run builders since 3.4. Customers have virtual infrastructure (both OCP and RHEL) available but with Quay demanding build jobs to be run in their own virtual machines this would require nested virtualization which is not supported by either RHEL or OCP.

Why is this important: 
For customers using Quay to build their containers there needs to be an option to run those builds on existing infrastructure which is most often virtualized. Also for Quay as a multi-cluster centralized registry, as well as customers who wish to use Quay for image builds need a tighter integration with an OpenShift-centric development model.

Dependencies (internal and external):
Developer Experience
Tekton

Deliverables (in scope / not in scope):

• As a user of Tekton, I want to be able to target image builds to Quay with a minimum of effort or configuration.
• As a user of Quay, I want to be able to run image builds on virtual RHEL nodes.

Estimate (XS, S, M, L, XL, XXL):  TBD
S

Previous Work: 
Quay Integration Operator

Open questions:

• TBD