We already know that many use cases (multiarch/hypershift/acm) may not have an internal registry available. We were asked to provide another option to allow users to use other registries. Let's consider a path forward on this that's baked into our design initially.

Current Assumptions (according to our UX layering walkthrough ):

1. We need to do an in-cluster image build FROM the "customer base image" to build the cluster-specific bits/machineconfig  into the "final pool image"
2.  We need a place to keep that "final pool image" once we build it such that rpm-ostree can pull it (right now it's the internal imageregistry)

Things we know: 

1. The internal imageregistry won't always be present 
2. The "build" system is separate from the imageregistry so we think we should be able to depend on it as long as openshift-controller-manager isn't also optional
3. Tagging into ImageStreams still "works" without a backing store ( cases where imageregistry  spec.managementState: Removed) but obviously it doesn't pull/mirror the image or anything
4. Builds (using build api) still work when pulling from these ImageStreams – they appear to map the imagestream tag to the upstream tag and pull the upstream tag 
5. Imagestreams don't currently support manifestlists at this time but that's not an immediate concern, we just think we'll need them later for multiarch
6.  There exist plenty of registries (docker, quay.io,etc) that can satisfy this need and our eventual desire for manifests lists ( brainstorming a software that does this is not the "long pole in the tent")

Questions around limited imagestream functionality/what we can depend on (maybe this should be a generic 'what can we depend on' spike):

1.  Is openshift-controller-manager/the features it provides ever going to be optional or can we depend on it?
2.  Without the internal imageregistry, imagestreams can't really be pulled from/pushed to. Are imagestreams useful if we can't push/pull directly from them, or should we track/poll which images we're using for what purpose ourselves in our own CRD?
3.  What are the negative externalities (if any) associated with using imagestreams without a backing store? (e.g. we can obviously update/use the ImageStream objects to track images, but is there some controller somewhere that will have issues/start spewing errors if we do this?) There do not appear to be any negatives, in fact, the image-references file in the MCO is actually an ImageStream object

Questions we need answers to:  

1. Where do we store the "final pool image" image that we create in-cluster that contains machineconfig/cluster-specific config so the nodes can pull it?
2. Do we push it outside the cluster to an external registry and have the Nodes pull it back in? (this seems...not super efficient)
3. What are the cluster bootstrapping ramifications of where we keep the "final pool image"? 
4. Do we expect a customer have to explicitly set up a registry (or quay) to install a layered openshift cluster or use layering? 
5. Are we going to maintain two code paths, one that can depend on the internal registry if it's present and one that doesn't? 
6. Are we worried about pushing any "sensitive" cluster-specific information (certificates, etc) out of the cluster into an external registry?
7. If it's external, how do we expect the user to specify the external registry and the relevant credentials? 
8.  There's kind of this range from "use as much of the builtin builds/imagestreams stuff as we can" to "depend on nothing and be fully self contained" – generally where do we want to set our "notch" there? 
       

Acceptance Criteria

The design, phases, and constraints are represented inside the enhancement.