Internal registry cannot be used to store custom layered osImage

4.12.0-0.nightly-2022-09-07-032607

always

1. Create a new namespace
$ oc new-project osimages

2. Get base image for our custom osImage

$ oc adm release info --pullspecs | grep rhel
Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version of oc. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations.
  rhel-coreos-8                                  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03576350691b682ba1c7478c5c9029b3a4f677bef933164ccdcfabe2dfc3ac09


3. Create a new ioImage and store it in the internal registry. Use the base image that we got in step 2.

$ oc new-build -D $'FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03576350691b682ba1c7478c5c9029b3a4f677bef933164ccdcfabe2dfc3ac09\nRUN touch /etc/hello-from-build' --to test-new-osimage


4. Wait until build has finished
$ oc get build
NAME                 TYPE     FROM         STATUS    STARTED          DURATION
test-new-osimage-1   Docker   Dockerfile   Running   26 seconds ago   
$ oc get build
NAME                 TYPE     FROM         STATUS     STARTED         DURATION
test-new-osimage-1   Docker   Dockerfile   Complete   2 minutes ago   1m27s


5. Get osImage's url
$ oc get is test-new-osimage
NAME               IMAGE REPOSITORY                                                                                                     TAGS     UPDATED
test-new-osimage   default-route-openshift-image-registry.apps.sregidor1aws-t12.qe.devcluster.openshift.com/osimages/test-new-osimage   latest   2 minutes ago


5.Create a new MachineConfig with our new image
cat <<EOF | oc create -f -
kind: MachineConfig
apiVersion: machineconfiguration.openshift.io/v1
metadata:
  labels:
    machineconfiguration.openshift.io/role: "worker"
  name: "new-os-image"
spec:
  osImageURL: "default-route-openshift-image-registry.apps.sregidor1aws-t12.qe.devcluster.openshift.com/osimages/test-new-osimage"
EOF


You can see this error in the daemon logs:

I0907 15:05:50.484731    2797 run.go:19] Running: podman pull -q --authfile /var/lib/kubelet/config.json default-route-openshift-image-registry.apps.sregidor1aws-t12.qe.devcluster.openshift.com/osimages/test-new-osimage
Error: initializing source docker://default-route-openshift-image-registry.apps.sregidor1aws-t12.qe.devcluster.openshift.com/osimages/test-new-osimage:latest: pinging container registry default-route-openshift-image-registry.apps.sregidor1aws-t12.qe.devcluster.openshift.com: Get "https://default-route-openshift-image-registry.apps.sregidor1aws-t12.qe.devcluster.openshift.com/v2/": x509: certificate signed by unknown authority
W0907 15:05:50.598137    2797 run.go:45] podman failed: running podman pull -q --authfile /var/lib/kubelet/config.json default-route-openshift-image-registry.apps.sregidor1aws-t12.qe.devcluster.openshift.com/osimages/test-new-osimage failed: Error: initializing source docker://default-route-openshift-image-registry.apps.sregidor1aws-t12.qe.devcluster.openshift.com/osimages/test-new-osimage:latest: pinging container registry default-route-openshift-image-registry.apps.sregidor1aws-t12.qe.devcluster.openshift.com: Get "https://default-route-openshift-image-registry.apps.sregidor1aws-t12.qe.devcluster.openshift.com/v2/": x509: certificate signed by unknown authority
: exit status 125; retrying...


6. After many retries the worker pool will become degraded
$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-ce69fc69dc8db970804ce356c867b4b8   True      False      False      3              3                   3                     0                      6h48m
worker   rendered-worker-2b9ecfcbe0a7f41c7087f98c79343bf4   False     True       True       3              0                   0                     1                      6h48m

The machinepool becomes degraded if we use an osImage stored in the internal registry

No problem should happen if we store our custom osImage in the internal registry