-
Bug
-
Resolution: Won't Do
-
Normal
-
None
-
4.13, 4.12, 4.11
-
No
-
OCPNODE Sprint 239 (Blue)
-
1
-
False
-
Description of problem:
In the Openshift cluster, we enabled imageContentSourcePolicy [1] to allow OCP to pull images from brew when the images are not published.
On July 3rd, July 4th, the mirror registry(brew-registry) is Outage(CWFHEALTH-2148). the OCP can not pull the ose-rbac-proxy:v4.12 image[2]. For the source registry is healthy, the OCP should pull images from the source regardless of the mirror registry status.
I did more tests on the worker. podman can pull images using the default auth file[5]. while crio and podman with --authfile /var/lib/kubelet/config.json raise error "name unknown: Digest not found"
When disable the ICSP, the pod can pull the ose-rbac-proxy image from registry.redhat.io
[1] imagecontentsourcepolicy are enabled.
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
name: brew-registry
spec:
repositoryDigestMirrors:
- mirrors:
- brew.registry.redhat.io
source: registry.redhat.io
- mirrors:
- brew.registry.redhat.io
source: registry.stage.redhat.io
- mirrors:
- brew.registry.redhat.io
source: registry-proxy.engineering.redhat.com
[2] pod using ose-rhax-proxy:v4.12 can not be started.
#oc get pods NAME READY STATUS RESTARTS AGE elasticsearch-operator-dc8df494c-wfvrs 1/2 ImagePullBackOff 0 14s loki-operator-controller-manager-6855ff8cd6-fgvnz 1/2 ImagePullBackOff 0 16s ............... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 38s default-scheduler Successfully assigned openshift-operators-redhat/elasticsearch-operator-dc8df494c-wfvrs to ip-10-0-72-29.us-east-2.compute.internal Normal AddedInterface 37s multus Add eth0 [10.130.2.107/23] from openshift-sdn Normal Pulled 37s kubelet Container image "registry.redhat.io/openshift-logging/elasticsearch-rhel8-operator@sha256:df1311f2afd2f9276d47ee6517fea36d212ef741f31d9ba7b831f8513f16dd3a" already present on machine Normal Created 36s kubelet Created container elasticsearch-operator Normal Started 36s kubelet Started container elasticsearch-operator Normal Pulling 16s (x2 over 37s) kubelet Pulling image "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec" Warning Failed 15s (x2 over 37s) kubelet Failed to pull image "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec": rpc error: code = Unknown desc = fetching target platform image selected from manifest list: reading manifest sha256:1941f88bcd73332cba3f794ee2dd3815a2255c9adfbbab4d4aa71f1710e2f614 in brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy: name unknown: Digest not found Warning Failed 15s (x2 over 37s) kubelet Error: ErrImagePull Normal BackOff 4s (x4 over 36s) kubelet Back-off pulling image "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec" Warning Failed 4s (x4 over 36s) kubelet Error: ImagePullBackOff
[3] pull image using crio/podman using --authfile /var/lib/kubelet/config.json.
$oc debug nodes/ip-10-0-52-143.us-east-2.compute.internal sh-4.4# chroot /host sh-4.4# crictl pull registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec sh-4.4# crictl pull registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec E0704 09:35:06.716054 519763 remote_image.go:242] "PullImage from image service failed" err="rpc error: code = Unknown desc = fetching target platform image selected from manifest list: reading manifest sha256:1941f88bcd73332cba3f794ee2dd3815a2255c9adfbbab4d4aa71f1710e2f614 in brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy: name unknown: Digest not found" image="registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec" FATA[0000] pulling image: rpc error: code = Unknown desc = fetching target platform image selected from manifest list: reading manifest sha256:1941f88bcd73332cba3f794ee2dd3815a2255c9adfbbab4d4aa71f1710e2f614 in brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy: name unknown: Digest not found sh-4.4# podman pull registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec --authfile /var/lib/kubelet/config.json Trying to pull registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec... Error: copying system image from manifest list: determining manifest MIME type for docker://registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec: reading manifest sha256:1941f88bcd73332cba3f794ee2dd3815a2255c9adfbbab4d4aa71f1710e2f614 in brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy: name unknown: Digest not found
[4] pull this image using default credential file
sh-4.4# podman login registry.redhat.io -u "xxxx" -p "xxxx" Login Succeeded! sh-4.4# podman pull registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec Trying to pull registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec... Getting image source signatures Checking if image destination supports signatures Copying blob 61ff475ee58a skipped: already exists Copying blob d8190195889e skipped: already exists Copying blob b13d06e0fd96 skipped: already exists Copying blob 527605c37b24 skipped: already exists Copying blob 97da74cc6d8f skipped: already exists Copying config 08e3594bed done Writing manifest to image destination Storing signatures 08e3594beda4e0b048011ee502162ed62aa131688783025bee0db3f0ffb6968d
How reproducible:
Always
Steps to Reproduce:
1. pull image registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec 2.
Actual results:
the image can not be pulled when the mirror registry is healthy.
Expected results:
Image can be pulled from source regardless of mirror registry status.
Additional info:
