-
Bug
-
Resolution: Not a Bug
-
Normal
-
None
-
4.13.z
-
No
-
False
-
Summary:
ImagePullBackOff when using short-name (containerRuntimeSearchRegistries) for external registry. Wrong user credentials are used: "unauthorized: Invalid Username or Password"
docker.io credentials are used, instead of external registry credentials - which are both available in global pull secret
How reproducible:
Add quay.ocp4.rr.lab to allowedRegistries and containerRuntimeSearchRegistries
# oc get image.config.openshift.io/cluster -o yaml apiVersion: config.openshift.io/v1 kind: Image ... spec: registrySources: allowedRegistries: - docker.io - quay.io - quay.ocp4.rr.lab - cloud.openshift.com - registry.connect.redhat.com - registry.redhat.io - image-registry.openshift-image-registry.svc:5000 - registry.access.redhat.com containerRuntimeSearchRegistries: - quay.ocp4.rr.lab
Add quay.ocp4.rr.lab credentials to global pull secret
# oc -n openshift-config get secret/pull-secret --template='{{index .data ".dockerconfigjson" | base64decode}}' | jq | grep ":\ {" "auths": { "cloud.openshift.com": { "docker.io": { "quay.io": { "quay.ocp4.rr.lab": { "registry.connect.redhat.com": { "registry.redhat.io": {
Push nginxinc/nginx-unprivileged to external registry
# podman pull docker.io/nginxinc/nginx-unprivileged # podman tag docker.io/nginxinc/nginx-unprivileged quay.ocp4.rr.lab/nginxinc/nginx-unprivileged # podman push quay.ocp4.rr.lab/nginxinc/nginx-unprivileged
Actual results:
Simple oc run, using image short-name:
# oc run nginx-test --rm -it --image nginxinc/nginx-unprivileged pod "nginx-test" deleted error: timed out waiting for the condition
# oc get pods NAME READY STATUS RESTARTS AGE nginx-test 0/1 ImagePullBackOff 0 45s
# oc get events LAST SEEN TYPE REASON OBJECT MESSAGE 50s Normal Scheduled pod/nginx-test Successfully assigned rainer/nginx-test to worker2.ocp4.rr.lab ... 12s Normal Pulling pod/nginx-test Pulling image "nginxinc/nginx-unprivileged" 12s Warning Failed pod/nginx-test Failed to pull image "nginxinc/nginx-unprivileged": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Invalid Username or Password 12s Warning Failed pod/nginx-test Error: ErrImagePull
Expected results:
Image pull is successful, pod gets started
# oc run nginx-test --rm -it --image nginxinc/nginx-unprivileged
Additional info:
Quay logs show, that the docker.io credentials (global pull secret) are used
quay.ocp4.rr.lab credentials) # podman logs quay 2>&1 | grep "2023-09-29" | grep "nginx-unprivileged" ... gunicorn-registry stdout | 2023-09-29 13:23:02,634 [196] [INFO] [gunicorn.access] 192.168.60.65 - rbeyel@test.org [29/Sep/2023:13:23:02 +0000] "GET /v2/auth?account=rbeyel%40test.org&scope=repository%3Anginxinc%2Fnginx-unprivileged%3Apull&service=quay.ocp4.rr.lab HTTP/1.1" 401 90 "-" "cri-o/1.26.4-3.rhaos4.13.git615a02c.el9 go/go1.19.10 os/linux arch/amd64" ...
crictl pull on the worker is successful
# oc debug node/worker2.ocp4.rr.lab
...
sh-5.1# crictl images | grep nginx
sh-5.1#
sh-5.1# crictl pull nginxinc/nginx-unprivileged
Image is up to date for quay.ocp4.rr.lab/nginxinc/nginx-unprivileged@sha256:43a9610de730e43d0613ada2d0b9b7565c04ae1f11190317a05446911abc9c7c
sh-5.1#
sh-5.1# crictl images | grep nginx
quay.ocp4.rr.lab/nginxinc/nginx-unprivileged latest 8065cac9b75f3 191MB
quay logs
gunicorn-registry stdout | 2023-09-29 13:30:00,270 [201] [INFO] [gunicorn.access] 192.168.60.65 - rainer [29/Sep/2023:13:30:00 +0000] "GET /v2/auth?account=rainer&scope=repository%3Anginxinc%2Fnginx-unprivileged%3Apull&service=quay.ocp4.rr.lab HTTP/1.1" 200 1031 "-" "cri-o/1.26.4-3.rhaos4.13.git615a02c.el9 go/go1.19.10 os/linux arch/amd64"
- When using the full image name (instead of of short-name), it's successful
- When removing the docker.io credentials (global pull secret), the above described scenario is successful
- Customer is observing the same scenario with artifactory proxy (instead of the described quay scenario)
