Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-19948

ImagePullBackOff when using short-name (containerRuntimeSearchRegistries) for external registry. Wrong user credentials are used: "unauthorized: Invalid Username or Password"

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Normal
    • None
    • 4.13.z
    • Node / Kubelet
    • No
    • False
    • Hide

      None

      Show
      None

    Description

      Summary:
      ImagePullBackOff when using short-name (containerRuntimeSearchRegistries) for external registry. Wrong user credentials are used: "unauthorized: Invalid Username or Password"
      docker.io credentials are used, instead of external registry credentials - which are both available in global pull secret

      How reproducible:
      Add quay.ocp4.rr.lab to allowedRegistries and containerRuntimeSearchRegistries

      # oc get image.config.openshift.io/cluster -o yaml
      apiVersion: config.openshift.io/v1
      kind: Image
      ...
      spec:
        registrySources:
      	allowedRegistries:
      	- docker.io
      	- quay.io
      	- quay.ocp4.rr.lab
      	- cloud.openshift.com
      	- registry.connect.redhat.com
      	- registry.redhat.io
      	- image-registry.openshift-image-registry.svc:5000
      	- registry.access.redhat.com
      	containerRuntimeSearchRegistries:
      	- quay.ocp4.rr.lab
      

      Add quay.ocp4.rr.lab credentials to global pull secret

      # oc -n openshift-config get secret/pull-secret --template='{{index .data ".dockerconfigjson" | base64decode}}' | jq | grep ":\ {"
        "auths": {
      	"cloud.openshift.com": {
      	"docker.io": {
      	"quay.io": {
      	"quay.ocp4.rr.lab": {
      	"registry.connect.redhat.com": {
      	"registry.redhat.io": {
      

      Push nginxinc/nginx-unprivileged to external registry

      # podman pull docker.io/nginxinc/nginx-unprivileged
      # podman tag docker.io/nginxinc/nginx-unprivileged quay.ocp4.rr.lab/nginxinc/nginx-unprivileged
      # podman push quay.ocp4.rr.lab/nginxinc/nginx-unprivileged
      

      Actual results:

      Simple oc run, using image short-name:

      # oc run nginx-test --rm -it --image nginxinc/nginx-unprivileged
      pod "nginx-test" deleted
      error: timed out waiting for the condition
      
      # oc get pods
      NAME         READY   STATUS             RESTARTS   AGE
      nginx-test   0/1     ImagePullBackOff   0          45s
      
      # oc get events
      LAST SEEN   TYPE      REASON           OBJECT                                    MESSAGE
      50s         Normal    Scheduled        pod/nginx-test                            Successfully assigned rainer/nginx-test to worker2.ocp4.rr.lab
      ...
      12s         Normal    Pulling          pod/nginx-test                            Pulling image "nginxinc/nginx-unprivileged"
      12s         Warning   Failed           pod/nginx-test                            Failed to pull image "nginxinc/nginx-unprivileged": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Invalid Username or Password
      12s         Warning   Failed           pod/nginx-test                            Error: ErrImagePull
      

      Expected results:
      Image pull is successful, pod gets started

      # oc run nginx-test --rm -it --image nginxinc/nginx-unprivileged
      

      Additional info:
      Quay logs show, that the docker.io credentials (global pull secret) are used

      quay.ocp4.rr.lab credentials)
      # podman logs quay 2>&1 | grep "2023-09-29" | grep "nginx-unprivileged"
      ...
      gunicorn-registry stdout | 2023-09-29 13:23:02,634 [196] [INFO] [gunicorn.access] 192.168.60.65 - rbeyel@test.org [29/Sep/2023:13:23:02 +0000] "GET /v2/auth?account=rbeyel%40test.org&scope=repository%3Anginxinc%2Fnginx-unprivileged%3Apull&service=quay.ocp4.rr.lab HTTP/1.1" 401 90 "-" "cri-o/1.26.4-3.rhaos4.13.git615a02c.el9 go/go1.19.10 os/linux arch/amd64"
      ...
      

      crictl pull on the worker is successful

      # oc debug node/worker2.ocp4.rr.lab
      ...
      sh-5.1# crictl images | grep nginx
      sh-5.1# 
      sh-5.1# crictl pull nginxinc/nginx-unprivileged
      Image is up to date for quay.ocp4.rr.lab/nginxinc/nginx-unprivileged@sha256:43a9610de730e43d0613ada2d0b9b7565c04ae1f11190317a05446911abc9c7c
      sh-5.1# 
      sh-5.1# crictl images | grep nginx
      quay.ocp4.rr.lab/nginxinc/nginx-unprivileged                                    latest              8065cac9b75f3       191MB
      

      quay logs

      gunicorn-registry stdout | 2023-09-29 13:30:00,270 [201] [INFO] [gunicorn.access] 192.168.60.65 - rainer [29/Sep/2023:13:30:00 +0000] "GET /v2/auth?account=rainer&scope=repository%3Anginxinc%2Fnginx-unprivileged%3Apull&service=quay.ocp4.rr.lab HTTP/1.1" 200 1031 "-" "cri-o/1.26.4-3.rhaos4.13.git615a02c.el9 go/go1.19.10 os/linux arch/amd64"
      
      • When using the full image name (instead of of short-name), it's successful
      • When removing the docker.io credentials (global pull secret), the above described scenario is successful
      • Customer is observing the same scenario with artifactory proxy (instead of the described quay scenario)

      Attachments

        Activity

          People

            qiwan233 Qi Wang
            rhn-support-rbeyel Rainer Beyel
            xiujuan wang xiujuan wang
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: