Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-54748

Unexpected Authentication Required when Pulling Images from docker.io

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

         As per the customer, after rebooting the cluster, pods starts to fail pulling images. When pulling public image from docker.io, they are encountering error: 
      - failed to pull image docker.io/postgres:13 : unable to retrieve auth token : invalid username/passwordWhen 
      
      using the following commands to further investigate, we unable to determine the culprit of the issue.
      
      $ source /etc/mco/proxy.env
      $ env | grep -i http$ curl -v https://registry-1.docker.io/v2/
      $ AUTH_JSON=$(curl 'https://auth.docker.io/token?scope=repository:library/postgres:pull&service=registry.docker.io')
      $ ACCESS_TOKEN=$(echo $AUTH_JSON | jq -r '.access_token')
      $ echo $ACCESS_TOKEN
      $ curl -v -H "Authorization: Bearer $ACCESS_TOKEN" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" https://registry-1.docker.io/v2/library/postgres/manifests/sha256:3f9ef54b880d61f06bca81522dc9584bd959bada22e1d0de04eed04633450fd1
      $ curl -v -H "Authorization: Bearer $ACCESS_TOKEN"  https://registry-1.docker.io/v2/library/postgres/blobs/sha256:dcb21c24ea67de000ca47251c4de9e682b8a42777d2a7fb5f4e6f54e085f9ae3
      
      I'll attach the output.
      It seems that everything is okay, and no issue on proxy side and we didn't even see the `unable to retrieve auth token : invalid username/password` error.
      
      `unable to retrieve auth token : invalid username/password` error only occurs when there is an invalid credential.
      
      Based on the error we suspect that, there is something passing credentials through CRI-O, somehow, to the image pull service, which for this we should not be.
      
      Logs of the failure are below, from CRI-O and kubelet:
      Feb 19 17:47:53 dinooscn009.mi.vas.omnitel.it crio[2886]: time="2025-02-19 17:47:53.455140296Z" level=info msg="Checking image status: docker.io/postgres:13" id=dceede4e-7850-41e4-825d-068edc671be7 name=/runtime.v1.ImageService/ImageStatus
      Feb 19 17:47:53 dinooscn009.mi.vas.omnitel.it crio[2886]: time="2025-02-19 17:47:53.455258504Z" level=info msg="Image docker.io/postgres:13 not found" id=dceede4e-7850-41e4-825d-068edc671be7 name=/runtime.v1.ImageService/ImageStatus
      Feb 19 17:47:53 dinooscn009.mi.vas.omnitel.it crio[2886]: time="2025-02-19 17:47:53.455271556Z" level=info msg="Image docker.io/postgres:13 not found" id=dceede4e-7850-41e4-825d-068edc671be7 name=/runtime.v1.ImageService/ImageStatus
      Feb 19 17:47:53 dinooscn009.mi.vas.omnitel.it crio[2886]: time="2025-02-19 17:47:53.456113381Z" level=info msg="Pulling image: docker.io/postgres:13" id=9a74b630-5f02-46d7-a7c2-22ab0715356b name=/runtime.v1.ImageService/PullImage
      Feb 19 17:47:53 dinooscn009.mi.vas.omnitel.it crio[2886]: time="2025-02-19 17:47:53.470797239Z" level=info msg="Trying to access \"docker.io/library/postgres:13\""
      Feb 19 17:47:54 dinooscn009.mi.vas.omnitel.it kubenswrapper[2914]: E0219 17:47:54.055945    2914 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: incorrect username or password" image="docker.io/postgres:13"

      Additional info:

      That error only shows when bad credentials were supplied 
      
      $ curl -u baduser:badpassword 'https://auth.docker.io/token?scope=repository:library/postgres:pull&service=registry.docker.io'
      {"details":"incorrect username or password"}
      
      
      

              qiwan233 Qi Wang
              rhn-support-nsagun Nicolei Sagun
              None
              None
              Cameron Meadors Cameron Meadors
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: