-
Bug
-
Resolution: Done
-
Normal
-
None
-
4.16
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
As per the customer, after rebooting the cluster, pods starts to fail pulling images. When pulling public image from docker.io, they are encountering error: - failed to pull image docker.io/postgres:13 : unable to retrieve auth token : invalid username/passwordWhen using the following commands to further investigate, we unable to determine the culprit of the issue. $ source /etc/mco/proxy.env $ env | grep -i http$ curl -v https://registry-1.docker.io/v2/ $ AUTH_JSON=$(curl 'https://auth.docker.io/token?scope=repository:library/postgres:pull&service=registry.docker.io') $ ACCESS_TOKEN=$(echo $AUTH_JSON | jq -r '.access_token') $ echo $ACCESS_TOKEN $ curl -v -H "Authorization: Bearer $ACCESS_TOKEN" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" https://registry-1.docker.io/v2/library/postgres/manifests/sha256:3f9ef54b880d61f06bca81522dc9584bd959bada22e1d0de04eed04633450fd1 $ curl -v -H "Authorization: Bearer $ACCESS_TOKEN" https://registry-1.docker.io/v2/library/postgres/blobs/sha256:dcb21c24ea67de000ca47251c4de9e682b8a42777d2a7fb5f4e6f54e085f9ae3 I'll attach the output. It seems that everything is okay, and no issue on proxy side and we didn't even see the `unable to retrieve auth token : invalid username/password` error. `unable to retrieve auth token : invalid username/password` error only occurs when there is an invalid credential. Based on the error we suspect that, there is something passing credentials through CRI-O, somehow, to the image pull service, which for this we should not be. Logs of the failure are below, from CRI-O and kubelet: Feb 19 17:47:53 dinooscn009.mi.vas.omnitel.it crio[2886]: time="2025-02-19 17:47:53.455140296Z" level=info msg="Checking image status: docker.io/postgres:13" id=dceede4e-7850-41e4-825d-068edc671be7 name=/runtime.v1.ImageService/ImageStatus Feb 19 17:47:53 dinooscn009.mi.vas.omnitel.it crio[2886]: time="2025-02-19 17:47:53.455258504Z" level=info msg="Image docker.io/postgres:13 not found" id=dceede4e-7850-41e4-825d-068edc671be7 name=/runtime.v1.ImageService/ImageStatus Feb 19 17:47:53 dinooscn009.mi.vas.omnitel.it crio[2886]: time="2025-02-19 17:47:53.455271556Z" level=info msg="Image docker.io/postgres:13 not found" id=dceede4e-7850-41e4-825d-068edc671be7 name=/runtime.v1.ImageService/ImageStatus Feb 19 17:47:53 dinooscn009.mi.vas.omnitel.it crio[2886]: time="2025-02-19 17:47:53.456113381Z" level=info msg="Pulling image: docker.io/postgres:13" id=9a74b630-5f02-46d7-a7c2-22ab0715356b name=/runtime.v1.ImageService/PullImage Feb 19 17:47:53 dinooscn009.mi.vas.omnitel.it crio[2886]: time="2025-02-19 17:47:53.470797239Z" level=info msg="Trying to access \"docker.io/library/postgres:13\"" Feb 19 17:47:54 dinooscn009.mi.vas.omnitel.it kubenswrapper[2914]: E0219 17:47:54.055945 2914 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: incorrect username or password" image="docker.io/postgres:13"
Additional info:
That error only shows when bad credentials were supplied $ curl -u baduser:badpassword 'https://auth.docker.io/token?scope=repository:library/postgres:pull&service=registry.docker.io' {"details":"incorrect username or password"}