Pull image from ACR failed

4.19.0-0.nightly-2025-05-11-153555 

Always

1. Given the cluster resource-group is qe-uidaily-0508-pf2qf-rg
2. Create containerregistry on azure
$ az acr create --resource-group qe-uidaily-0508-pf2qf-rg --name zhsuncontainerregistry --sku Basic
3. Login into ACR
$ az acr login --name zhsuncontainerregistry
4. Grant AcrPull role to reourcegroup qe-uidaily-0508-pf2qf-rg
I tried manually created Identity  qe-uidaily-0508-pf2qf-identity (in 4.19 seems this is not created by default) and manually bind qe-uidaily-0508-pf2qf-identity to nodes 

5. $ docker pull openshift/hello-openshift
6. $ docker tag openshift/hello-openshift:latest zhsuncontainerregistry.azurecr.io/hello-acr:latest
7. Push the image to ACR
$ docker push zhsuncontainerregistry.azurecr.io/hello-acr:latest 
8. Create a new app using the image on ACR
$ oc new-project hello-acr  
$ oc new-app --name hello-acr --allow-missing-images   --image zhsuncontainerregistry.azurecr.io/hello-acr:latest
$ oc get po                                                                                             
NAME                         READY   STATUS             RESTARTS   AGE
hello-acr-79cd4b5bc4-dvqkt   0/1     ImagePullBackOff   0          3m31s Failed to pull image "zhsuncontainerregistry.azurecr.io/hello-acr:latest": [initializing source docker://zhsuncontainerregistry.azurecr.io/hello-acr:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 65fd5960-ee74-468a-86aa-277bec29cbdc, initializing source docker://zhsuncontainerregistry.azurecr.io/hello-acr:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: bc82f7f3-f184-4617-8a16-9de6237b2995]   

sh-5.1# cat /tmp/request.json | /usr/libexec/kubelet-image-credential-provider-plugins/acr-credential-provider
Error: requires at least 1 arg(s), only received 0
Usage:
  acr-credential-provider configFile [flags]

Flags:
  -h, --help                     help for acr-credential-provider
  -r, --registry-mirror string   Mirror a source registry host to a target registry host, and image pull credential will be requested to the target registry host when the image is from source registry host

sh-5.1# cat /tmp/request.json
{
  "apiVersion": "credentialprovider.kubelet.k8s.io/v1",
  "kind": "CredentialProviderRequest",
  "image": "zhsunregistry1.azurecr.io/hello-acr:latest"
}

Pull image from ACR failed 

Pull image from ACR succeed

This only happens in 4.19, Theo may have some finding for this, discussion https://redhat-internal.slack.com/archives/GE2HQ9QP4/p1746700433890159