Pull image from gcp artifact registry failed

    4.17

    100%

1. Create repo for gcp artifact registry: zhsun-repo1
 
2. Login to registry
gcloud auth login
gcloud auth configure-docker us-central1-docker.pkg.dev 
    
3. Push image to registry
$ docker pull openshift/hello-openshift
$ docker tag openshift/hello-openshift:latest us-central1-docker.pkg.dev/openshift-qe/zhsun-repo1/hello-gcr:latest
$ docker push us-central1-docker.pkg.dev/openshift-qe/zhsun-repo1/hello-gcr:latest
4. Create pod
$ oc new-project hello-gcr
$ oc new-app --name hello-gcr --allow-missing-images \  
  --image us-central1-docker.pkg.dev/openshift-qe/zhsun-repo1/hello-gcr:latest
5. Check pod status

Pull image failed.
must-gather: https://drive.google.com/file/d/1o9cyJB53vQtHNmL5EV_hIx9I_LzMTB0K/view?usp=sharing
kubelet log: https://drive.google.com/file/d/1tL7HGc4fEOjH5_v6howBpx2NuhjGKsTp/view?usp=sharing
$ oc get po               
NAME                          READY   STATUS             RESTARTS   AGE
hello-gcr-658f7f9869-76ssg    0/1     ImagePullBackOff   0          3h24m

$ oc describe po hello-gcr-658f7f9869-76ssg 
  Warning  Failed          14s (x2 over 15s)  kubelet  Error: ImagePullBackOff
  Normal   Pulling         2s (x2 over 16s)   kubelet  Pulling image "us-central1-docker.pkg.dev/openshift-qe/zhsun-repo1/hello-gcr:latest"
  Warning  Failed          1s (x2 over 16s)   kubelet  Failed to pull image "us-central1-docker.pkg.dev/openshift-qe/zhsun-repo1/hello-gcr:latest": rpc error: code = Unknown desc = Requesting bearer token: invalid status code from registry 403 (Forbidden)

Can pull image from artifact registry succeed

gcr.io works as expected. 
us-central1-docker.pkg.dev/openshift-qe/zhsun-repo1/hello-gcr:latest doesn't work.
$ oc get po -n hello-gcr            
NAME                          READY   STATUS             RESTARTS   AGE
hello-gcr-658f7f9869-76ssg    0/1     ImagePullBackOff   0          156m
hello-gcr2-6d98c475ff-vjkt5   1/1     Running            0          163m
$ oc get po -n hello-gcr -o yaml | grep image                                                                                                                                       
    - image: us-central1-docker.pkg.dev/openshift-qe/zhsun-repo1/hello-gcr:latest
    - image: gcr.io/openshift-qe/hello-gcr:latest