Uploaded image for project: 'OpenShift Cloud Credential Operator'
  1. OpenShift Cloud Credential Operator
  2. CCO-260

invalid_grant error in the image-registry operator on GCP using WIF

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • None
    • False
    • None
    • False
    • Hide

      Deploy cluster following the steps:  https://github.com/openshift/cloud-credential-operator/blob/master/docs/gcp_workload_identity.md#steps-to-install-an-openshift-cluster-with-workload-identity

      Wait at least an hour and run 'oc get co'

      [cahartma@red-hat-thinkpad clo]$ oc get co
      NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      authentication                             4.11.2    True        False         False      5h58m   
      baremetal                                  4.11.2    True        False         False      6h19m   
      cloud-controller-manager                   4.11.2    True        False         False      6h21m   
      cloud-credential                           4.11.2    True        False         False      6h18m   
      cluster-autoscaler                         4.11.2    True        False         False      6h18m   
      config-operator                            4.11.2    True        False         False      6h20m   
      console                                    4.11.2    True        False         False      6h4m    
      csi-snapshot-controller                    4.11.2    True        False         False      6h19m   
      dns                                        4.11.2    True        False         False      6h18m   
      etcd                                       4.11.2    True        False         False      6h17m   
      image-registry                             4.11.2    False       True          True       4h9m    Available: The deployment does not have available replicas...
      ingress                                    4.11.2    True        False         False      6h10m   
      insights                                   4.11.2    True        False         False      6h13m   
      kube-apiserver                             4.11.2    True        False         False      6h16m   
      kube-controller-manager                    4.11.2    True        False         False      6h16m   
      kube-scheduler                             4.11.2    True        False         False      6h15m   
      kube-storage-version-migrator              4.11.2    True        False         False      6h20m   
      machine-api                                4.11.2    True        False         False      6h15m   
      machine-approver                           4.11.2    True        False         False      6h19m   
      machine-config                             4.11.2    True        False         False      6h18m   
      marketplace                                4.11.2    True        False         False      6h19m   
      monitoring                                 4.11.2    True        False         False      6h5m    
      network                                    4.11.2    True        False         False      6h21m   
      node-tuning                                4.11.2    True        False         False      6h19m   
      openshift-apiserver                        4.11.2    True        False         False      6h13m   
      openshift-controller-manager               4.11.2    True        False         False      6h15m   
      openshift-samples                          4.11.2    True        False         False      6h13m   
      operator-lifecycle-manager                 4.11.2    True        False         False      6h19m   
      operator-lifecycle-manager-catalog         4.11.2    True        False         False      6h19m   
      operator-lifecycle-manager-packageserver   4.11.2    True        False         False      6h9m    
      service-ca                                 4.11.2    True        False         False      6h20m   
      storage                                    4.11.2    True        False         False      6h19m   
      

      Error message from 'oc describe co image-registry'

      {"error":"invalid_grant","error_description":"Unable to verify the ID Token signature."}

       

      Show
      Deploy cluster following the steps:   https://github.com/openshift/cloud-credential-operator/blob/master/docs/gcp_workload_identity.md#steps-to-install-an-openshift-cluster-with-workload-identity Wait at least an hour and run 'oc get co' [cahartma@red-hat-thinkpad clo]$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.11.2 True False False 5h58m baremetal 4.11.2 True False False 6h19m cloud-controller-manager 4.11.2 True False False 6h21m cloud-credential 4.11.2 True False False 6h18m cluster-autoscaler 4.11.2 True False False 6h18m config-operator 4.11.2 True False False 6h20m console 4.11.2 True False False 6h4m csi-snapshot-controller 4.11.2 True False False 6h19m dns 4.11.2 True False False 6h18m etcd 4.11.2 True False False 6h17m image-registry 4.11.2 False True True 4h9m Available: The deployment does not have available replicas... ingress 4.11.2 True False False 6h10m insights 4.11.2 True False False 6h13m kube-apiserver 4.11.2 True False False 6h16m kube-controller-manager 4.11.2 True False False 6h16m kube-scheduler 4.11.2 True False False 6h15m kube-storage-version-migrator 4.11.2 True False False 6h20m machine-api 4.11.2 True False False 6h15m machine-approver 4.11.2 True False False 6h19m machine-config 4.11.2 True False False 6h18m marketplace 4.11.2 True False False 6h19m monitoring 4.11.2 True False False 6h5m network 4.11.2 True False False 6h21m node-tuning 4.11.2 True False False 6h19m openshift-apiserver 4.11.2 True False False 6h13m openshift-controller-manager 4.11.2 True False False 6h15m openshift-samples 4.11.2 True False False 6h13m operator-lifecycle-manager 4.11.2 True False False 6h19m operator-lifecycle-manager-catalog 4.11.2 True False False 6h19m operator-lifecycle-manager-packageserver 4.11.2 True False False 6h9m service-ca 4.11.2 True False False 6h20m storage 4.11.2 True False False 6h19m Error message from 'oc describe co image-registry' {"error":"invalid_grant","error_description":"Unable to verify the ID Token signature."}  

      OCP 4.11.2

      I've attached must-gather.tar.gz

      We are installing on GCP using the cco utility and "manual" credentialsMode.  After some time, the image-registry operator begins to crashloop, with the error: 

      controller.go:373] unable to sync: unable to sync storage configuration: Get "https://storage.googleapis.com/storage/v1/b/cahartma-0921cluster1-qcjgt-image-registry-us-east1-hqrjkdcwvy?alt=json&prettyPrint=false&projection=full": oauth2/google: unable to generate access token: Post "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/cahartma-092-openshift-i-wxvrd@openshift-observability.iam.gserviceaccount.com:generateAccessToken": oauth2/google: status code 400: {"error":"invalid_grant","error_description":"Unable to verify the ID Token signature."
      

              Unassigned Unassigned
              cahartma@redhat.com Casey Hartman
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: