Uploaded image for project: 'OCP Technical Release Team'
  1. OCP Technical Release Team
  2. TRT-1368

4.15 Nightly Payloads Failing on GCP Credentials Quota

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • None
    • None
    • False
    • None
    • False

      Hit seemingly every job in the last payload:

      https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-gcp-ovn-rt-upgrade/1728713834463498240

      Credentials request shows:

      "conditions": [                                                                                                
            {                                                                                                            
              "lastProbeTime": "2023-11-26T11:20:40Z",                                                                   
              "lastTransitionTime": "2023-11-26T11:20:40Z",
              "message": "failed to grant creds: error syncing creds in mint-mode: error creating custom role: rpc error: code = ResourceExhausted desc = Maximum number of roles reached. Maximum is: 300\nerror details: retry in 24h0m1s",
              "reason": "CredentialsProvisionFailure",
              "status": "True",                             
              "type": "CredentialsProvisionFailure"         
            }                                         
          ],                    
      

      We've heard a new gcp account is live, but we're not sure if these are landing in it or not. Perhaps they are and a limit needs to be bumped?

      Additional info

      This issue shows up as a Cluster Version Operator component readiness regression due to failing the following tests:

      • [sig-cluster-lifecycle] Cluster completes upgrade
      • [sig-arch][Feature:ClusterUpgrade] Cluster should be upgradeable after finishing upgrade [Late][Suite:upgrade]
      • [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]

            rhn-engineering-dgoodwin Devan Goodwin
            rhn-engineering-dgoodwin Devan Goodwin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: