Uploaded image for project: 'OCP Technical Release Team'
  1. OCP Technical Release Team
  2. TRT-1368

4.15 Nightly Payloads Failing on GCP Credentials Quota

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • None
    • None
    • False
    • None
    • False

      Hit seemingly every job in the last payload:

      https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-gcp-ovn-rt-upgrade/1728713834463498240

      Credentials request shows: 

      "conditions": [                                                                                                
      {                                                                                                            
        "lastProbeTime": "2023-11-26T11:20:40Z",                                                                   
        "lastTransitionTime": "2023-11-26T11:20:40Z",
        "message": "failed to grant creds: error syncing creds in mint-mode: error creating custom role: rpc error: code = ResourceExhausted desc = Maximum number of roles reached. Maximum is: 300\nerror details: retry in 24h0m1s",
        "reason": "CredentialsProvisionFailure",
        "status": "True",                             
        "type": "CredentialsProvisionFailure"         
      }                                         
    ],

      We've heard a new gcp account is live, but we're not sure if these are landing in it or not. Perhaps they are and a limit needs to be bumped?

      Additional info

      This issue shows up as a Cluster Version Operator component readiness regression due to failing the following tests:

      • [sig-cluster-lifecycle] Cluster completes upgrade
      • [sig-arch][Feature:ClusterUpgrade] Cluster should be upgradeable after finishing upgrade [Late][Suite:upgrade]
      • [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]

              rhn-engineering-dgoodwin Devan Goodwin
              rhn-engineering-dgoodwin Devan Goodwin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: