EPIC Goal

Our CI jobs are very often reaching Github Rate Limts that make the tests fail.

A single GitHub App provides a 5k/hour rate limit.

It seems we have significant room for improvement.

1. Leftover Test Repositories in janus-qe Org (approx impact on rate limit 25-30%)
There are more than 61 leftover test repositories in the janus-qe organization.
The org has 70+ repositories, and 61 of them are orphaned test repositories created by e2e tests, such as:
 
test-annotator-,test-relation-, test-scaffolded-, bulk-import-test-
There may be an issue preventing proper cleanup. * https://github.com/orgs/janus-qe/repositories?q=test-

• https://github.com/orgs/janus-qe/repositories?q=bulk-import-test

2. Catalog Entity Refresh Interval  (approx impact on rate limit 30-40%)
There is no custom processingInterval configured for catalog entity refresh. According to the documentation, the default is 30 minutes and our tests runs for longer than this.
We may want to increase this to 24 hours: * https://github.com/redhat-developer/rhdh/blob/d434aa5/.ibm/pipelines/resources/config_map/app-config-rhdh.yaml#L144-L169

3. Unnecessary GitHub Providers in showcase-sanity-plugins Deployment  (approx impact on rate limit 25-30%)
The showcase-sanity-plugins deployment runs GitHub providers unnecessarily.
It inherits values_showcase.yaml, which includes GitHub and GitHubOrg providers. However, it appears that this deployment does not require GitHub org/entity discovery, yet it still consumes the rate limit.
We can override or disable this in the overlay config: * https://github.com/redhat-developer/rhdh/blob/d434aa5/.ibm/pipelines/value_files/diff-values_showcase-sanity-plugins.yaml

4. There is duplicate GitHub Catalog Provider Configurations (Approx impact 3-5%)
The GitHub entity provider is configured in two places, and both configurations are loaded, effectively doubling the discovery calls:
• In app-config-rhdh.yaml https://github.com/redhat-developer/rhdh/blob/d434aa5/.ibm/pipelines/resources/config_map/app-config-rhdh.yaml#L194-L200
• In values_showcase.yaml as dynamic plugin config: https://github.com/redhat-developer/rhdh/blob/d434aa5/.ibm/pipelines/value_files/values_showcase.yaml#L126-L142

5. githubOrg Provider Configured Twice  (Approx impact 5-10%)
The githubOrg provider is configured twice with different settings. Although these configurations are merged at runtime, they should exist in only one place. * https://github.com/redhat-developer/rhdh/blob/d434aa5/.ibm/pipelines/resources/config_map/app-config-rhdh.yaml#L201-L204

• https://github.com/redhat-developer/rhdh/blob/d434aa5/.ibm/pipelines/value_files/values_showcase.yaml#L166-L182

We should probably keep the second configuration and extend the frequency to 24 hours.

6.  We all have already discussed this for release-x.y : Separate GitHub Apps.
Finally, using separate GitHub Apps would further reduce the rate limit issue.
It would be better to have a separate app for each deployment within the same job (e.g., showcase and showcase-rbac) rather than having different apps for release branches (or both). Currently, three deployments including showcase-sanity-plugins share the same GitHub App, which increases the chance of hitting rate limits.

Background/Feature Origin

Why is this important?

To avoid unnecessary flakiness in our CI jobs

User Scenarios

Dependencies (internal and external)

Acceptance Criteria

Release Enablement/Demo - Provide necessary release enablement details
and documents

DEV - Upstream code and tests merged: <link to meaningful PR or GitHub
Issue>

DEV - Upstream documentation merged: <link to meaningful PR or GitHub
Issue>

DEV - Downstream build attached to advisory: <link to errata>

QE - Test plans in Playwright: <link or reference to playwright>

QE - Automated tests merged: <link or reference to automated tests>

DOC - Downstream documentation merged: <link to meaningful PR>