Feature Overview:

As we have expanded our use of gitlab jobs in new pipelines for managing the package index, we have started seeing capacity issues with runner hosts for aarch64 and x86_64 hosts. We need to significantly increase the number of runners we have available. This may require restructuring how we manage those runners (a larger VPC, multiple VPCs, etc.).

Product(s) associated:

RHAIIS: yes
RHEL AI: yes
RHOAI: yes

Goals:
Developers and component teams will all benefit from having more capacity for CI jobs.

Requirements:
We need to analyze the RHAI and builder repos to determine a reasonable maximum number of concurrent jobs and then ensure that our build infrastructure can support that. We want to support several sets of jobs for MRs in the builder repo, for example, as well as 1 merge pipeline there and another in the RHAI repo. 

Then we need to also consider the number of other repositories that use the runners and ensure that we do not starve them, either.

Done - Acceptance Criteria:
This work will be done when we set a new maximum number of jobs.

Use Cases - i.e. User Experience & Workflow:
As a developer I want to submit multiple MRs to the builder repo to have them tested in parallel.

As a component owner, I want to have my packages built in the RHAI repo without waiting for developers' MRs to be tested.

Out of Scope:
We should focus on the number of runners as part of this work, and not change the sizes of the nodes.

Documentation Considerations :
N/A