Description of problem:

On high load it was observed that paac when using a max-keep-run and concurrency would get out of order scheduling pipelineruns.

Prerequisites (if any, like setup, operators/versions):

Steps to Reproduce

run a pull request with 3 pipelineruns, a max-keep-run of 2, a concurrency of 5 on repo CR.

send over 50 /retest on the Pull Request..

You should observe:

• 5 PipelineRuns Running at all times
• every other PipelineRuns as Pending.
• A max of 6 PipelineRuns as Succeeded
• 150 PipelineRuns spawned in total

At the end you should oberve:

• 6 PipelineRuns left
• 3 check-runs updated on the Github interface.

You can track memory consumptions of the controllers/watcher as well to make sure it's linear..

Actual results:

Expected results:

Reproducibility (Always/Intermittent/Only Once):

Acceptance criteria: 

Definition of Done:

Build Details:

Additional info (Such as Logs, Screenshots, etc):

 *