Description of problem:

When running with High Availability (HA) enabled and varying concurrency levels (10, 20, 30, 40), a noticeable increase in both PipelineRun (PR) and TaskRun (TR) durations was observed in version 1.19 compared to 1.18. This increase is consistent across all tested concurrency levels. Additionally, metrics such as TaskRun-to-pod-creation exhibited a similar increase, while the workqueue depth of the Pipelines Controller decreased in 1.19, which warrants attention.

Test Setup:

• HA Setup: Enabled

• Tested Concurrency Levels: 10, 20, 30, 40

• Metrics: Average duration (in seconds) for PR and TR executions.

Observed Results:

Impact:

• Increased latency in TaskRun and PipelineRun execution in HA setup for 1.19.

• Potential performance regression compared to 1.18.

• Could affect scalability and throughput in production environments.

Expected Behavior:
PR/TR durations should remain consistent or improve with newer versions under similar load and HA configurations.

Pipelines Controller - HA

Report: https://docs.google.com/document/d/156iBOAZA6DRd7m8C1WMHSeHL83HhIqG1V1_8HHtTliU/edit?tab=t.w8jk5w8z8qgt
ProwJob:

• 1.18 (Left in chart): 

  1946009077700104192

  1945753385374322688

• 1.19 (Right in chart):

Prerequisites (if any, like setup, operators/versions):

Steps to Reproduce

Trigger jobs in CI release with prow commands as shown here:

• https://github.com/openshift/release/pull/67153

/pj-rehearse pull-ci-openshift-pipelines-performance-main-max-concurrency-downstream-1-19-1000-x-math-ha-10

Actual results:

Expected results:

Reproducibility (Always/Intermittent/Only Once):

Acceptance criteria: 

Definition of Done:

Build Details:

Additional info (Such as Logs, Screenshots, etc):

 *