Backstory of this issue
During performance testing of Chains signing with large pipeline runs, we observed potential regressions in OpenShift Pipelines 1.19 compared to 1.18, especially at higher workloads.

For 7500 PipelineRuns (PRs), signing duration shows a clear regression across multiple setups:

• Default Setup

• • 1.18: 3676s

• • 1.19: 3691s

• HA Setup

• • 1.18: 1047s

• • 1.19: 1493s

• QBT (QPS, Burst, Threads)

• • 1.18: 947s

• • 1.19: 967s

• HA + QBT

• • 1.18: 316s

• • 1.19: 386s

The gap is significant for the HA setup and visible in other configurations as well.
While performance was consistent for 2500 and 5000 PRs, the regression emerges at 7500 PRs.

Acceptance Criteria 

• Run regression tests with 10000 and 12500 PRs to confirm whether the regression persists or worsens in 1.19.

• Collect and analyze logs to identify potential bottlenecks or causes of the delay in signing.

• Compare resource utilization and behavior across setups to isolate contributing factors.

• Present those findings to Engineering Team