EPIC Goal

What are we trying to solve here?

Kueue, the job queuing system in Konflux, is capped at 60 concurrent build requests.

But when I make a global change to the plugin pipeline and need to roll it out to the 80+ plugin builds, at least 20 of those attempted pipelineruns fail immediately.

In thread https://redhat-internal.slack.com/archives/C04PZ7H0VA8/p1761338199396799?thread_ts=1761337848.790629&cid=C04PZ7H0VA8 two suggestions were made:

Stagger your build triggers. The primary issue is submitting more concurrent builds (80) than your quota allows (60). Modify your process to trigger builds in smaller batches that fit within the limit.

Or,

If a higher concurrency limit is essential for your workflow, you must file a request with the Konflux operations team to have the `count/statefulsets.apps` limit in the `for-rhdh-jobs` ResourceQuota increased for your tenant.

If we can get a higher limit for the plugin builds, cool... but if not then we need to build our own Kueue-queuer because of this baked-in limit. I mean who would ever want to use Konflux for 100 container builds for a single product release?

Instead of queuing we might have to implement a system where:

• changes to shared template pipeline .tekton/oci-plugin-build-pipeline.yaml do not trigger all the builds to fire, and
• .tekton/updateDigests.sh and .tekton/generatePipelineRunsForPlugins.sh are limited to MAX 50 changed files per commit, with a 2 min cooling off period between automated pushes