Description of problem:

During a scale test of StoneSoup, build pipelines began to fail at random because the pipeline service account did not have permission to use the `pipeline-scc` cluster role.

Prerequisites (if any, like setup, operators/versions):

OpenShift Pipelines 1.8

ROSA cluster with 4 m5.xlarge worker nodes
Pipelines controller scaled to 15 replicas, webhooks scaled to 10 replicas

Steps to Reproduce

1. Run a script that does the following repeatedly:
   1. Create a user namespace
   2. Launch a sample StoneSoup build pipeline.

Actual results:

Pipelines begin to fail on tasks that run root containers or otherwise require "baseline" privileges.

code
// failed to create task run pod "loadtestsxfetfjp8752-tenant-component-828kt-clone-repository": pods "loadtestsxfetfjp8752-tenant14117cf396e0107f9af9e99e7f03c1d1-pod" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "pipelines-scc": Forbidden: not usable by user or serviceaccount, spec.containers[0].securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1001560000, 1001569999], provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, spec.containers[0].securityContext.runAsUser: Invalid value: 0: running with the root UID is forbidden, provider "csi-scc": Forbidden: not usable by user or serviceaccount, provider "pcap-dedicated-admins": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "splunkforwarder": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]. Maybe missing or invalid Task loadtestsxfetfjp8752-tenant/git-clone
code

Expected results:

Pipelines succeed

Reproducibility (Always/Intermittent/Only Once):

Intermittent

Build Details:

Additional info (Such as Logs, Screenshots, etc):

Slack thread: https://redhat-internal.slack.com/archives/C02FTKEMBU1/p1675882005320469

If possible, the Pipeline Service team would like to track the time it takes to set up the Pipeline SA (create it, set up permissions, etc.) in Prometheus so we can establish a service level objective.