Description of problem:

With a Repository's concurrency_limit set to 1, periodically the pac-watcher will get into a state in which it's internal concurrency queue contains no pending pipelineruns in spite of there being many pipelineruns pending in the cluster. In this state, new PipelineRuns are created in a pending state, but since no pipelineruns are running for the given Repository, no PipelineRuns are completed, so the pac-watcher never starts any pipelineruns.

The issue appears to be due to two issues:

1. When a PLR is removed from the queue manager's queue to be started, but fails to be started (e.g. due to a k8s patch conflict or transient error), the still-pending PLR is not placed back into the queue manager's queue.
2. Since the queue manager only starts the next PipelineRun when a running PLR completes, once all PLRs are in a pending state the watcher will not get out of the bad state without manual intervention.

Workaround

The only workround is to manually patch the PLR to remove the queued state allowing it to start. However this is not always possible for unprivileged users who are experiencing their PLRs being stuck in pending. 

Prerequisites (if any, like setup, operators/versions):

Steps to Reproduce

 # <steps>

Actual results:

Expected results:

Reproducibility (Always/Intermittent/Only Once):

Acceptance criteria: 

Definition of Done:

Build Details:

Additional info (Such as Logs, Screenshots, etc):

 *