Intermittent operator failed rollout, incorrect deployment status

Description of problem:

The tl;dr is in https://redhat-internal.slack.com/archives/CSPS1077U/p1687885706624179

The summary, intermittently, though sometimes frequently:

• on both 1.10 and 1.11
• various component, for example triggers or Pac, have their deployments functional
• but have messages about the components not being in ready state, and needing to reconcile again and proceed in the TektonConfig status
• analogous, continually repeating error logs along those line also appear in the openshift-pipelines-operator pod in the openshift-operators namespace
• one such log was at https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/redhat-appstudio_infra-deployments/2024/pull-ci-redhat-appstudio-infra-deployments-main-appstudio-e2e-tests/1673709729014091776/artifacts/appstudio-e2e-tests/redhat-appstudio-hypershift-gather/artifacts/pods/openshift-operators_openshift-pipelines-operator-cb486dccf-2qkxg_openshift-pipelines-operator-lifecycle.log

Prerequisites (if any, like setup, operators/versions):

We are installing OSP in RHTAP via the yaml at https://github.com/openshift-pipelines/pipeline-service/tree/main/operator/gitops/argocd/pipeline-service/openshift-pipelines

We've tried both with main branch (currently still at 1.10) and our test branches around trying to get 1.11 to work with RHTAP.

Steps to Reproduce

 It is intermittent, but a dummy / test PRs against the https://github.com/redhat-appstudio/infra-deployments repo would be the easiest means.

But hopefully you all can sort out via the controller log provided with those repeating logs.

Seems like a reconcile path is not error'ing out and retrying correctly.

Actual results:{}

Expected results:

Reproducibility (Always/Intermittent/Only Once):

Intermitent

Acceptance criteria: 

Definition of Done:

Build Details:

Additional info (Such as Logs, Screenshots, etc):

 *