Failed to pull pod images for amq-operator and certified-operators-custom catalogsources.

image: registry.kni-qe-60.lab.eng.rdu2.redhat.com:5000/olm/far-edge-sriov-fec:v4.14
image: registry.kni-qe-60.lab.eng.rdu2.redhat.com:5000/olm/amq-operator:v4.14

============Logs Snippet==================
violation - catalogsources not found: [certified-operators-custom]
        in namespace openshift-marketplace found but not as specified; violation -
        catalogsources not found: [amq-operator] in namespace openshift-marketplace
        found but not as specified'

[~]$ oc -n openshift-marketplace logs amq-operator-7d2dh
Error from server (BadRequest): container "registry-server" in pod "amq-operator-7d2dh" is waiting to start: trying and failing to pull image
[ ~]$ 

[ ~]$ oc -n openshift-marketplace get po
NAME                                                              READY   STATUS             RESTARTS   AGE
amq-operator-79n69                                                0/1     ImagePullBackOff   0          8h
amq-operator-7d2dh                                                0/1     ImagePullBackOff   0          7h53m
c57c9f9735d5e773fc76c0511d80c0a6fc72c75c529b18baabe03d683a49qwc   0/1     Completed          0          16h
certified-operators-custom-jf6br                                  0/1     ImagePullBackOff   0          7h53m
certified-operators-custom-qz5cv                                  0/1     ImagePullBackOff   0          8h
redhat-operators-custom-lw66h                                     1/1     Running            0          8h
[ ~]$ 

[ ~]$ oc -n openshift-marketplace logs certified-operators-custom-jf6br
Error from server (BadRequest): container "registry-server" in pod "certified-operators-custom-jf6br" is waiting to start: trying and failing to pull image
[ ~]$ 
==============================================

the above mentioned pods are controlled by respective catalogsources - certified-operators-custom and amq-operator catalog sources are not in READY state. need to debug what makes it become TRANSIENT_FAILURE state. 

4.14.0-0.nightly-2023-08-08-094653

Encountered Once

create catalogsource CRs

lastObservedState: TRANSIENT_FAILURE

lastObservedState: READY

[ ~]$ oc -n openshift-marketplace get catalogsources.operators.coreos.com certified-operators-custom -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  annotations:
    target.workload.openshift.io/management: '
{"effect": "PreferredDuringScheduling"}
'
  creationTimestamp: "2023-08-08T16:32:03Z"
  generation: 2
  name: certified-operators-custom
  namespace: openshift-marketplace
  resourceVersion: "299588"
  uid: 4dccc55b-7e71-4e5e-bd21-f254f7aacba1
spec:
  displayName: Intel SRIOV-FEC Operator
  image: registry.kni-qe-60.lab.eng.rdu2.redhat.com:5000/olm/far-edge-sriov-fec:v4.14
  publisher: Red Hat
  sourceType: grpc
  updateStrategy:
    registryPoll:
      interval: 1h
status:
  connectionState:
    address: certified-operators-custom.openshift-marketplace.svc:50051
    lastConnect: "2023-08-09T09:38:39Z"
    lastObservedState: TRANSIENT_FAILURE
  latestImageRegistryPoll: "2023-08-09T01:34:44Z"
  registryService:
    createdAt: "2023-08-09T01:15:07Z"
    port: "50051"
    protocol: grpc
    serviceName: certified-operators-custom
    serviceNamespace: openshift-marketplace
[kni@registry.kni-qe-60 ~]$ oc -n openshift-marketplace get catalogsources.operators.coreos.com amq-operator -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  annotations:
    target.workload.openshift.io/management: '
{"effect": "PreferredDuringScheduling"}
'
  creationTimestamp: "2023-08-08T16:32:09Z"
  generation: 2
  name: amq-operator
  namespace: openshift-marketplace
  resourceVersion: "299728"
  uid: ad6e4dd7-14f1-4490-943f-fd844d3c116d
spec:
  displayName: AMQ Operator
  image: registry.kni-qe-60.lab.eng.rdu2.redhat.com:5000/olm/amq-operator:v4.14
  publisher: Red Hat
  sourceType: grpc
  updateStrategy:
    registryPoll:
      interval: 1h
status:
  connectionState:
    address: amq-operator.openshift-marketplace.svc:50051
    lastConnect: "2023-08-09T09:39:05Z"
    lastObservedState: TRANSIENT_FAILURE
  latestImageRegistryPoll: "2023-08-09T01:34:45Z"
  registryService:
    createdAt: "2023-08-09T01:15:08Z"
    port: "50051"
    protocol: grpc
    serviceName: amq-operator
    serviceNamespace: openshift-marketplace
[ ~]$