While installing and applying the du profile to many disconnected clusters via ACM/ZTP, 2 clusters failed to install any du profile operators because their catalogsources remained stuck in CONNECTING/TRANSIENT_FAILURE state.  This prevented those clusters from completely rolling out the du profile.

Hub OCP - 4.14.0-rc.2
Deployed clusters 4.14.0-rc.2
ACM - 2.9.0-DOWNSTREAM-2023-09-27-22-12-46

2 out of 4 failures related to the du profile being applied. (2 out of 1816 clusters which had the du profile applied to)

1.
2.
3.

Affected clusters:
standard-00172 - 5 node cluster
vm01131 - Single Node OpenShift

# oc --kubeconfig /root/hv-vm/kc/standard-00172/kubeconfig get catalogsource -n openshift-marketplace -o yaml
apiVersion: v1
items:
- apiVersion: operators.coreos.com/v1alpha1
  kind: CatalogSource
  metadata:
    annotations:
      scale-test-label-1: value3
      scale-test-label-2: value4
      target.workload.openshift.io/management: '{"effect": "PreferredDuringScheduling"}'
    creationTimestamp: "2023-10-02T22:52:44Z"
    generation: 1
    name: rh-du-operators
    namespace: openshift-marketplace
    resourceVersion: "908795"
    uid: f851a104-7a05-4b98-b2b0-1f18e82b3035
  spec:
    displayName: disconnected-redhat-operators
    image: e27-h01-000-r650.rdu2.scalelab.redhat.com:5000/olm-mirror/redhat-operator-index:v4.13
    publisher: Red Hat
    sourceType: grpc
    updateStrategy:
      registryPoll:
        interval: 1h
  status:
    connectionState:
      address: rh-du-operators.openshift-marketplace.svc:50051
      lastConnect: "2023-10-04T14:48:55Z"
      lastObservedState: TRANSIENT_FAILURE
    latestImageRegistryPoll: "2023-10-04T13:53:05Z"
    registryService:
      createdAt: "2023-10-02T22:52:45Z"
      port: "50051"
      protocol: grpc
      serviceName: rh-du-operators
      serviceNamespace: openshift-marketplace
kind: List
metadata:
  resourceVersion: ""

# oc --kubeconfig /root/hv-vm/kc/vm01131/kubeconfig get catalogsource -n openshift-marketplace -o yaml
apiVersion: v1
items:
- apiVersion: operators.coreos.com/v1alpha1
  kind: CatalogSource
  metadata:
    annotations:
      scale-test-label-1: value3
      scale-test-label-2: value4
      target.workload.openshift.io/management: '{"effect": "PreferredDuringScheduling"}'
    creationTimestamp: "2023-10-03T02:28:33Z"
    generation: 1
    name: rh-du-operators
    namespace: openshift-marketplace
    resourceVersion: "517268"
    uid: cc30a95e-63b0-4b61-8f98-a0cbd809fd3f
  spec:
    displayName: disconnected-redhat-operators
    image: e27-h01-000-r650.rdu2.scalelab.redhat.com:5000/olm-mirror/redhat-operator-index:v4.13
    publisher: Red Hat
    sourceType: grpc
    updateStrategy:
      registryPoll:
        interval: 1h
  status:
    connectionState:
      address: rh-du-operators.openshift-marketplace.svc:50051
      lastConnect: "2023-10-04T14:48:13Z"
      lastObservedState: CONNECTING
    latestImageRegistryPoll: "2023-10-04T14:28:37Z"
    registryService:
      createdAt: "2023-10-03T02:28:33Z"
      port: "50051"
      protocol: grpc
      serviceName: rh-du-operators
      serviceNamespace: openshift-marketplace
kind: List
metadata:
  resourceVersion: ""