Description of problem:
when we check the olm qe automation case, we found some cases fails:
when creating sub, it get error in conditions, like
—
message: 'error using catalog cs-43073 (in namespace openshift-marketplace): failed
to list bundles: rpc error: code = Unavailable desc = connection error: desc
= "transport: Error while dialing dial tcp: lookup cs-43073.openshift-marketplace.svc
on 172.30.0.10:53: no such host"'
reason: ErrorPreventedResolution
status: "True"
type: ResolutionFailed
lastUpdated: "2021-11-05T00:45:42Z"
—
Actually case 43073 is already executed successfully and the catsrc cs-43073 is delete successfully.

it has nothing with case 43073 because other case also trigger this error.
I tried to manually reproduce it with creating and deleting catsrc in openshift-marketplace and creating sub in default ns. failed to reproduced.

but it does happen in recent auto case regression.

maybe there is race condition for some cache. during auto regresion, the catsrc is deleted indeed, but when creating sub, it think the catsrc still exists during catalogHealth, but meet this error.
I guess it is why it happen during auto case regression, and difficult to reproduce it manually.

Per our current observation, it happened for 4.10, and does not happen in 4.9.
Please help check it because it impacts the automation regression seriously.

here is the sub output from auto case execution.FYI:
–
spec:
channel: stable-5.x
installPlanApproval: Automatic
name: cockroachdb
source: catsrc-33241-operator
sourceNamespace: openshift-marketplace
startingCSV: ""
status:
catalogHealth:

catalogSourceRef:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
name: bucket-test1
namespace: openshift-marketplace
resourceVersion: "631015"
uid: 24cf00c1-a438-424d-a14a-be1e6ae789c2
healthy: true
lastUpdated: "2021-11-05T00:42:42Z"

catalogSourceRef:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
name: bucket-test2
namespace: openshift-marketplace
resourceVersion: "630644"
uid: 6e9de4f7-e5c0-458e-8240-0ec792936721
healthy: true
lastUpdated: "2021-11-05T00:42:42Z"

catalogSourceRef:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
name: catsrc-33241-operator
namespace: openshift-marketplace
resourceVersion: "632418"
uid: 8a013fb8-1c4b-47bd-a438-02951b667dd0
healthy: true
lastUpdated: "2021-11-05T00:42:42Z"

catalogSourceRef:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
name: certified-operators
namespace: openshift-marketplace
resourceVersion: "631163"
uid: 43c7d50a-160d-45d6-af25-f589d5d6e963
healthy: true
lastUpdated: "2021-11-05T00:42:42Z"

catalogSourceRef:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
name: community-operators
namespace: openshift-marketplace
resourceVersion: "631030"
uid: 58175359-2a9f-47bc-b3a6-6a1d8d9ffd05
healthy: true
lastUpdated: "2021-11-05T00:42:42Z"

catalogSourceRef:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
name: ocs-cs
namespace: openshift-marketplace
resourceVersion: "631673"
uid: 71d64cc4-0db3-4b8f-b819-55181b7b1031
healthy: true
lastUpdated: "2021-11-05T00:42:42Z"

catalogSourceRef:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
name: qe-app-registry
namespace: openshift-marketplace
resourceVersion: "630907"
uid: 9a0bc1e5-7d63-4e77-a9cc-ef88f448ffde
healthy: true
lastUpdated: "2021-11-05T00:42:42Z"

catalogSourceRef:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
name: redhat-marketplace
namespace: openshift-marketplace
resourceVersion: "630627"
uid: 8553db56-9ce5-4193-8a07-78d0ca57ecc1
healthy: true
lastUpdated: "2021-11-05T00:42:42Z"

catalogSourceRef:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
name: redhat-operators
namespace: openshift-marketplace
resourceVersion: "630983"
uid: e76b2bf8-fc68-4244-a7ce-3e62c041c709
healthy: true
lastUpdated: "2021-11-05T00:42:42Z"
conditions:

lastTransitionTime: "2021-11-05T00:42:42Z"
message: all available catalogsources are healthy
reason: AllCatalogSourcesHealthy
status: "False"
type: CatalogSourcesUnhealthy

message: 'error using catalog cs-43073 (in namespace openshift-marketplace): failed
to list bundles: rpc error: code = Unavailable desc = connection error: desc
= "transport: Error while dialing dial tcp: lookup cs-43073.openshift-marketplace.svc
on 172.30.0.10:53: no such host"'
reason: ErrorPreventedResolution
status: "True"
type: ResolutionFailed
lastUpdated: "2021-11-05T00:45:42Z"
–