Description of problem:

No way to create cluster nodes if they are lost/deleted, unless indexer pod (search) restarts. One impact from application point of view, it shows the cluster as offline un Subscription.

Version-Release number of selected component (if applicable): v2.9.0-188

How reproducible:

When cluster nodes are deleted/lost, can happen if postgres pod restarts for some reason.

Steps to Reproduce:

1. Log into ACM (with spoke/s), create a subscription based application
2. Restart postgres pod (search)
3. Check app topology page

Actual results:

Spoke is appearing offline under subscription

Expected results:

Spoke is showing online under subscription

Additional info:

here, managed cluster mist10-0 status is not reported correctly. The managed cluster is online:

% oc get managedclusters 
NAME HUB ACCEPTED MANAGED CLUSTER URLS JOINED AVAILABLE AGE
local-cluster true 
https://api.mist12-0.qe.red-chesterfield.com:6443
True True 75d
mist10-0 true 
https://api.mist10-0.qe.red-chesterfield.com:6443
True True 74d 

 
Checking the appsub deployed on the minst10-0 managed cluster, it's deployed the configmap as expected. And all appsub status are reported correctly in the backend.