When you deploy CRO on a cluster with more operand replicas than nodes, the extra operand pods will go into pending mode and cause the operator to constantly reconcile until reducing the number of replicas (using deploymentOverrides in the CR)

    4.17 (since this is the version that we switched from daemonset to deployment), 4.18, 4.19 

    Every time you have the condition above.

    1. Install CRO on a cluster.
    2. Take note of how many nodes you have in the cluster but it's easiest to do on a hypershift cluster (1 worker/control-plane node)
    3. By default, CRO will deploy with 2 replicas.
    4. The second CRO pod will not deploy and be stuck in pending because it cannot be scheduled.

oc get pods -n clusterresourceoverride-operator 
NAME                                                        READY   STATUS    RESTARTS   AGE
clusterresourceoverride-55d67cff7-g5mtt                     1/1     Running   0          7m36s
clusterresourceoverride-55d67cff7-p6q4d                     0/1     Pending   0          7m36s
clusterresourceoverride-operator-774696fdc8-nh244           1/1     Running   0          8m10s
clusterresourceoverride-operator-registry-b766fccb5-7mxtd   1/1     Running   0          8m21s

The operator waits for all operand pods to be healthy but cannot since some pods will stay pending forever, which will block the operator from doing anything until the problem is fixed by the cluster admin.


W1212 22:44:40.688982       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1212 22:44:40.691511       1 start.go:100] [operator] configuration - name=clusterresourceoverride namespace=clusterresourceoverride-operator operand-image=quay.io/macao/clusterresourceoverride:dev operand-version=1.0.0
I1212 22:44:40.691532       1 start.go:101] [operator] starting
I1212 22:44:40.692632       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1212 22:44:40.692826       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1212 22:44:40.702159       1 reflector.go:368] Caches populated for *v1.MutatingWebhookConfiguration from k8s.io/client-go@v0.31.2/tools/cache/reflector.go:243
I1212 22:44:40.702536       1 reflector.go:368] Caches populated for *v1.DaemonSet from k8s.io/client-go@v0.31.2/tools/cache/reflector.go:243
I1212 22:44:40.703983       1 reflector.go:368] Caches populated for *v1.Service from k8s.io/client-go@v0.31.2/tools/cache/reflector.go:243
I1212 22:44:40.704721       1 reflector.go:368] Caches populated for *v1.ServiceAccount from k8s.io/client-go@v0.31.2/tools/cache/reflector.go:243
I1212 22:44:40.705898       1 reflector.go:368] Caches populated for *v1.Pod from k8s.io/client-go@v0.31.2/tools/cache/reflector.go:243
I1212 22:44:40.705928       1 reflector.go:368] Caches populated for *v1.ConfigMap from k8s.io/client-go@v0.31.2/tools/cache/reflector.go:243
I1212 22:44:40.706121       1 reflector.go:368] Caches populated for *v1.Deployment from k8s.io/client-go@v0.31.2/tools/cache/reflector.go:243
I1212 22:44:40.708418       1 reflector.go:368] Caches populated for *v1.Secret from k8s.io/client-go@v0.31.2/tools/cache/reflector.go:243
I1212 22:44:40.793171       1 runner.go:39] [controller] name=clusterresourceoverride starting informer
I1212 22:44:40.793590       1 runner.go:42] [controller] name=clusterresourceoverride waiting for informer cache to sync
I1212 22:44:40.796374       1 reflector.go:368] Caches populated for *v1.ClusterResourceOverride from k8s.io/client-go@v0.31.2/tools/cache/reflector.go:243
I1212 22:44:40.894722       1 worker.go:22] [controller] name=clusterresourceoverride starting to process work item(s)
I1212 22:44:40.894667       1 runner.go:52] [controller] name=clusterresourceoverride started 1 worker(s)
I1212 22:44:40.896546       1 runner.go:54] [controller] name=clusterresourceoverride waiting 
I1212 22:44:40.896621       1 run.go:102] operator is waiting for controller(s) to be done
I1212 22:44:40.896649       1 start.go:110] [operator] operator is running, waiting for the operator to be done.
I1212 22:45:04.607820       1 configuration.go:59] key=cluster resource=*v1.ConfigMap/clusterresourceoverride-configuration successfully created
I1212 22:45:04.607843       1 configuration.go:74] key=cluster resource=*v1.ConfigMap/clusterresourceoverride-configuration configuration has drifted
I1212 22:45:04.619560       1 configuration.go:94] key=cluster resource=*v1.ConfigMap/clusterresourceoverride-configuration resource-version=1274064 setting object reference
I1212 22:45:04.665798       1 service.go:52] key=cluster resource=*v1.Service/clusterresourceoverride successfully created
I1212 22:45:04.665838       1 service.go:66] key=cluster resource=*v1.Service/clusterresourceoverride resource-version=1274066 setting object reference
I1212 22:45:04.681943       1 deploy.go:195] key=cluster ensured RBAC resource clusterresourceoverride
I1212 22:45:04.768744       1 deploy.go:195] key=cluster ensured RBAC resource extension-server-authentication-reader-clusterresourceoverride
I1212 22:45:04.810555       1 deploy.go:195] key=cluster ensured RBAC resource system:clusterresourceoverride-requester
I1212 22:45:04.872128       1 deploy.go:195] key=cluster ensured RBAC resource default-aggregated-apiserver-clusterresourceoverride
I1212 22:45:04.905692       1 deploy.go:195] key=cluster ensured RBAC resource default-aggregated-apiserver-clusterresourceoverride
I1212 22:45:04.941150       1 deploy.go:195] key=cluster ensured RBAC resource auth-delegator-clusterresourceoverride
I1212 22:45:05.014306       1 deploy.go:93] key=cluster resource=*v1.Deployment/clusterresourceoverride successfully ensured
I1212 22:45:05.014335       1 deploy.go:107] key=cluster resource=*v1.Deployment/clusterresourceoverride resource-version=1274082 setting object reference
I1212 22:45:05.014347       1 deployment_ready.go:37] key=cluster resource=clusterresourceoverride deployment is not ready
E1212 22:45:05.027072       1 worker.go:67] "Unhandled Error" err="error syncing '/cluster': waiting for deployment spec update name=clusterresourceoverride, requeuing"
I1212 22:45:05.028613       1 configuration.go:69] key=cluster resource=*v1.ConfigMap/clusterresourceoverride-configuration is in sync
I1212 22:45:05.028645       1 service.go:66] key=cluster resource=*v1.Service/clusterresourceoverride resource-version=1274078 setting object reference
I1212 22:45:05.036936       1 deploy.go:80] operand image mismatch: current:  original: quay.io/macao/clusterresourceoverride:dev
I1212 22:45:05.621593       1 deploy.go:195] key=cluster ensured RBAC resource clusterresourceoverride
I1212 22:45:06.219076       1 deploy.go:195] key=cluster ensured RBAC resource extension-server-authentication-reader-clusterresourceoverride
I1212 22:45:06.861477       1 deploy.go:195] key=cluster ensured RBAC resource system:clusterresourceoverride-requester
I1212 22:45:07.427912       1 deploy.go:195] key=cluster ensured RBAC resource default-aggregated-apiserver-clusterresourceoverride
I1212 22:45:08.019772       1 deploy.go:195] key=cluster ensured RBAC resource default-aggregated-apiserver-clusterresourceoverride
I1212 22:45:08.612824       1 deploy.go:195] key=cluster ensured RBAC resource auth-delegator-clusterresourceoverride
I1212 22:45:09.224911       1 deploy.go:93] key=cluster resource=*v1.Deployment/clusterresourceoverride successfully ensured
I1212 22:45:09.224943       1 deploy.go:107] key=cluster resource=*v1.Deployment/clusterresourceoverride resource-version=1274120 setting object reference
I1212 22:45:09.224959       1 deployment_ready.go:37] key=cluster resource=clusterresourceoverride deployment is not ready
E1212 22:45:09.232968       1 worker.go:67] "Unhandled Error" err="error syncing '/cluster': waiting for deployment spec update name=clusterresourceoverride, requeuing"

The pod extra pod should get scheduled anyways and not block the operator from working.   

    This bug was introduced since we added this pod anti-affinity spec in the CRO pod spec definition:

affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: clusterresourceoverride
operator: In
values:
- 'true'
topologyKey: kubernetes.io/hostname