OLM operator recurring errors after ansible based operator uninstall

We are experiencing weird behavior with OLM and an ansible based operator, developed for a customer using the Operator SDK.

The custom operator supports single namespace installation only.
When it is uninstalled from the OCP namespace deleting the subscription and then the CSV, the OLM operator pod (in the openshift-operator-lifecycle-manager namespace) starts logging recurring errors, referring to missing OperatorCondition and ClusterServiceVersion resources for the uninstalled operator:

{"level":"error","ts":1683101778.578233,"logger":"controller.clusterserviceversion","msg":"Reconciler error","reconciler group":"operators.coreos.com","reconciler kind":"ClusterServiceVersion","name":"adc3.v0.1.20","namespace":"isp-adc-datastore","error":"ClusterServiceVersion.operators.coreos.com \"adc3.v0.1.20\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

{"level":"error","ts":1683101778.589105,"logger":"controllers.operatorcondition-generator","msg":"Unable to find ClusterServiceVersion","request":"isp-adc-datastore/adc3.v0.1.20","error":"ClusterServiceVersion.operators.coreos.com \"adc3.v0.1.20\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

{"level":"error","ts":1683101778.605787,"logger":"controllers.operatorcondition","msg":"Unable to find operatorcondition","request":"isp-adc-datastore/adc3.v0.1.20","error":"OperatorCondition.operators.coreos.com \"adc3.v0.1.20\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

{"level":"error","ts":1683101778.6060314,"logger":"controller.operatorcondition","msg":"Reconciler error","reconciler group":"operators.coreos.com","reconciler kind":"OperatorCondition","name":"adc3.v0.1.20","namespace":"isp-adc-datastore","error":"OperatorCondition.operators.coreos.com \"adc3.v0.1.20\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

These 4 messages are recurring and a CsvAbnormalOver30Min alert is triggered.

The adc3.v0.1.20 operator is a custom ansible operator based on the base image: quay.io/operator-framework/ansible-operator:v1.20.0

operator-lifecycle-manager 4.10.43

On OCP cluster 4.10 (or 4.8), after installing the custom operator we simply need to uninstall it in order to start seeing the error messages in the olm-operator pod (corresponding alert is generated after the threeshold is reached)

1. oc delete subscriptions.operators.coreos.com intesa-adc-subscription
2. oc delete csv adc3.v0.1.20

OLM operator pod logs:

{"level":"info","ts":1683101727.3684852,"logger":"controllers.adoption","msg":"Could not find Subscription","request":"isp-adc-datastore/intesa-adc-subscription"}

{"level":"error","ts":1683101778.5724018,"logger":"controllers.operatorcondition-generator","msg":"Unable to find ClusterServiceVersion","request":"isp-adc-datastore/adc3.v0.1.20","error":"ClusterServiceVersion.operators.coreos.com \"adc3.v0.1.20\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

{"level":"error","ts":1683101778.572465,"logger":"controller.clusterserviceversion","msg":"Reconciler error","reconciler group":"operators.coreos.com","reconciler kind":"ClusterServiceVersion","name":"adc3.v0.1.20","namespace":"isp-adc-datastore","error":"ClusterServiceVersion.operators.coreos.com \"adc3.v0.1.20\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

time="2023-05-03T08:16:18Z" level=info msg="gcing children" csv=adc3.v0.1.20 id=GKMKo namespace=isp-adc-datastore phase=Succeeded

{"level":"error","ts":1683101778.5781834,"logger":"controllers.operatorcondition-generator","msg":"Unable to find ClusterServiceVersion","request":"isp-adc-datastore/adc3.v0.1.20","error":"ClusterServiceVersion.operators.coreos.com \"adc3.v0.1.20\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

{"level":"error","ts":1683101778.578233,"logger":"controller.clusterserviceversion","msg":"Reconciler error","reconciler group":"operators.coreos.com","reconciler kind":"ClusterServiceVersion","name":"adc3.v0.1.20","namespace":"isp-adc-datastore","error":"ClusterServiceVersion.operators.coreos.com \"adc3.v0.1.20\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

{"level":"error","ts":1683101778.589105,"logger":"controllers.operatorcondition-generator","msg":"Unable to find ClusterServiceVersion","request":"isp-adc-datastore/adc3.v0.1.20","error":"ClusterServiceVersion.operators.coreos.com \"adc3.v0.1.20\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

{"level":"error","ts":1683101778.605787,"logger":"controllers.operatorcondition","msg":"Unable to find operatorcondition","request":"isp-adc-datastore/adc3.v0.1.20","error":"OperatorCondition.operators.coreos.com \"adc3.v0.1.20\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

{"level":"error","ts":1683101778.6060314,"logger":"controller.operatorcondition","msg":"Reconciler error","reconciler group":"operators.coreos.com","reconciler kind":"OperatorCondition","name":"adc3.v0.1.20","namespace":"isp-adc-datastore","error":"OperatorCondition.operators.coreos.com \"adc3.v0.1.20\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

No errors regarding missing CSV or OperatorCondition for the uninstalled operator.

The error messages are recurring and alert is being triggered continuously until the olm-operator pod is restarted.