Description of problem:

met bug https://issues.redhat.com/browse/OCPBUGSM-23122  again

The olm-operator pod jumps to 100% CPU utilization if Pending CSV replaces itself

Version-Release number of selected component (if applicable):

release-4.15  

How reproducible:

not always    

Steps to Reproduce:

1, create ns olm, create og
zhaoxia@xzha-mac OCP-40316 % oc create ns olm
namespace/olm created
zhaoxia@xzha-mac OCP-40316 % cat og.yaml 
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: default-og
  namespace: olm
spec:
  targetNamespaces:
  - olm

2, create bad csv
https://bugzilla-attachments.redhat.com/attachment.cgi?id=1747237

3, check CPU usage of olm operator
zhaoxia@xzha-mac OCP-40316 % oc adm top pods olm-operator-85c465b9f5-8cvr2 -n openshift-operator-lifecycle-manager

During daily CI, the cpu usage may higher than 100m

vsphere-ipi-disk-encryption-tang-fips: 4.15.0-0.nightly-2023-11-18-191907: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/470371

Nov 19 03:28:40.227: INFO: Running 'oc --kubeconfig=/tmp/kubeconfig-4169283463 adm top pods -l app=olm-operator -n openshift-operator-lifecycle-manager --no-headers'
Nov 19 03:28:41.366: INFO: result: olm-operator-75cdf546b5-6s6x8 122m 201Mi

baremetalds-ipi-ovn-dualstack-primaryv6: 4.15.0-0.nightly-2023-11-19-145055: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/470530

 Nov 20 03:43:24.973: INFO: Running 'oc --kubeconfig=/tmp/kubeconfig-2364663174 adm top pods -l app=olm-operator -n openshift-operator-lifecycle-manager --no-headers'
Nov 20 03:43:26.144: INFO: result: olm-operator-5444546b6c-9z6s4 108m 124Mi

azure-ipi-fullyprivate-proxy: 4.15.0-0.nightly-2023-11-20-045323: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/470871

Nov 20 19:10:27.513: INFO: Running 'oc --kubeconfig=/tmp/kubeconfig-1250177599 adm top pods -l app=olm-operator -n openshift-operator-lifecycle-manager --no-headers'
Nov 20 19:10:29.098: INFO: result: olm-operator-7bf64695f4-qlkh2 145m 139Mi

aws-ipi-ovn-efs: 4.15.0-0.nightly-arm64-2023-11-20-115847: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/471028

Nov 21 01:07:11.614: INFO: Running 'oc --kubeconfig=/tmp/kubeconfig-1924753740 adm top pods -l app=olm-operator -n openshift-operator-lifecycle-manager --no-headers'
Nov 21 01:07:12.835: INFO: result: olm-operator-78b5489d95-77vwr 143m 159Mi

azure-ipi-sno: 4.15.0-0.nightly-2023-11-20-205649: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/471712

Nov 22 08:16:18.512: INFO: Running 'oc --kubeconfig=/tmp/kubeconfig-1591226758 adm top pods -l app=olm-operator -n openshift-operator-lifecycle-manager --no-headers'
Nov 22 08:16:20.076: INFO: result: olm-operator-54f746c495-wkqsn 119m 152Mi

gcp-ipi-compact-filestore-csi-amd: 4.15.0-0.nightly-multi-2023-11-14-154423: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/471895

Nov 15 10:36:19.360: INFO: Running 'oc --kubeconfig=/tmp/kubeconfig-1695293809 adm top pods -l app=olm-operator -n openshift-operator-lifecycle-manager --no-headers'
Nov 15 10:36:20.928: INFO: result: olm-operator-6d6fcff675-v6tq8 140m 150Mi

gcp-ipi-disk-encryption-amd: 4.15.0-0.nightly-2023-11-08-062604: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/471898

Nov 13 05:45:59.052: INFO: Running 'oc --kubeconfig=/tmp/kubeconfig-867271747 adm top pods -l app=olm-operator -n openshift-operator-lifecycle-manager --no-headers'
Nov 13 05:46:00.461: INFO: result: olm-operator-6c87b68c5c-4b8vq 109m 157Mi

gcp-ipi-sdn-amd: 4.15.0-0.nightly-multi-2023-11-14-154423: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/471906

Nov 15 08:23:24.077: INFO: Running 'oc --kubeconfig=/tmp/kubeconfig-1708098826 adm top pods -l app=olm-operator -n openshift-operator-lifecycle-manager --no-headers'
Nov 15 08:23:24.543: INFO: result: olm-operator-6d6fcff675-kt4cm 270m 169Mi

vsphere-ipi-proxy-workers-rhel8: 4.15.0-0.nightly-2023-11-20-205649: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/472174

 Nov 22 19:12:04.071: INFO: Running 'oc --kubeconfig=/tmp/kubeconfig-2581276365 adm top pods -l app=olm-operator -n openshift-operator-lifecycle-manager --no-headers'
Nov 22 19:12:05.260: INFO: result: olm-operator-54f746c495-lv67t 122m 153Mi

vsphere-upi-platform-external: 4.15.0-0.nightly-2023-11-20-205649: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/472176

 Nov 22 19:29:57.542: INFO: result: olm-operator-54f746c495-pqnrn 101m 143Mi

azure-ipi-proxy-workers-rhcos-rhel8: 4.15.0-0.nightly-2023-11-20-205649: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/472907

Nov 24 02:29:18.646: INFO: result: olm-operator-54f746c495-d9jmz 127m 185Mi

azure-ipi-ingress-custom-controller-fips: 4.15.0-0.nightly-2023-11-24-164813: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/473328

Nov 25 05:30:26.959: INFO: result: olm-operator-7fdc7d954f-xtz2f 124m 123Mi

gcp-ipi-xpn-fips: 4.15.0-0.nightly-2023-11-25-011505: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#prow/launches/all/473645
 Nov 25 17:56:51.853: INFO: result: olm-operator-579d489b5-hgs7w 127m 199Mi

86_sno-packet-disk_encryption-etcd_encryption: 4.15.0-0.nightly-2023-11-20-205649: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#ocp/launches/all/471274

 Nov 22 01:05:09.867: INFO: result: olm-operator-54f746c495-jzr2q 102m 230Mi

116_UPI on Baremetal & OVN IPSEC_NoScoring: 4.15.0-0.nightly-2023-11-24-034436: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#ocp/launches/all/472794

 Nov 24 14:28:26.150: INFO: result: olm-operator-56b7889bf8-w824z 116m 234Mi

119_UPI_on_Baremetal-packet_OVN-single-stack_http-proxy_FIPS_on_IPSEC: 4.15.0-0.nightly-2023-11-24-074451: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#ocp/launches/all/473007

Nov 25 05:16:19.716: INFO: result: olm-operator-6857887497-44pgl 136m 248Mi

11_UPI on vSphere 8.0& FIPS ON & OVN IPSEC & Static Network & Bonding & HW19 & Secureboot: 4.15.0-0.nightly-2023-11-25-011505: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#ocp/launches/all/473234

Nov 25 15:43:17.914: INFO: result: olm-operator-579d489b5-pg8xr 2041m 257Mi

123_IPI on AWS & pre-released RHEL RHCOS: 4.15.0-0.nightly-2023-11-25-110147: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#ocp/launches/all/473465

 Nov 26 01:35:16.995: INFO: result: olm-operator-85c465b9f5-mjzq5 105m 206M

127_IPI_on_baremetal-rdu: 4.15.0-0.nightly-2023-11-25-110147: https://reportportal-openshift.apps.ocp-c1.prod.psi.redhat.com/ui/#ocp/launches/all/473469

Nov 26 04:00:11.496: INFO: result: olm-operator-85c465b9f5-g2ttq 180m 243Mi