While running HCPs on Azure we have experienced continuous spike every few minutes by the catalog operator pods, it has spiked up to 200%(max) by a single pod. When running multiple HCPs, they were spiking all at once randomly that caused CPU starvation on the worker, that affected kubelet and containerd(AKS) daemons.

    4.17.0-rc.0

    Always

    1. Create HCPs on Azure (it may happen on AWS)
    2. Watch catalog operator pod CPU usage for a few hours
    3. PROMQL - sum(irate(container_cpu_usage_seconds_total{container!="POD",name!="",namespace=~"clusters.*",instance=~".*user.*",pod=~".*catalog.*"}[$interval])) by (pod) * 100     

    CPU spike randomly every few minutes, per document 12h is the default interval but its happening frequently
https://olm.operatorframework.io/docs/advanced-tasks/configuring-olm/#changing-the-package-server-sync-interval 

    Every 12h is the expected interval based on the document and reduced CPU usage
https://olm.operatorframework.io/docs/advanced-tasks/configuring-olm/#changing-the-package-server-sync-interval

    This was supposed to be fixed by this - https://issues.redhat.com/browse/OCPBUGS-17950 but still happening in latest release