Description of problem:

During perfscale testing of max-namespaces kube burner test on 120 nodes on Alicloud, found that insights operator degraded even though load on the cluster is not too heavy and number of pods is around 200 pods per worker node. 

Cluster was still running and was able to take additional load without any other issues.

Version-Release number of selected component (if applicable):

Issue can be observed on both 4.11.0 release and 4.10.28 z release.

Server Version: 4.10.28
Operator version: 4.10.28
Kubernetes Version: v1.23.5+012e945

and

Server Version: 4.11.0
Kubernetes Version: v1.24.0+9546431

Initially tested on 4.10.28 to collect new baselines. Then tested again on 4.11 to confirm issue still exists.

How reproducible: Run kube burner max-namespaces performance test (https://github.com/cloud-bulldozer/e2e-benchmarking/tree/master/workloads/kube-burner)

Steps to Reproduce:
1. Create a cluster using Alibaba Cloud using following instance and network type.

vm_type_masters: 'ecs.g6.4xlarge'
vm_type_workers: 'ecs.g6.2xlarge'
networkType: "OVNKubernetes"

installer_payload_image: 'registry.ci.openshift.org/ocp/release:4.10.28'

2. Scale cluster up to 120 Worker nodes and install ditty bopper

3. Execute kube burner max-namespaces test using NAMESPACE_COUNT of 3500. 

export NAMESPACE_COUNT=3500 

Actual results: Insights operator is degraded.

Expected results: Did not expect insights operator to be degraded.

Additional info: Tests could be successfully executed up to NAMESPACE_COUNT=4000 except for insights operator degradation.