(v0.4.0) OOMKilled Operator Pod after lots of resource creation in other namespaces

Issue is not happening on 0.4.2.

Seems to be fixed by Convert OADP operator to a namespace scoped operator by shubham-pampattiwar · Pull Request #427 · openshift/oadp-operator (github.com)

which appears in v0.4.2 which no longer have this issue from their testing.

The issue here is growing number of service accounts and other resources in non-oadp namespaces caused increased memory usage.

We want to know why 0.4.0 operator goes into crash loop after creating lot of resources in a different namespace. Should be some connection with it?
 
( 5 pods, 5 Deployment.apps, 5 pvcs, 100 buildconfig, 100 imagestream, 10 route, 10 deploymentconfig.apps, 10 replicaset.apps, 3000 Services )
We created it in across 10 namespaces.
 
to recreate # OpenShift cluster with running oadp 0.4.0

1. Create lots of resource in other namespaces
2. Oadp pod crash

Step 2 may be switched with step 1 to trigger the same issue.

Resolution attempted

1. Resource bumping in clusterserviceversion with following
   1. resources:
              limits:
                cpu: "2"
                memory: 1Gi
              requests:
                cpu: "1"
                memory: 256Mi

Script to recreate resource is attached with following instruction

1. PosgreSql

1. 1. /home/core/performance/postgres/*
   2. oc apply -f /home/core/performance/postgres -n namespace 1 …10
   3. creates 5 deployments, pvc, pods under a single namespace

1. Service Accounts

1. 1. /home/core/performance/generteServiceAccounts.sh
   2. Creates 3000 service accounts per namespace

1. Ruby

1. 1. /home/core/performance/ruby/genertaeMetadata1.sh
   2. Creates 50 "ImageStream", “service”, buildconfig

You may find path "/home/core/performance/ruby" in the "genertaeMetadata1" sh file. Please feel free to change it accordingly when running on your own cluster.

Reported from kubernetes slack from Yunqian Chen