Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-179

(v0.4.0) OOMKilled Operator Pod after lots of resource creation in other namespaces

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • False
    • False
    • Not Required
    • 0
    • 0
    • 0
    • Untriaged
    • None

      Issue is not happening on 0.4.2.

      Seems to be fixed by Convert OADP operator to a namespace scoped operator by shubham-pampattiwar · Pull Request #427 · openshift/oadp-operator (github.com)

      which appears in v0.4.2 which no longer have this issue from their testing.

       

      The issue here is growing number of service accounts and other resources in non-oadp namespaces caused increased memory usage.

       

      We want to know why 0.4.0 operator goes into crash loop after creating lot of resources in a different namespace. Should be some connection with it?
       
      ( 5 pods, 5 Deployment.apps, 5 pvcs, 100 buildconfig, 100 imagestream, 10 route, 10 deploymentconfig.apps, 10 replicaset.apps, 3000 Services )
      We created it in across 10 namespaces.
       
      to recreate # OpenShift cluster with running oadp 0.4.0

      1. Create lots of resource in other namespaces
      2. Oadp pod crash

      Step 2 may be switched with step 1 to trigger the same issue.

       

      Resolution attempted

      1. Resource bumping in clusterserviceversion with following
        1. resources:
                  limits:
                    cpu: "2"
                    memory: 1Gi
                  requests:
                    cpu: "1"
                    memory: 256Mi

      Script to recreate resource is attached with following instruction

      1. PosgreSql
        1. /home/core/performance/postgres/*
        2. oc apply -f /home/core/performance/postgres -n namespace 1 …10
        3. creates 5 deployments, pvc, pods under a single namespace
      1. Service Accounts
        1. /home/core/performance/generteServiceAccounts.sh
        2. Creates 3000 service accounts per namespace
      1. Ruby
        1. /home/core/performance/ruby/genertaeMetadata1.sh
        2. Creates 50 "ImageStream", “service”, buildconfig

      You may find path "/home/core/performance/ruby" in the "genertaeMetadata1" sh file. Please feel free to change it accordingly when running on your own cluster.

      Reported from kubernetes slack from Yunqian Chen

        1. image-2022-01-07-22-06-15-553.png
          112 kB
          Tiger Kaovilai

              Unassigned Unassigned
              tkaovila@redhat.com Tiger Kaovilai
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: