Uploaded image for project: 'OpenShift GitOps'
  1. OpenShift GitOps
  2. GITOPS-5673

Investigate ways to reduce Operator memory consumption

XMLWordPrintable

    • Icon: Spike Spike
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • Operator
    • False
    • None
    • False
    • 5

      We recently discovered(GITOPS-5665) that the operator consumes significantly more memory on clusters with a large number of secrets/configmaps. For example, on a test cluster with 2000 secrets and 2000 configmaps spread across 100 namespaces, the operator manager pod consumed over 2GB at peak with just one ArgoCD CR instance.

      The operator watches many resources(code). Use this spike to perform a feasibility check on optimizing these watches. Additionally, explore other potential optimizations in the reconciliation or other parts of the code, to reduce memory consumption.

       

      Here are some resources to get started:

       

      Steps to reproduce the problem:

      1. Create 100 namespaces

      for i in {1..100}; do oc create ns argo-test$i; done
      

      2. Create a large key, value environment file to be used for creating config map and secret.

      rm -f /tmp/environment.txt
      touch /tmp/environment.txt
      for i in {1..10000}; do echo "key${i}=value${i}" >> /tmp/environment.txt; done
      

      3. Create 20 config maps per namespace (total 20 * 100 = 2000) using the environment file created in step 2.

      for i in {1..100}; do for j in {1..20}; do oc create cm argo-cm$j --from-file /tmp/environment.txt -n argo-test$i; done; done;
      

      4. Create 20 secrets per namespace (total 20 * 100 = 2000) using the environment file created in step 2

      for i in {1..100}; do for j in {1..20}; do oc create secret generic argo-secret$j --from-file /tmp/environment.txt -n argo-test$i; done; done;
      

      5. Try installing OpenShift GitOps 1.14.0 now and you should be able to see the POD OOM killed error

      openshift-gitops-operator-controller-manager-b6d7788d9-pf5z6    1/2     Running             5 (85s ago)   5m
      openshift-gitops-operator-controller-manager-b6d7788d9-pf5z6    2/2     Running             5 (96s ago)   5m11s
      openshift-gitops-operator-controller-manager-b6d7788d9-pf5z6    1/2     OOMKilled           5 (109s ago)   5m24s
      openshift-gitops-operator-controller-manager-b6d7788d9-pf5z6    1/2     CrashLoopBackOff    5 (8s ago)     5m31s
      openshift-gitops-operator-controller-manager-b6d7788d9-pf5z6    1/2     Running             6 (2m48s ago)   8m11s
      openshift-gitops-operator-controller-manager-b6d7788d9-pf5z6    2/2     Running             6 (2m58s ago)   8m21s
      openshift-gitops-operator-controller-manager-b6d7788d9-pf5z6    1/2     OOMKilled           6 (3m12s ago)   8m35s
      openshift-gitops-operator-controller-manager-b6d7788d9-pf5z6    1/2     CrashLoopBackOff    6 (7s ago)      8m41s

              Unassigned Unassigned
              rh-ee-sghadi Siddhesh Ghadi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: