Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-8379

operator controller-manager pods get OOMkilled

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Blocker Blocker
    • rhos-18.0.0
    • rhos-18.0.0
    • openstack-operator
    • None
    • Critical

      The memory usage compared to OCP 4.15 is higher and as a result the current set resource limits result in the operator pods to get oom killed:

       

      ironic-operator-controller-manager-64c8758dbb-skzxw               1/2     CrashLoopBackOff   27 (2m29s ago)   131m                                                                                                  
      keystone-operator-controller-manager-8454f968cf-r7sdx             1/2     CrashLoopBackOff   27 (4m23s ago)   131m                                                                                                  
      manila-operator-controller-manager-68c5b99f85-mh9zs               1/2     CrashLoopBackOff   16 (4m28s ago)   132m                                                                                                  
      mariadb-operator-controller-manager-6d6675f7b5-bx7gv              1/2     CrashLoopBackOff   27 (3m41s ago)   131m                                                                                                  
      neutron-operator-controller-manager-598b787665-mkxrj              1/2     CrashLoopBackOff   27 (3m9s ago)    132m                                                                                                  
      nova-operator-controller-manager-7c9f44d57f-h4kjs                 1/2     CrashLoopBackOff   27 (2m48s ago)   132m   

       

       

      4.15.18:

      [zuul@controller-0 ~]$ kubectl top pod -n openstack-operators                                                                                                                                                       
      NAME                                                              CPU(cores)   MEMORY(bytes)
      barbican-operator-controller-manager-fc799cc85-8l679              1m           142Mi
      cinder-operator-controller-manager-6bb54778c7-9p9d2               1m           159Mi
      designate-operator-controller-manager-b6fdc6bb5-lpgqc             1m           166Mi
      glance-operator-controller-manager-87c8bd89c-s7vl2                1m           141Mi
      heat-operator-controller-manager-65b77f5c66-xcnrm                 2m           149Mi
      horizon-operator-controller-manager-6c79dd8467-8bgnd              2m           157Mi
      infra-operator-controller-manager-5f7876b87b-4z976                2m           204Mi
      ironic-operator-controller-manager-5ff794f488-hsb8w               2m           156Mi
      keystone-operator-controller-manager-776fd59d9d-gqs27             1m           175Mi
      manila-operator-controller-manager-546b784cd6-tsql8               1m           154Mi
      mariadb-operator-controller-manager-7b4fd55df6-jl8x6              1m           170Mi
      neutron-operator-controller-manager-5776bcdb99-dgdhk              1m           144Mi
      nova-operator-controller-manager-5b988fbd5b-j8tx2                 2m           192Mi
      octavia-operator-controller-manager-6b78456f88-vbxs4              2m           161Mi
      openstack-ansibleee-operator-controller-manager-56b59dfdffdbwtk   1m           117Mi
      openstack-baremetal-operator-controller-manager-57dd66974bmj2z8   0m           123Mi
      openstack-operator-controller-manager-6ffc95d8dc-sbt44            2m           221Mi
      openstack-operator-index-9nn8g                                    2m           96Mi
      ovn-operator-controller-manager-6f5fd549b8-9wmpt                  2m           207Mi
      placement-operator-controller-manager-5b68cf64bf-896p6            3m           172Mi
      rabbitmq-cluster-operator-67987d7bc-tpr52                         1m           132Mi
      swift-operator-controller-manager-dcf55b4f6-8bj2k                 2m           149Mi
      telemetry-operator-controller-manager-844bf8f9d4-brcpf            1m           160Mi

       

      4.16.1

      [zuul@controller-0 ~]$ kubectl top pod -n openstack-operators
      NAME                                                              CPU(cores)   MEMORY(bytes)
      barbican-operator-controller-manager-58cdfd4464-vcc6r             0m           23Mi
      cinder-operator-controller-manager-79b98dd44f-62ksf               1m           12Mi
      designate-operator-controller-manager-6d754cf765-cfnj4            2m           465Mi
      glance-operator-controller-manager-85f5997d7b-52gxs               0m           20Mi
      heat-operator-controller-manager-69b696d8b8-45hv4                 2m           480Mi
      horizon-operator-controller-manager-84645b9b56-2mrpf              0m           14Mi
      infra-operator-controller-manager-7685bb8894-l5hf9                0m           20Mi
      ironic-operator-controller-manager-646764cb5b-b647f               2m           478Mi
      keystone-operator-controller-manager-665678b57c-8z4pc             0m           14Mi
      manila-operator-controller-manager-c874b9f47-24x7p                2m           466Mi
      mariadb-operator-controller-manager-68f8b77b47-f5lnf              1m           20Mi
      neutron-operator-controller-manager-c4567dbbc-z6pxw               1m           21Mi
      nova-operator-controller-manager-798ddc68f-84kfd                  0m           21Mi
      octavia-operator-controller-manager-789b4c8f7b-5jh8b              2m           459Mi
      openstack-ansibleee-operator-controller-manager-b57d75c79-svzpz   2m           37Mi
      openstack-baremetal-operator-controller-manager-6fb7cf6c46x5h68   2m           91Mi
      openstack-operator-controller-manager-66c7c59d9d-rd5vr            0m           12Mi
      openstack-operator-index-r252g                                    3m           70Mi
      ovn-operator-controller-manager-c97c56d5d-vds5n                   1m           10Mi
      placement-operator-controller-manager-59b4bc865f-7zfq2            0m           16Mi
      rabbitmq-cluster-operator-9d49f8d9b-9lxzj                         10m          399Mi
      swift-operator-controller-manager-6764c568ff-r4mzl                2m           468Mi
      telemetry-operator-controller-manager-7c4fd577b4-9nnvq            2m           482Mi

      for the ones which has low memory usage, in the env where this was taken from, the services were not yet deployed.

              rhn-support-mschuppe Martin Schuppert
              rhn-support-mschuppe Martin Schuppert
              rhos-dfg-ospk8s
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: