Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-15760

keepalived pod running on worker nodes consuming more memory than usual.

XMLWordPrintable

    • Moderate
    • No
    • False
    • Hide

      None

      Show
      None
    • 4.9

      Description of problem:

      Keepalived pod in project openshift-kni-infra running on worker nodes consumes more memory than usual, there is linear memory growth seen in keepalived.

      $oc adm top pods -n openshift-kni-infra
      NAME                                 CPU(cores)   MEMORY(bytes)
      coredns-master1.offline.att.com      2m           112Mi
      coredns-master2.offline.att.com      1m           121Mi
      coredns-master3.offline.att.com      0m           112Mi
      coredns-worker1.offline.att.com      2m           108Mi
      coredns-worker2.offline.att.com      0m           110Mi
      coredns-worker3.offline.att.com      2m           100Mi
      coredns-worker4.offline.att.com      2m           104Mi
      coredns-worker5.offline.att.com      0m           104Mi
      coredns-worker6.offline.att.com      2m           92Mi
      haproxy-master1.offline.att.com      12m          236Mi
      haproxy-master2.offline.att.com      6m           241Mi
      haproxy-master3.offline.att.com      6m           237Mi
      keepalived-master1.offline.att.com   47m          526Mi
      keepalived-master2.offline.att.com   85m          601Mi
      keepalived-master3.offline.att.com   83m          541Mi
      keepalived-worker1.offline.att.com   55m          2626Mi
      keepalived-worker2.offline.att.com   73m          1334Mi
      keepalived-worker3.offline.att.com   54m          4102Mi
      keepalived-worker4.offline.att.com   60m          2092Mi
      keepalived-worker5.offline.att.com   56m          4171Mi
      keepalived-worker6.offline.att.com   49m          2100Mi

      SOS report of worker-1 node says the overall memory consumption of node is good. Also, can't see any memory leak by keepalived pods.

      MEMORY
        Stats graphed as percent of MemTotal:
          MemUsed    ▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊...............  69.8%
          Buffers    ..................................................   0.1%
          Cached     ▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊▊.........................  50.3%
          HugePages  ..................................................   0.0%
          Dirty      ..................................................   0.0%
        RAM:
          187.5 GiB total ram
          130.9 GiB (70%) used
      
      Top MEM-using processes: 
          USER      PID      %CPU  %MEM  VSZ-MiB  RSS-MiB  TTY    STAT   START  TIME       COMMAND  
          nfsnobo+  44553    39.0  2.1   26205    4174     ?      -      2022   105763:56  /bin/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/pr
          core      261495   4.3   0.8   19006    1563     ?      -      Jun07  318:53     /opt/nokia/ric/traffic-steering/traffic-steering -f /opt/ric/config/config-file.json 
          1000730+  3040267  5.3   0.6   14024    1255     ?      -      May15  2126:46    java -Xms1024M -Xmx2048M 
          core      2371090  18.2  0.4   15950    864      ?      -      Jun10  618:53     ./admin-cli/node/linux/bin/node --expose_gc --no-warnings 
          core      107974   15.8  0.4   15898    812      ?      -      05:11  48:56      ./admin-cli/node/linux/bin/node --expose_gc --no-warnings 
          core      1898427  14.4  0.4   14870    780      ?      -      Jun09  689:04     ./admin-cli/node/linux/bin/node --expose_gc --no-warnings 
          core      3037542  0.9   0.3   3800     732      ?      -      May15

       

            bnemec@redhat.com Benjamin Nemec
            rhn-support-ankhande Anjali Khandelwal
            Zhanqi Zhao Zhanqi Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: