Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-11501

BZ#2322922 [BUG] Collectd keep using memory if it's disconnected

XMLWordPrintable

    • 2
    • False
    • Hide

      None

      Show
      None
    • False
    • Committed
    • Not Set
    • Not Set
    • None
    • CloudOps 2024 Sprint 24

      Description of problem:

      If collectd has been configured and is active it will continue to send data to the corresponding endpoint and thats what causes the issue as there's no endpoint listening and the queue keeps growing. And all of the memory will be used up

      Similar BZ are and it seems that it still there in RHOSP 17.1.2

      https://bugzilla.redhat.com/show_bug.cgi?id=1817124
      https://bugzilla.redhat.com/show_bug.cgi?id=1798617
      https://bugzilla.redhat.com/show_bug.cgi?id=1790928
      https://access.redhat.com/solutions/4855731

      It can be reproduced in Lab
      ~~~

      1. podman stats collectd
        ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS CPU TIME AVG CPU %
        f61051856726 collectd 1.98% 53.41MB / 33.37GB 0.16% 0B / 0B 327.7kB / 81.92kB 39 13h12m51.737656s 1.98%
      1. free -g
        total used free shared buff/cache available
        Mem: 31 8 13 0 10 23

      => 16.2

      1. podman stats collectd
        ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS
        ad91dff0cdea collectd – 70.93MB / 536.9MB 13.21% – / – 1.839MB / 519MB 38
      1. free -g
        total used free shared buff/cache available
        Mem: 31 12 3 0 15 18
        Swap: 0 0 0

      Need to understand why in 17.x environment the limit of 500 MB is not configured or applicable
      ~~~
      On the other hand, setting the memory limit does not work.

      ~~~

      1. get the ID of the collect container
        id=$(podman ps | grep collectd | awk ' {print $1}

        ')

      1. setting need to change swap to update mem
        podman update --memory-swap 300m $id
      1. setting max memory to 100 mb
        podman update m 100m $id <-- adjust to the correct value needed if more than 100m
      1. validating the new setting
        podman inspect $id | jq '.[].HostConfig.Memory'
        ~~~

      ~~~
      parameter_defaults:
      ExtraConfig:
      collectd::write_queue_limit_high: 100
      collectd::write_queue_limit_low: 100
      ~~~

      Version-Release number of selected component (if applicable):

      How reproducible:

      Steps to Reproduce:
      1.
      2.
      3.

      Actual results:

      Expected results:

      Additional info:

              mmagr@redhat.com Martin Magr
              jira-bugzilla-migration RH Bugzilla Integration
              Leonid Natapov Leonid Natapov
              rhos-dfg-cloudops
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: