Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-19056

tripleo_container_manage ansible module doesn't parse "mem_limit", which causes huge memory usage of collectd

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Undefined Undefined
    • rhos-17.1.5
    • rhos-17.1.z
    • tripleo-ansible
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • None
    • Important

      To Reproduce Steps to reproduce the behavior:

      1. Deploy RHOSP 17.1 overcloud with collectd
      2. Login to a overcloud node
      3. "mem_limit: 512m" is written into collectd.json.
        # cat /var/lib/tripleo-config/container-startup-config/step_3/collectd.json
        
            :   
          "mem_limit": "512m",
            : 
      1. This setting is supposed to be applied into the container, but actually it's not.
        Memory limit is not set in the collectd container.
        # podman inspect collectd | jq .[0].HostConfig.Memory
        0 

      Expected behavior

      Collectd container should have the memory usage limitation:

      # podman inspect collectd | jq .[0].HostConfig.Memory
      536870912

       

      Bug impact

      • This allow collectd to consume extremely huge memory, which results in OOM or unstable behavior on the node

      Known workaround

      • I was wondering if running "podman update --memory 512m collecd" might work.
        But unfortunately this doesn't work due to [1].
        Even if it worked, this setting won't be persistent. If the container is restarted, the setting is reverted [2]
        Therefore, if OOM occurs once, I think this workaround will be reverted.
        I don't think this can be a good workaround.
        [1] https://issues.redhat.com/browse/RHEL-19137
        [2] https://access.redhat.com/solutions/7058735
      • I don't come up with any other workarounds...

      Additional context

      In RHOSP 16, we introduced the following changes to set the limit of collectd memory usage:

      I think this worked well in RHOSP 16.2.
      But it seems that it is not working well in RHOPS 17.1.

              Unassigned Unassigned
              rhn-support-yatanaka Yamato Tanaka
              rhos-dfg-df
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: