Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-11742

High memory consumption when dealing with big CA cert file

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • None
    • openstack-tempest
    • None
    • 3
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • ?
    • openstack-tempest-41.0.0-18.0.20241119124712.a25e0df.el9osttrunk
    • ?
    • ?
    • None
    • Moderate

      1) Describe the error and the expected behavior.
      Tempest seems to be consuming too much memory when

      • CA cert file is too big (e.g., 290 KiB)
      • urllib3 1.26.x is used as a dependency [1][2]

      Take a look at the attached graphs in the first comment. The issue is not present when urllib3 2.2.3 is used.

      2) What OS did you run into the issue on?

      • Ubuntu 22.04
      • CentOS Stream 9

      3) Specify the steps to reproduce the bug:
      1. Install tempest
      2. Make sure that tempest uses urllib3 1.26.x
      3. Set identity.ca_certificates_file so that it points to a big ca_cert file
      4. Run tempest (e.g., tempest run --smoke --concurrency=4)

      It looks like the issue is related to HTTPConnections in PoolManager keeping the ca cert files in memory long after the test class finished its execution [3].

      WIP ideas of how to fix this:

      • Run clear() on PoolManagers in the tearDown() in test.py [4]
      • Run clear() when new HTTPConnection is created [5]
      • Wait for bump up to a newer version of urllib3.

      [1] https://opendev.org/openstack/tempest/src/commit/a350e320b162abc42c4b5236b258e1e8742048d0/requirements.txt#L22
      [2] https://opendev.org/openstack/requirements/src/commit/1daffb38e8d91ac33de84c850ba406bcec7a8aaa/upper-constraints.txt#L179
      [3] https://opendev.org/openstack/tempest/src/branch/master/tempest/lib/common/http.py
      [4] https://review.opendev.org/c/openstack/tempest/+/934979
      [5] https://review.opendev.org/c/openstack/tempest/+/934980

        1. urllib-2.2.3.png
          74 kB
          Lukáš Piwowarski
        2. urllib3-1.26.20.png
          79 kB
          Lukáš Piwowarski

              lpiwowar Lukáš Piwowarski
              lpiwowar Lukáš Piwowarski
              rhos-tempest
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: