Uploaded image for project: 'Satellite'
  1. Satellite
  2. SAT-18782

Publish/promote of CV with puppet environments consume GBs of memory for scaled EnvironmentClass counts

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 6.8.0
    • Puppet
    • 0
    • False
    • Moderate
    • None
    • None
    • None
    • None

      Description of problem:
      User story:
      publishing or promoting a CV with puppet modules does consume up to 30GB of memory in sidekiq process.

      The root cause is in this call flow:

      (the above statement "method 'changes' itself causes the high memory usage" was confirmed by running a rake script at a customer where a PuppetClassImporter(..).changes was the method that bumped memory usage by >4GB, alone )

      Further internal testing revealed that the scaling factor affecting the high memory usage is EnvironmentClass.count . For 120k EnvironmentClass items, rake script executing the "changes" method consumed 900MB memory (400MB rake itself, 500MB the method). For 260k EnvironmentClass items, I got 1552MB usage (400MB rake, 1152MB the method). Lineraly prolonging this to 1M EnvironmentClass items (that the customer has), we get 4GB memory per one execution of the method.

      Due to the way how ruby does (not) return freed memory, repeated calling of the method by different PuppetClassImporter instances does consume further GBs of memory, every time. Until the usage is (probably) stabilised at 30GB per one sidekiq process.

      Please optimise the foreman method (being used in 6.10 as well, hence we do support it there also) to be more memory efficient for scaled EnvironmentClass instances.

      Version-Release number of selected component (if applicable):
      Sat6.8 (same affects 6.9 and will 6.10)

      How reproducible:
      100%

      Steps to Reproduce:
      1. Import many puppet modules (I will provide a script for it)
      2. Have any repo enabled
      3. Create many CVs with many CV versions having many puppet modules associated each.
      4. Monitor memory usage of sidekiq processes during the CV publish/promote
      5. Monitor # of puppet env. classes:
      su - postgres -c "psql foreman -c \"select count from environment_classes;\""

      Actual results:
      memory usage grows linearly with the EnvironmentClass.count / count from environment_classes. For 1M classes, memory bumps by 4GB in a CV publish/promote.

      Expected results:
      something more sustainable

      Additional info:
      Some stats of the impact of # of env.classes to RSS usage of the 'changes' method (RSS usage of rake running the method):

      RSS EnvClass.count
      878920 119903
      1059592 129122
      1034964 138341
      1029156 147560
      897756 156779
      1010816 165998
      1233568 175217
      1187076 184436
      1244400 193655
      1244380 202874
      1300772 212093
      1269580 221312
      1286400 230531
      1316340 239750
      1472676 248969
      1551976 258188
      1569808 267407
      1578724 276626
      .. (test continues) ..

              jira-bugzilla-migration RH Bugzilla Integration
              jira-bugzilla-migration RH Bugzilla Integration
              Vladimír Sedmík Vladimír Sedmík
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: