Uploaded image for project: 'Satellite'
  1. Satellite
  2. SAT-4984

[RFE] max_memory_per_executor support

XMLWordPrintable

    • None

      Given once Ruby allocates some memory, it doesn't give it back, bigger
      set of larger actions can lead to quite big memory consumption that
      persists and can accumulate over time. With this, it's hard to keep
      memory consumption fully under control, especially in an environment
      with other systems (passenger, pulp, candlepin, qpid). Since the
      executors can terminate nicely without affecting the tasks itselves,
      it should be pretty easy to extend it to watch the memory consumption.

      The idea:

      1. config options:
      max_memory_per_executor - the threshold for the memory size per executor
      min_executors_count - minimal count executors (default 1)
      minimal_executor_age - the period it will check whether the memory consumption didn't grow (default 1h)

      2. the executor will periodically check it's memory usage,
      (http://stackoverflow.com/a/24423978/457560 seems to be a sane
      approach for us)

      3. if memory usage exceeds `max_memory_per_executor`, the executor is
      older than `minimal_executor_age` (to prevent situation, where the
      memory would grow too fast over the max_memory_per_executor, which
      would mean we wouldn't do anything than restarting the executors
      without getting anything done and the amount of current executors
      would not go under `min_executors_count`, politely terminate executor

      4. the polite termination should be able to hand over all the tasks to
      the other executors and once everything is finalized on the executor, it would just exit

      5. the daemon monitor would notice the executor getting closed and running a new executor

      It would be configurable, turned off by default (for development) but we would configure
      this in production, where we can rely on the monitor being present.

              jira-bugzilla-migration RH Bugzilla Integration
              jira-bugzilla-migration RH Bugzilla Integration
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: