What is the nature and description of the request?

• To be able to limit the number of Ansible runs each Capsule server is executing at one time so that an individual host can complete it's run faster.

Why does the customer need this?

• Being able to adjust the simultaneous runs allows the customer to maximize successful runs and avoid disruptions that would be caused by the job scheduler e.g. rebooting nodes.
• Enables the customer to adjust the frequency of Ansible runs and number of nodes to assign per Capsule server.

Is there already an existing RFE upstream or in Red Hat Bugzilla?

• None found

Background:

• Customer is migrating hosts from Satellite 6.9 with Puppet to Satellite 6.14 with Ansible.
• Since Ansible uses a push model and takes longer to run, they elected to create a task once per day to run all the playbooks assigned to all managed hosts.
• This was working fine during development with only 20 or so attached systems, but now have over 400, and will be expanding up to about 4500.
• Sometimes, the task to run Ansible fails on most of the nodes. Customer re-runs the failed nodes, and each of the runs appears to be very slow.
• All of the systems they migrated are in one datacentre with 3 Capsules assigned to distribute the load.
• When looking at individual Ansible runs on a host, it seems significantly slower than when doing single runs prior to the migration starting.
• Regardless of if foreman_ansible_proxy_batch_size is set to 50 or 100, when we don't limit the concurrency in the job, a task that should run on all managed hosts would kick off on every node at the same time, and run times would go from about 10 or 15 minutes to 2 hours+.
• By limiting the concurrency in the job, sites with fewer nodes still take much longer to run all the Ansible runs because they will be interspersed between the large number of nodes in the Houston (datacentre).