Description of problem:
When planning ansible task on large batch of host multiple errors appears. We are talking about 300 and more hosts.

Version-Release number of selected component (if applicable):
satellite-6.9.7-1.el7sat.noarch

How reproducible:
I was not able to reproduce myself with 500+ hosts.

Steps to Reproduce:
1. have over 300 hosts
2. schedule ansible job, no matter of complexity

Actual results:
some hosts will stay forever with N/A status

Expected results:
Execution on large inventory possible

Additional info:
/var/log/messages
Mar 23 16:39:52 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:52 CET] "GET /dynflow/tasks/count?state=running HTTP/1.1" 200 29
Mar 23 16:39:52 satellite smart-proxy: - -> /dynflow/tasks/count?state=running
Mar 23 16:39:53 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:53 CET] "GET /version HTTP/1.1" 200 282
Mar 23 16:39:53 satellite smart-proxy: - -> /version
Mar 23 16:39:53 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:53 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:53 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:54 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:53 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:54 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:54 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:54 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:54 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:55 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:55 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:55 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:55 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:55 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:55 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:56 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:56 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:56 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:57 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:56 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:57 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:57 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:57 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:57 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:58 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:58 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:58 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:59 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:58 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:59 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: E, 2022-03-23T16:40:42.383222 #61768 ERROR – /parallel-executor-core/pool default/worker-0: "\xC3" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:19:in `encode'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:19:in `to_json'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:19:in `dump'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapter.rb:25:in `dump'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json.rb:139:in `dump'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:397:in `dump_data'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:300:in `block in serialize_columns'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:298:in `each'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:298:in `reduce'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:298:in `serialize_columns'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:288:in `prepare_record'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:312:in `save'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:173:in `save_action'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence.rb:46:in `save_action'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:33:in `open_action'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/parallel/worker.rb:14:in `on_message'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: [ concurrent-ruby ]
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: E, 2022-03-23T16:41:13.261767 #61768 ERROR – /client-dispatcher: Could not find an executor for Dynflow::Dispatcher::Enveloperequest_id: ac1bae45-fd9b-4391-9507-58ccfe0a91bc-319838, sender_id: ac1bae45-fd9b-4391-9507-58ccfe0a91bc, receiver_id: Dynflow::Dispatcher::UnknownWorld, message: Dynflow::Dispatcher::Event[execution_plan_id: 002c6216-8a0f-4767-a15e-09083206dde2, step_id: 2, event: #<ForemanTasksCore::Runner::Update:0x0000000004093f18>, time: ] (Dynflow::Error)
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:147:in `dispatch_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:118:in `block (2 levels) in publish_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:206:in `track_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:117:in `block in publish_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:248:in `with_ping_request_caching'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:116:in `publish_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: [ concurrent-ruby ]
...
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: E, 2022-03-23T16:49:52.395648 #62075 ERROR – /parallel-executor-core: no manager for Dynflow::Director::Eventrequest_id: a2a59f6c-9a5e-4853-8b3a-f2eeb495e389-231, execution_plan_id: b73e2c8a-165a-47af-a84d-0f2831832948, step_id: 2, event: Actions::Middleware::WatchDelegatedProxySubTasks::CheckOnProxyActions, result: #<Concurrent::Promises::ResolvableFuture:0x00007f791ac15af8 pending> (Dynflow::Error)
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:167:in `handle_event'
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/abstract/core.rb:35:in `handle_event'
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: [ concurrent-ruby ]
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: [ concurrent-ruby ]

/var/log/foreman/production.log
2022-03-23T16:41:02 [E|bac|] undefined method `wait' for nil:NilClass (NoMethodError)