https://github.com/ansible-collections/vmware.vmware_rest/issues/364
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete all sections as described, this form is processed automatically -->

1. 1. 1. 1. 1. SUMMARY
               <!--- Explain the problem briefly below -->
               Might be a repeat of #336 where playbooks with multiple tasks that use the vmware.vmware_rest modules against multiple vCenters at the same time may error. It is not completely repeatable but does happen often. Sometimes a retry fixes it. Unable to reproduce with a single vCenter. Running cloud.common 2.1.2 and vmware.vmware_rest 2.2.0

1. 1. 1. 1. 1. ISSUE TYPE

• Bug Report

1. 1. 1. 1. 1. COMPONENT NAME
               <!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
               Observed in
               appliance_access_ssh
               appliance_networking_proxy
               and possibly others
            2. ANSIBLE VERSION
               <!--- Paste verbatim output from "ansible --version" between quotes -->
               ```paste below
               ansible [core 2.13.6]
               config file = /runner/ansible.cfg
               configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
               ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
               ansible collection location = /runner/collections
               executable location = /usr/local/bin/ansible
               python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]
               jinja version = 3.1.2
               libyaml = True
               ```

1. 1. 1. 1. 1. COLLECTION VERSION
               <!--- Paste verbatim output from "ansible-galaxy collection list <namespace>.<collection>" between the quotes
               for example: ansible-galaxy collection list community.general
               -->
               ```paste below
               Collection Version
               • • • • • • • • • • • • • -------
                                         vmware.vmware_rest 2.2.0
                                         cloud.common 2.1.2
                                         ```

1. 1. 1. 1. 1. CONFIGURATION
               <!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
               ```paste below
               COLLECTIONS_PATHS(/runner/ansible.cfg) = ['/runner/collections']
               DEFAULT_FILTER_PLUGIN_PATH(/runner/ansible.cfg) = ['/runner/custom_filters']
               DEFAULT_ROLES_PATH(/runner/ansible.cfg) = ['/runner/roles']
               DEFAULT_STRATEGY_PLUGIN_PATH(/runner/ansible.cfg) = ['/runner/custom_plugins/mitogen-0.3.0-rc.0/ansible_mitogen/plugins/strategy']
               DEFAULT_TIMEOUT(/runner/ansible.cfg) = 40
               HOST_KEY_CHECKING(/runner/ansible.cfg) = False
               PARAMIKO_LOOK_FOR_KEYS(/runner/ansible.cfg) = False
               ```

1. 1. 1. 1. 1. OS / ENVIRONMENT
               <!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->
               Ansible Execution Environment from ansible-builder
               vCenter 7.0 U3

1. 1. 1. 1. 1. STEPS TO REPRODUCE
               <!--- Describe exactly how to reproduce the problem, using a minimal test-case -->
               Playbook with several tasks in a row all using the same collection. After
               <!--- Paste example playbooks or commands between quotes below -->
               ```yaml
               tasks:

• name: DNS Servers
  vmware.vmware_rest.appliance_networking_dns_servers:
  vcenter_hostname: "{{ inventory_hostname }}"
  vcenter_username: "{{ vcenter_user }}"
  vcenter_password: "{{ vcenter_password }}"
  vcenter_validate_certs: "{{ vmware_validate_certs }}"
  servers: "{{ dns_servers }}"
  mode: is_static
  delegate_to: localhost
  retries: 12
  delay: 10
  tags:
• dns

• name: Content Library
  community.vmware.vmware_content_library_manager:
  hostname: "{{ inventory_hostname }}"
  username: "{{ vcenter_user }}"
  password: "{{ vcenter_password }}"
  validate_certs: "{{ vmware_validate_certs }}"
  library_name: "{{ item.library_name }}"
  library_description: "{{ item.library_description | default(omit) }}"
  library_type: "{{ item.library_type | default(omit) }}"
  datastore_name: "{{ item.datastore_name | default(omit) }}"
  subscription_url: "{{ item.subscription_url | default(omit) }}"
  ssl_thumbprint: "{{ item.ssl_thumbprint | default(omit) }}"
  update_on_demand: "{{ item.update_on_demand | default(omit) }}"
  state: "{{ item.state | default(omit) }}"
  delegate_to: localhost
  loop: "{{ content_libraries }}"
  loop_control:
  label: "{{ item.library_name }}"
  when: content_libraries is defined
  retries: 12
  delay: 10
  tags:
• content_library

• name: Timezone
  vmware.vmware_rest.appliance_system_time_timezone:
  vcenter_hostname: "{{ inventory_hostname }}"
  vcenter_username: "{{ vcenter_user }}"
  vcenter_password: "{{ vcenter_password }}"
  vcenter_validate_certs: "{{ vmware_validate_certs }}"
  name: America/Chicago
  delegate_to: localhost
  retries: 12
  delay: 10
  tags:
• timezone

• name: NTP configuration
  vmware.vmware_rest.appliance_ntp:
  vcenter_hostname: "{{ inventory_hostname }}"
  vcenter_username: "{{ vcenter_user }}"
  vcenter_password: "{{ vcenter_password }}"
  vcenter_validate_certs: "{{ vmware_validate_certs }}"
  servers: "{{ ntp_servers }}"
  delegate_to: localhost
  retries: 12
  delay: 10
  tags:
• ntp
  notify: Restart the ntpd service

• name: Enable NTP time sync
  vmware.vmware_rest.appliance_timesync:
  vcenter_hostname: "{{ inventory_hostname }}"
  vcenter_username: "{{ vcenter_user }}"
  vcenter_password: "{{ vcenter_password }}"
  vcenter_validate_certs: "{{ vmware_validate_certs }}"
  mode: NTP
  delegate_to: localhost
  retries: 12
  delay: 10
  tags:
• ntp

• name: Network Proxy - HTTP
  vmware.vmware_rest.appliance_networking_proxy:
  vcenter_hostname: "{{ inventory_hostname }}"
  vcenter_username: "{{ vcenter_user }}"
  vcenter_password: "{{ vcenter_password }}"
  vcenter_validate_certs: "{{ vmware_validate_certs }}"
  enabled: true
  server: "{{ http_proxy }}"
  port: "{{ http_proxy_port }}"
  protocol: http
  delegate_to: localhost
  when: http_proxy is defined
  retries: 12
  delay: 10
  tags:
• proxy
• http

• name: Network Proxy - HTTPS
  vmware.vmware_rest.appliance_networking_proxy:
  vcenter_hostname: "{{ inventory_hostname }}"
  vcenter_username: "{{ vcenter_user }}"
  vcenter_password: "{{ vcenter_password }}"
  vcenter_validate_certs: "{{ vmware_validate_certs }}"
  enabled: true
  server: "{{ https_proxy }}"
  port: "{{ https_proxy_port }}"
  protocol: https
  delegate_to: localhost
  when: https_proxy is defined
  retries: 12
  delay: 10
  tags:
• proxy
• https
  ```

<!--- HINT: You can paste gist.github.com links for larger files -->

1. 1. 1. 1. 1. EXPECTED RESULTS
               <!--- Describe what you expected to happen when running the steps above -->
               All tasks complete the same as if they were run with the option `serial: 1` being set

1. 1. 1. 1. 1. ACTUAL RESULTS
               <!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->
               On the 7th task, one of the 12 vcenters in the inventory threw the error below while all of the others completed successfully. Retrying it may succeed on all, or fail with similar message on a different vcenter with the same error
               <!--- Paste verbatim command output between quotes -->
               ```paste below
               An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible_collections.cloud.common.plugins.module_utils.turbo.exceptions.EmbeddedModuleUnexpectedFailure: Cannot decode plugin answer: b''
               fatal: [randomvcenter.domain.com -> localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_vmware.vmware_rest.appliance_access_ssh_payload_ypjpjcua/ansible_vmware.vmware_rest.appliance_access_ssh_payload.zip/ansible_collections/cloud/common/plugins/module_utils/turbo/common.py\", line 106, in communicate\n File \"/usr/lib64/python3.8/json/__init__.py\", line 357, in loads\n return _default_decoder.decode(s)\n File \"/usr/lib64/python3.8/json/decoder.py\", line 337, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File \"/usr/lib64/python3.8/json/decoder.py\", line 355, in raw_decode\n raise JSONDecodeError(\"Expecting value\", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/root/.ansible/tmp/ansible-tmp-1668545865.656959-11481-114100221361235/AnsiballZ_appliance_access_ssh.py\", line 107, in <module>\n _ansiballz_main()\n File \"/root/.ansible/tmp/ansible-tmp-1668545865.656959-11481-114100221361235/AnsiballZ_appliance_access_ssh.py\", line 99, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/root/.ansible/tmp/ansible-tmp-1668545865.656959-11481-114100221361235/AnsiballZ_appliance_access_ssh.py\", line 47, in invoke_module\n runpy.run_module(mod_name='ansible_collections.vmware.vmware_rest.plugins.modules.appliance_access_ssh', init_globals=dict(_module_fqn='ansible_collections.vmware.vmware_rest.plugins.modules.appliance_access_ssh', _modlib_path=modlib_path),\n File \"/usr/lib64/python3.8/runpy.py\", line 207, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.8/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib64/python3.8/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_vmware.vmware_rest.appliance_access_ssh_payload_ypjpjcua/ansible_vmware.vmware_rest.appliance_access_ssh_payload.zip/ansible_collections/vmware/vmware_rest/plugins/modules/appliance_access_ssh.py\", line 261, in <module>\n File \"/usr/lib64/python3.8/asyncio/base_events.py\", line 616, in run_until_complete\n return future.result()\n File \"/tmp/ansible_vmware.vmware_rest.appliance_access_ssh_payload_ypjpjcua/ansible_vmware.vmware_rest.appliance_access_ssh_payload.zip/ansible_collections/vmware/vmware_rest/plugins/modules/appliance_access_ssh.py\", line 177, in main\n File \"/tmp/ansible_vmware.vmware_rest.appliance_access_ssh_payload_ypjpjcua/ansible_vmware.vmware_rest.appliance_access_ssh_payload.zip/ansible_collections/cloud/common/plugins/module_utils/turbo/module.py\", line 119, in __init__\n File \"/tmp/ansible_vmware.vmware_rest.appliance_access_ssh_payload_ypjpjcua/ansible_vmware.vmware_rest.appliance_access_ssh_payload.zip/ansible_collections/cloud/common/plugins/module_utils/turbo/module.py\", line 154, in run_on_daemon\n File \"/tmp/ansible_vmware.vmware_rest.appliance_access_ssh_payload_ypjpjcua/ansible_vmware.vmware_rest.appliance_access_ssh_payload.zip/ansible_collections/cloud/common/plugins/module_utils/turbo/common.py\", line 109, in communicate\nansible_collections.cloud.common.plugins.module_utils.turbo.exceptions.EmbeddedModuleUnexpectedFailure: Cannot decode plugin answer: b''\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

               ```