Background:

Satellite tries to keep track which capsules were used to run remote execution jobs for which hosts, and if satellite determines it is the first execution, it tries to remove the known host keys from /usr/share/foreman-proxy/.ssh/known_hosts before trying to connect to the host.

This cleanup mechanism works as follows:

• If the first remote execution job run against a specific host uses script, then the cleanup mechanism works. i.e., it checks /usr/share/foreman-proxy/.ssh/known_hosts on the Satellite (or Capsule) server, and if an old host key is stored there, it removes it.
• If the first remote execution job run against a specific host uses Ansible, the hosts key cleanup is not performed. This still counts as an execution through a proxy, and when a remote execution job using a script is executed, the hosts key cleanup does not take place because it is not the first remote task executed against the host. 

Description of problem:

A problem arises when the remote execution job run against a specific host uses Ansible. When this occurs, the old host key is not removed from /usr/share/foreman-proxy/.ssh/known_hosts on the Satellite (or Capsule) server. In this case, if a remote execution job using script is then run against the host, it fails due to the old host key still present in /usr/share/foreman-proxy/.ssh/known_hosts on the Satellite (or Capsule) server, reporting the following error:
Output:

—
proxy_task_id: 67849cb6-ef94-472b-b482-5d95926ca5fa
proxy_output:
result:

• output_type: debug
  output: "Error initializing command: RuntimeError - Could not establish connection to remote host using any available authentication method, tried publickey\n\nAuthentication
  method 'publickey' failed with:\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@
  \ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nIT
  IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!\r\nSomeone could be eavesdropping on you right now (man-in-the-middle attack)!\r\nIt is also possible that a host key has just been changed.\r\nThe fingerprint for the ECDSA key sent by the remote host is\nSHA256:2nI+S3w9OOOuDAletMfRLNiAmS8dA953sDMcdbjAaMA.\r\nPlease contact your system administrator.\r\nAdd correct host key in /usr/share/foreman-proxy/.ssh/known_hosts to get rid of this message.\r\nOffending ECDSA key in /usr/share/foreman-proxy/.ssh/known_hosts:17\r\nECDSA
  host key for host.example.com has changed and you have requested strict checking.\r\nHost key verification failed.\r\n"
  timestamp: 1723718281.3572552
  runner_id: a9e3dc44-0927-49c1-9017-5dd8a3f0c573
  exit_status: EXCEPTION
  The problem arises when a host is re-provisioned via Satellite. Default Ansible roles are run as part of the provisioning, which means that the first execution against the host is using Ansible.

How reproducible:

Always, when a host is re-provisioned via Satellite.

Is this issue a regression from an earlier version:

No.

Steps to Reproduce:

1. Re-provision a host via Satellite.

2. Run a remote execution job using a script against the host.

Actual behavior:

See the description of the problem above.

Expected behavior:

When a host is being re-provisioned via Satellite, the host key in /usr/share/foreman-proxy/.ssh/known_hosts on the Satellite (or Capsule) server is removed regardless whether the first remote execution job is using Ansible or script.

Business Impact / Additional info:

• Unable to run remote execution jobs, and hence need to fail back to performing the actions manually on each host.
• The impact is significant when performing patching against several hundreds or several thousands of hosts registered to Satellite.