Jenkins CRW_CI jobs use rhel8 and rhel83 tags to pick an x86 machine whose image is generated with ansible.  These machines are launched and deleted for each job using openstack jenkins integration.  Each job will also perform system configuration--a lengthy install process often with network failure workarounds (CRW-1610) in https://github.com/redhat-developer/codeready-workspaces/blob/crw-2-rhel-8/product/util.groovy#L526-L562

This task will remove system configuration from CRW_CI jenkinsfiles and place it in existing ansible roles https://gitlab.cee.redhat.com/codeready-workspaces/ansible-scripts/-/tree/master/roles.  Jobs using older rhel8 and rhel83 images will be migrated to new image.  The rhel8 and rhel83 tags will be removed and will instead use x86_64-rhel8 (consistent with s390x-rhel8 and ppc64le-rhel8).  Three permanent x86_64-rhel8 nodes will be deployed to reduce/eliminate openstack deployment time–speeding up job execution time and producing cleaner logs.

The image will be kept up-to-date with a new system/jenkins-node-setup job.  This job will run the ansible roles on a new deployment and take a snapshot of the machine.  It will also run the ansible roles on all our permanent machines.

Ansible changes to note:

• new users role – refactor all user specific configuration from existing roles and place it within a specific users profile in users role.
• removed verify roles – ensure roles fail if not run correctly
• python and node are installed per user.
• python is managed by pyenv.  currently only the latest version of python is installed as default for hudson user.
• node is managed by npm.  ~hudson/crw_env.sh must be sourced with crw version to add node to path (ex. "source ~/crw_env.sh 2.14").  Node and dependency versions to install are picked up from job-config.yaml.
• authentication secrets (kerberos, registries, etc) are picked up from crw-jenkins/secrets instead of being kept in ansible-scripts repo (see jenkins-node-setup job).
• current yum repos are removed and rebuilt from repos role
• common role does system cleanup, updates packages, installs common packages, disables selinux