Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Undefined
Fix Version/s: rhos-17.1.6
Affects Version/s: None
Component/s: tripleo-ansible
Labels:
None

Story Points:
2
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Docs Approval:
?
Regression:
None
Intelligence Requested:
Market:

Severity:
Moderate

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

In a given RHOSP 16.2.6 environment, I have (at a minimum) the following ssh keys:

Undercloud:

[stack@director.9d78 ~]$ sudo md5sum /home/stack/.ssh/id_rsa /home/tripleo-admin/.ssh/id_rsa /var/lib/mistral/.ssh/tripleo-admin-rsa
b13451cfe05a3168c0575f73af85e0a4  /home/stack/.ssh/id_rsa
45a4c282418b68c7e46c88c9fc55281a  /home/tripleo-admin/.ssh/id_rsa
45a4c282418b68c7e46c88c9fc55281a  /var/lib/mistral/.ssh/tripleo-admin-rsa
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ sudo ls -ltr /home/stack/.ssh/id_rsa /home/tripleo-admin/.ssh/id_rsa /var/lib/mistral/.ssh/tripleo-admin-rsa
-rw-------. 1         42430         42430 2635 Mar 24  2023 /var/lib/mistral/.ssh/tripleo-admin-rsa
-rw-------. 1 tripleo-admin tripleo-admin 2635 Mar 24  2023 /home/tripleo-admin/.ssh/id_rsa
-rw-------. 1 stack         stack         1052 Mar 24  2023 /home/stack/.ssh/id_rsa
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ for x in /home/stack/.ssh/id_rsa /home/tripleo-admin/.ssh/id_rsa /var/lib/mistral/.ssh/tripleo-admin-rsa ;do
> sudo ssh-keygen -l -f $x
> done
1024 SHA256:AK7nnqUB/QsNpjtaI9pFJaTGPmCCmzpvUmCWcddKA7E root@director.example.com (RSA)
3072 SHA256:Yum9VxfqNBu+LVuWgAQCJL4u+r1/BdAivTPz1teZTAk ansible-generated on director.example.com (RSA)
3072 SHA256:Yum9VxfqNBu+LVuWgAQCJL4u+r1/BdAivTPz1teZTAk ansible-generated on director.example.com (RSA)
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ cat .ssh/id_rsa.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
[stack@director.9d78 ~]$

When I check the overcloud node, only /home/stack/.ssh/id_rsa.pub is in use, both for heat-admin and tripleo-admin (2 times the same key on each account, for whatever reaon):

[stack@director.9d78 ~]$ ansible -i inventory.yaml -m shell -a 'cat /home/*-admin/.ssh/authorized_keys' -b Controller[0]:Compute[0]
overcloud-novacompute-0 | CHANGED | rc=0 >>
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
                
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
overcloud-controller-0 | CHANGED | rc=0 >>
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
                
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
[stack@director.9d78 ~]$

After the undercloud and overcloud containers are upgraded, there one more key present, and I check whether these are hard links but they are not, they are different files:

[stack@director.9d78 ~]$ sudo ls -ltri /home/stack/overcloud-deploy/overcloud/ssh_private_key /home/stack/.ssh/id_rsa /home/tripleo-admin/.ssh/id_rsa /var/lib/mistral/.ssh/tripleo-admin-rsa
134396420 -rw-------. 1         42430         42430 2635 Mar 24  2023 /var/lib/mistral/.ssh/tripleo-admin-rsa
 25267570 -rw-------. 1 tripleo-admin tripleo-admin 2635 Mar 24  2023 /home/tripleo-admin/.ssh/id_rsa
161391353 -rw-------. 1 stack         stack         1052 Mar 24  2023 /home/stack/.ssh/id_rsa
177123194 -rw-------. 1 stack         stack         1052 Feb 27 21:27 /home/stack/overcloud-deploy/overcloud/ssh_private_key
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ cat /etc/rhosp-release /etc/redhat-release 
Red Hat OpenStack Platform release 17.1.4 (Wallaby)
Red Hat Enterprise Linux release 8.4 (Ootpa)
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ sudo md5sum /home/stack/overcloud-deploy/overcloud/ssh_private_key /home/stack/.ssh/id_rsa /home/tripleo-admin/.ssh/id_rsa /var/lib/mistral/.ssh/tripleo-admin-rsa
b13451cfe05a3168c0575f73af85e0a4  /home/stack/overcloud-deploy/overcloud/ssh_private_key
b13451cfe05a3168c0575f73af85e0a4  /home/stack/.ssh/id_rsa
45a4c282418b68c7e46c88c9fc55281a  /home/tripleo-admin/.ssh/id_rsa
45a4c282418b68c7e46c88c9fc55281a  /var/lib/mistral/.ssh/tripleo-admin-rsa
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ for x in /home/stack/overcloud-deploy/overcloud/ssh_private_key /home/stack/.ssh/id_rsa /home/tripleo-admin/.ssh/id_rsa /var/lib/mistral/.ssh/tripleo-admin-rsa ;do sudo ssh-keygen -l -f $x; done
1024 SHA256:AK7nnqUB/QsNpjtaI9pFJaTGPmCCmzpvUmCWcddKA7E root@director.example.com (RSA)
1024 SHA256:AK7nnqUB/QsNpjtaI9pFJaTGPmCCmzpvUmCWcddKA7E root@director.example.com (RSA)
3072 SHA256:Yum9VxfqNBu+LVuWgAQCJL4u+r1/BdAivTPz1teZTAk ansible-generated on director.example.com (RSA)
3072 SHA256:Yum9VxfqNBu+LVuWgAQCJL4u+r1/BdAivTPz1teZTAk ansible-generated on director.example.com (RSA)
[stack@director.9d78 ~]$

The adopted overcloud is still using one single ssh key everywhere:

stack@director.9d78 ~]$ ansible -i inventory.yaml -m shell -a 'grep -vH ^$ {/root,/home/*-admin}/.ssh/authorized_keys' -b Controller[0]:Compute[0]
overcloud-novacompute-0 | CHANGED | rc=0 >>
/root/.ssh/authorized_keys:no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"heat-admin\" rather than the user \"root\".';echo;sleep 10;exit 142" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
/home/heat-admin/.ssh/authorized_keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
/home/heat-admin/.ssh/authorized_keys:                
/home/heat-admin/.ssh/authorized_keys:                
/home/tripleo-admin/.ssh/authorized_keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
/home/tripleo-admin/.ssh/authorized_keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
/home/tripleo-admin/.ssh/authorized_keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
overcloud-controller-0 | CHANGED | rc=0 >>
/root/.ssh/authorized_keys:no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"heat-admin\" rather than the user \"root\".';echo;sleep 10;exit 142" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
/home/heat-admin/.ssh/authorized_keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
/home/heat-admin/.ssh/authorized_keys:                
/home/heat-admin/.ssh/authorized_keys:                
/home/tripleo-admin/.ssh/authorized_keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
/home/tripleo-admin/.ssh/authorized_keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
/home/tripleo-admin/.ssh/authorized_keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
[stack@director.9d78 ~]$

Given that the length of the ssh key in /home/stack/.ssh/id_rsa is 1024, I run the rotation process as indicated in 13.3 on the FFU guide, without doing any other cleanup or customization.

[stack@director.9d78 ~]$ ansible-playbook -i /home/stack/overcloud-deploy/overcloud/tripleo-ansible-inventory.yaml \
> -e undercloud_backup_folder=/home/stack/overcloud_backup_keys \
> -e stack_name=overcloud \
> /usr/share/ansible/tripleo-playbooks/ssh_key_rotation.yaml
...
TASK [ssh to overcloud nodes from undercloud] ****************************************************************************************************************************
changed: [overcloud-novacompute-0 -> undercloud]
changed: [overcloud-novacompute-1 -> undercloud]
changed: [overcloud-controller-0 -> undercloud]
changed: [overcloud-controller-1 -> undercloud]
changed: [overcloud-controller-2 -> undercloud]

PLAY RECAP ***************************************************************************************************************************************************************
overcloud-controller-0     : ok=11   changed=4    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
overcloud-controller-1     : ok=11   changed=4    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
overcloud-controller-2     : ok=11   changed=4    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
overcloud-novacompute-0    : ok=11   changed=4    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
overcloud-novacompute-1    : ok=11   changed=4    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
undercloud                 : ok=32   changed=11   unreachable=0    failed=0    skipped=9    rescued=0    ignored=0   

[stack@director.9d78 ~]$

At this point it looks like the only key that was changed is /home/stack/overcloud-deploy/overcloud/ssh_private_key:

[stack@director.9d78 ~]$ sudo md5sum /home/stack/overcloud-deploy/overcloud/ssh_private_key /home/stack/.ssh/id_rsa /home/tripleo-admin/.ssh/id_rsa /var/lib/mistral/.ssh/tripleo-admin-rsa
c27e17944348d6c45ddb973611cfd84b  /home/stack/overcloud-deploy/overcloud/ssh_private_key
b13451cfe05a3168c0575f73af85e0a4  /home/stack/.ssh/id_rsa
45a4c282418b68c7e46c88c9fc55281a  /home/tripleo-admin/.ssh/id_rsa
45a4c282418b68c7e46c88c9fc55281a  /var/lib/mistral/.ssh/tripleo-admin-rsa
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ for x in /home/stack/overcloud-deploy/overcloud/ssh_private_key /home/stack/.ssh/id_rsa /home/tripleo-admin/.ssh/id_rsa /var/lib/mistral/.ssh/tripleo-admin-rsa ;do sudo ssh-keygen -l -f $x; done
4096 SHA256:ZqYvBZ07Yqw2otyzqo09cUARBA4CVrgqNinLZtHw9m4 stack@director.example.com (RSA)
1024 SHA256:AK7nnqUB/QsNpjtaI9pFJaTGPmCCmzpvUmCWcddKA7E root@director.example.com (RSA)
3072 SHA256:Yum9VxfqNBu+LVuWgAQCJL4u+r1/BdAivTPz1teZTAk ansible-generated on director.example.com (RSA)
3072 SHA256:Yum9VxfqNBu+LVuWgAQCJL4u+r1/BdAivTPz1teZTAk ansible-generated on director.example.com (RSA)
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ sudo ls -ltri /home/stack/overcloud-deploy/overcloud/ssh_private_key /home/stack/.ssh/id_rsa /home/tripleo-admin/.ssh/id_rsa /var/lib/mistral/.ssh/tripleo-admin-rsa
134396420 -rw-------. 1         42430         42430 2635 Mar 24  2023 /var/lib/mistral/.ssh/tripleo-admin-rsa
 25267570 -rw-------. 1 tripleo-admin tripleo-admin 2635 Mar 24  2023 /home/tripleo-admin/.ssh/id_rsa
161391353 -rw-------. 1 stack         stack         1052 Mar 24  2023 /home/stack/.ssh/id_rsa
343934903 -rw-------. 1 stack         stack         3389 Feb 28 07:56 /home/stack/overcloud-deploy/overcloud/ssh_private_key
[stack@director.9d78 ~]$

And the new key was added only to tripleo-admin in the overcloud:

[stack@director.9d78 ~]$ ansible -i inventory.yaml -m shell -a 'ls -ltr {/root,/home/*-admin}/.ssh/authorized_keys' -b Controller[0]:Compute[0]
overcloud-novacompute-0 | CHANGED | rc=0 >>
-rw-------. 1 root          root          407 Aug  8  2023 /root/.ssh/authorized_keys
-rw-------. 1 heat-admin    heat-admin    273 Feb 27 19:47 /home/heat-admin/.ssh/authorized_keys
-rw-------. 1 tripleo-admin tripleo-admin 992 Feb 28 06:56 /home/tripleo-admin/.ssh/authorized_keys
overcloud-controller-0 | CHANGED | rc=0 >>
-rw-------. 1 root          root          407 Aug  8  2023 /root/.ssh/authorized_keys
-rw-------. 1 heat-admin    heat-admin    273 Feb 27 19:47 /home/heat-admin/.ssh/authorized_keys
-rw-------. 1 tripleo-admin tripleo-admin 992 Feb 28 06:56 /home/tripleo-admin/.ssh/authorized_keys
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ ansible -i inventory.yaml -m shell -a 'wc -l {/root,/home/*-admin}/.ssh/authorized_keys' -b Controller[0]:Compute[0]
overcloud-novacompute-0 | CHANGED | rc=0 >>
   1 /root/.ssh/authorized_keys
   3 /home/heat-admin/.ssh/authorized_keys
   3 /home/tripleo-admin/.ssh/authorized_keys
   7 total
overcloud-controller-0 | CHANGED | rc=0 >>
   1 /root/.ssh/authorized_keys
   3 /home/heat-admin/.ssh/authorized_keys
   3 /home/tripleo-admin/.ssh/authorized_keys
   7 total
[stack@director.9d78 ~]$
[stack@director.9d78 ~]$ ssh -l tripleo-admin overcloud-controller-0.ctlplane
Last login: Fri Feb 28 07:07:30 2025 from 172.25.250.1
[tripleo-admin@overcloud-controller-0 ~]$ sudo -i
[root@overcloud-controller-0 ~]# cat /home/tripleo-admin/.ssh/authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1ABn1Iqt1YLHAtmZruT7LYrVwF//dVBVRR79oNQy7aOpcmerZ2ZQ/el6EwRjCHn/VcD6xKnOoc/BYVTI8YjvwIdgV2suuBa8YKlp84hyxQakCRQTGhU3mnSJN3+gOPJsA2gC9Ghkgp1f0XTcAEyWP2V6IXGGEo2SVz49yQFStI8Q3GjCbTX79zvkINrr1X9rFq/8fU8Vc1FGFTQhmA6yY27Hc5Ob5N3YTPybAOeAsOfG5XDgtNWKBUKxmauPrsEt6s3cQbpvo0VSIDxVBbanRTqqsk7RHy9gZaZfbKJIW5eDxw6j4hmcO/09stV9tKfhipgKfW+p4Ftolo/uY1ypoY45EEOpwbWY/07IK6F7vfxv+XKHP/urjUGokKWcldB6Pjnn9bsRoVq2WBLHuep7IQ/LT+xfVqJpmYk+0lHSYzjD8CP75BMWjZ8ZpM0jIR+z8nOTYKE7ALFS8UMIym1tB7jcvYPYG1IxBeYf/TFJB88I5jX0mCHcSvMArY/eLlM3M7wFyoP3oye6/8ZwVPooxLZeCUy0kkFc75/idVM7vn923vy9tI6pzdoDG4FfEGBtFFqPcqWOwdUvCkOfjg8vZ2wY3RhjQdY3L5nbnRlU46EyBetpGI6PK3SEajnjgiSdTac4XAwBUMBo8kGqmnVaC1RDwRQRSH6fBeQzUdOIlvw== stack@director.example.com
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com

[root@overcloud-controller-0 ~]# cat /home/heat-admin/.ssh/authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
                
                
[root@overcloud-controller-0 ~]#

Again without any changes, cleanup or customization, I proceed with the undercloud OS upgrade as documented.

Not surprisingly given the above, ssh to overcloud does not work after the undercloud reboot to run LEAPP, and so a manual fix is required:

[stack@director.9d78 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.2 (Plow)
[stack@director.9d78 ~]$ ssh -l tripleo-admin overcloud-controller-0.ctlplane
load pubkey "/home/stack/.ssh/id_rsa": Invalid key length
tripleo-admin@overcloud-controller-0.ctlplane: Permission denied (publickey).
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ ll /home/stack/overcloud-deploy/overcloud/ssh_private_key /home/stack/.ssh/id_rsa /home/stack/overcloud-deploy/overcloud/ssh_private_key.pub /home/stack/.ssh/id_rsa.pub
-rw-------. 1 stack stack 3389 Feb 28 07:56 /home/stack/overcloud-deploy/overcloud/ssh_private_key
-rw-------. 1 stack stack  752 Feb 28 07:56 /home/stack/overcloud-deploy/overcloud/ssh_private_key.pub
-rw-------. 1 stack stack 1052 Mar 24  2023 /home/stack/.ssh/id_rsa
-rw-r--r--. 1 stack stack  239 Mar 24  2023 /home/stack/.ssh/id_rsa.pub
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ cp /home/stack/overcloud-deploy/overcloud/ssh_private_key /home/stack/.ssh/id_rsa
[stack@director.9d78 ~]$ cp /home/stack/overcloud-deploy/overcloud/ssh_private_key.pub /home/stack/.ssh/id_rsa.pub
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ ssh tripleo-admin@overcloud-controller-0.ctlplane uptime
 07:49:52 up 11:40,  0 users,  load average: 3.23, 2.04, 1.29
[stack@director.9d78 ~]$ ssh tripleo-admin@overcloud-controller-0.ctlplane
Last login: Fri Feb 28 07:48:03 2025 from 172.25.250.1
[tripleo-admin@overcloud-controller-0 ~]$ logout
Connection to overcloud-controller-0.ctlplane closed.
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ ansible -i inventory.yaml -m ping Controller[0]:Compute[0]
overcloud-novacompute-0 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
overcloud-controller-0 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
[stack@director.9d78 ~]$

After that manual fix, I do the overcloud OS upgrade and the FFU process, as documented, is finished. Then I check one more time the presence of keys in the undercloud:

[stack@director.9d78 ~]$ sudo ls -ltri /home/stack/overcloud-deploy/overcloud/ssh_private_key /home/stack/.ssh/id_rsa /home/tripleo-admin/.ssh/id_rsa /var/lib/mistral/.ssh/tripleo-admin-rsa
134396420 -rw-------. 1         42430         42430 2635 Mar 24  2023 /var/lib/mistral/.ssh/tripleo-admin-rsa
 25267570 -rw-------. 1 tripleo-admin tripleo-admin 2635 Mar 24  2023 /home/tripleo-admin/.ssh/id_rsa
343934903 -rw-------. 1 stack         stack         3389 Feb 28 07:56 /home/stack/overcloud-deploy/overcloud/ssh_private_key
161391353 -rw-------. 1 stack         stack         3389 Feb 28 08:46 /home/stack/.ssh/id_rsa
[stack@director.9d78 ~]$

As well as the authorized_keys in the overcloud:

[stack@director.9d78 ~]$ cat .ssh/id_rsa.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1ABn1Iqt1YLHAtmZruT7LYrVwF//dVBVRR79oNQy7aOpcmerZ2ZQ/el6EwRjCHn/VcD6xKnOoc/BYVTI8YjvwIdgV2suuBa8YKlp84hyxQakCRQTGhU3mnSJN3+gOPJsA2gC9Ghkgp1f0XTcAEyWP2V6IXGGEo2SVz49yQFStI8Q3GjCbTX79zvkINrr1X9rFq/8fU8Vc1FGFTQhmA6yY27Hc5Ob5N3YTPybAOeAsOfG5XDgtNWKBUKxmauPrsEt6s3cQbpvo0VSIDxVBbanRTqqsk7RHy9gZaZfbKJIW5eDxw6j4hmcO/09stV9tKfhipgKfW+p4Ftolo/uY1ypoY45EEOpwbWY/07IK6F7vfxv+XKHP/urjUGokKWcldB6Pjnn9bsRoVq2WBLHuep7IQ/LT+xfVqJpmYk+0lHSYzjD8CP75BMWjZ8ZpM0jIR+z8nOTYKE7ALFS8UMIym1tB7jcvYPYG1IxBeYf/TFJB88I5jX0mCHcSvMArY/eLlM3M7wFyoP3oye6/8ZwVPooxLZeCUy0kkFc75/idVM7vn923vy9tI6pzdoDG4FfEGBtFFqPcqWOwdUvCkOfjg8vZ2wY3RhjQdY3L5nbnRlU46EyBetpGI6PK3SEajnjgiSdTac4XAwBUMBo8kGqmnVaC1RDwRQRSH6fBeQzUdOIlvw== stack@director.example.com
[stack@director.9d78 ~]$ 
[stack@director.9d78 ~]$ ssh tripleo-admin@overcloud-controller-0.ctlplane
Register this system with Red Hat Insights: insights-client --register
Create an account or view all your systems at https://red.ht/insights-dashboard
Last login: Fri Feb 28 11:28:37 2025 from 172.25.250.1
[tripleo-admin@overcloud-controller-0 ~]$ 
[tripleo-admin@overcloud-controller-0 ~]$ cat .ssh/authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1ABn1Iqt1YLHAtmZruT7LYrVwF//dVBVRR79oNQy7aOpcmerZ2ZQ/el6EwRjCHn/VcD6xKnOoc/BYVTI8YjvwIdgV2suuBa8YKlp84hyxQakCRQTGhU3mnSJN3+gOPJsA2gC9Ghkgp1f0XTcAEyWP2V6IXGGEo2SVz49yQFStI8Q3GjCbTX79zvkINrr1X9rFq/8fU8Vc1FGFTQhmA6yY27Hc5Ob5N3YTPybAOeAsOfG5XDgtNWKBUKxmauPrsEt6s3cQbpvo0VSIDxVBbanRTqqsk7RHy9gZaZfbKJIW5eDxw6j4hmcO/09stV9tKfhipgKfW+p4Ftolo/uY1ypoY45EEOpwbWY/07IK6F7vfxv+XKHP/urjUGokKWcldB6Pjnn9bsRoVq2WBLHuep7IQ/LT+xfVqJpmYk+0lHSYzjD8CP75BMWjZ8ZpM0jIR+z8nOTYKE7ALFS8UMIym1tB7jcvYPYG1IxBeYf/TFJB88I5jX0mCHcSvMArY/eLlM3M7wFyoP3oye6/8ZwVPooxLZeCUy0kkFc75/idVM7vn923vy9tI6pzdoDG4FfEGBtFFqPcqWOwdUvCkOfjg8vZ2wY3RhjQdY3L5nbnRlU46EyBetpGI6PK3SEajnjgiSdTac4XAwBUMBo8kGqmnVaC1RDwRQRSH6fBeQzUdOIlvw== stack@director.example.com
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com

[tripleo-admin@overcloud-controller-0 ~]$ 
[tripleo-admin@overcloud-controller-0 ~]$ sudo cat /home/heat-admin/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
                
                
[tripleo-admin@overcloud-controller-0 ~]$ sudo cat /root/.ssh/authorized_keys
no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"heat-admin\" rather than the user \"root\".';echo;sleep 10;exit 142" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC09mEFjXVaVV6m5QQS9yo2X/gu6IGxJZ0Pm3D3g2WcXj1i8i1nSEiLl+g0yUgCtPG8HqQUxQvjQK3QRLZ0OmGB8TF3Yjs+AC+1IE1ihdZ0q4+lI2W1Qkh9JhmH7eu1ZDCtjpP2AYifwV/ugsBeNGwRPrto4XMRPQTc4QuoeqyySQ== root@director.example.com
[tripleo-admin@overcloud-controller-0 ~]$

I see the following (potential?) issues with this process:

There's no task in the documentation indicating that the ssh key must be copied from /home/stack/overcloud-deploy/overcloud/ssh_private_key to /home/stack/.ssh/id_rsa, but is nevertheless required if you rotated the key before the OS upgrade. Otherwise ssh to overcloud nodes doesn't work by default.
There's 2 different keys in three different places in the undercloud before the upgrade. /home/stack/.ssh/id_rsa is different than /var/lib/mistral/.ssh/tripleo-admin-rsa, nevertheless the document indicates that I have to chown /var/lib/mistral/.ssh/tripleo-admin-rsa (Section 2.2, step 5), but this key is not present in authorized-keys of neither tripleo-admin or heat-admin in the overcloud nodes. Is this key even needed?
/var/lib/mistral/.ssh (or the whole mistral for that matter) is not cleaned up after the upgrade. Since mistral is not used in RHOSP 17.1, I assume this key is no longer needed after FFU and could be removed.

Note: This affects the ssh key rotation procedure in RHOSP 16.2, as I don't know for sure if the solution proposed in ~~OSPRH-14348~~ is complete.

is blocked by

OSPRH-16814 RHOSP 17.1 ssh key rotation process is incomplete

Closed

relates to

OSPRH-14348 ssh key rotation procedure in RHOSP 16.2 is broken

Closed

Assignee:: Andre Aranha

Reporter:: Eric Nothen

Team:: rhos-dfg-security

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Created:: 2025/03/03 9:36 AM

Updated:: 2025/05/30 9:45 PM

Resolved:: 2025/05/30 9:45 PM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates

PagerDuty