-
Bug
-
Resolution: Done
-
Minor
-
None
-
rhos-18.0 Beta
-
None
-
False
-
-
False
-
?
-
?
-
?
-
?
-
None
-
Low
After an upgrade from 4.15.18 to 4.15.19 (not sure if it's related though), I have noticed that nova_compute service is down on both EDPM nodes.
The message I see in the logs is:
2024-07-08 15:42:28.839 2 ERROR oslo.messaging._drivers.impl_rabbit [-] [d747231a-14d1-489b-aa23-2f4d855acc24] AMQP server on rabbitmq-cell1.openstack.svc:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 2 seconds.: OSError: [Errno 113] EHOSTUNREACH 2024-07-08 15:42:28.839 2 ERROR oslo.messaging._drivers.impl_rabbit [-] [4ba05b6b-1dca-4cc6-9cf0-14bfd8a5249b] AMQP server on rabbitmq-cell1.openstack.svc:5671 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 2 seconds.: OSError: [Errno 113] EHOSTUNREACH 2024-07-08 15:42:31.911 2 ERROR oslo.messaging._drivers.impl_rabbit [-] [d747231a-14d1-489b-aa23-2f4d855acc24] AMQP server on rabbitmq-cell1.openstack.svc:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
A ping seems to unsuccessful.
[root@edpm-compute1 containers]# ping rabbitmq-cell1.openstack.svc PING rabbitmq-cell1.openstack.svc (172.17.0.86) 56(84) bytes of data.
From the worker node where all three rabbitmq-cell1 pods are running, containers are up and running.
sh-5.1# crictl ps | grep rabbitmq-cell1 2e5b222c0f768 db6025adad103c241a04ea49d00aeff5470cf4b28f708d0895e9d682df6abc14 29 minutes ago Running rabbitmq 0 855eaa9898b05 rabbitmq-cell1-server-2 84cd84ac37015 db6025adad103c241a04ea49d00aeff5470cf4b28f708d0895e9d682df6abc14 29 minutes ago Running rabbitmq 0 655d55cd43c82 rabbitmq-cell1-server-1 d8eb2408b5567 db6025adad103c241a04ea49d00aeff5470cf4b28f708d0895e9d682df6abc14 29 minutes ago Running rabbitmq 0 24cd84ec5c2a8 rabbitmq-cell1-server-0
However, I don't see anything listening on 5671 port.
sh-5.1# ss -laputen | grep 5671 tcp TIME-WAIT 0 0 10.131.0.1:46276 10.131.0.53:5671 timer:(timewait,49sec,0) ino:0 sk:2002 tcp TIME-WAIT 0 0 10.131.0.1:34198 10.131.0.56:5671 timer:(timewait,52sec,0) ino:0 sk:200a tcp TIME-WAIT 0 0 10.131.0.1:57452 10.131.0.47:5671 timer:(timewait,59sec,0) ino:0 sk:2010 tcp TIME-WAIT 0 0 10.131.0.1:58806 10.131.0.53:5671 timer:(timewait,59sec,0) ino:0 sk:2011 tcp TIME-WAIT 0 0 10.131.0.1:38280 10.131.0.53:5671 timer:(timewait,39sec,0) ino:0 sk:2023 tcp TIME-WAIT 0 0 10.131.0.1:46150 10.131.0.56:5671 timer:(timewait,12sec,0) ino:0 sk:2024 tcp TIME-WAIT 0 0 10.131.0.1:60878 10.131.0.47:5671 timer:(timewait,39sec,0) ino:0 sk:2027 tcp TIME-WAIT 0 0 10.131.0.1:54404 10.131.0.47:5671 timer:(timewait,29sec,0) ino:0 sk:2028
Please advise.
