-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
None
-
None
-
False
-
None
-
False
-
-
On OCP, when we deploy for example 4 broker pods, we can easily see all queue stats using --clustered option of artemis queue stat. But after scale-down scenario, URLs seems to no working at all. (4 -> 2) we expect to work 0 and 1 pod urls, but they are not reachable anymore.
Example before scale-down
2024-08-19T07:01:03.022Z DEBUG [default][ExecutorOperator:79] [mirror-prod-tests] prod-broker-ss-0 Running command: ./amq-broker/bin/artemis queue stat --user=admin --password=adminPass --maxRows=1000 --queueName=queue --maxColumnSize=-1 --url=tcp://prod-broker-ss-0:61616 --clustered 2024-08-19T07:01:06.062Z DEBUG [default][BundledArtemisClient:85] NOTE: Picked up JDK_JAVA_OPTIONS: -Dbroker.properties=/amq/extra/secrets/prod-broker-props/broker.properties Connection brokerURL = tcp://prod-broker-ss-0:61616 2024-08-19 07:00:58,849 DEBUG [org.apache.activemq.artemis.utils.UUIDGenerator] using hardware address a:58:a:ffffff81:2:71 2024-08-19 07:00:59,168 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] Setting up call broker::getNodeID::[] 2024-08-19 07:00:59,182 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] management result:: 473bab17-5df8-11ef-a427-0a580a810247 2024-08-19 07:00:59,194 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] Setting up call broker::listNetworkTopology::[] 2024-08-19 07:00:59,200 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] management result:: [{"nodeID":"473bab17-5df8-11ef-a427-0a580a810247","live":"prod-broker-ss-0.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616","primary":"prod-broker-ss-0.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616"},{"nodeID":"5aefc4ad-5df8-11ef-8dd0-0a580a810248","live":"prod-broker-ss-1.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616","primary":"prod-broker-ss-1.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616"},{"nodeID":"725cc669-5df8-11ef-a607-0a580a81024a","live":"prod-broker-ss-3.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616","primary":"prod-broker-ss-3.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616"},{"nodeID":"65b778ee-5df8-11ef-9b5a-0a580a810249","live":"prod-broker-ss-2.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616","primary":"prod-broker-ss-2.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616"}] ******************************************************************************************************************************* >>> Queue stats on node 473bab17-5df8-11ef-a427-0a580a810247, url=tcp://prod-broker-ss-0:61616 |NAME |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL| | | | COUNT | COUNT | ADDED | COUNT | ACKED | COUNT | TYPE | | |queuea|queuea | 0 | 2 | 2 | 0 | 0 | 0 |ANYCAST| false | |queueb|queueb | 0 | 1 | 1 | 0 | 0 | 0 |ANYCAST| false | ******************************************************************************************************************************* >>> Queue stats on node 5aefc4ad-5df8-11ef-8dd0-0a580a810248, url=tcp://prod-broker-ss-1.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616 |NAME |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL| | | | COUNT | COUNT | ADDED | COUNT | ACKED | COUNT | TYPE | | |queuea|queuea | 0 | 2 | 2 | 0 | 0 | 0 |ANYCAST| false | |queueb|queueb | 0 | 1 | 1 | 0 | 0 | 0 |ANYCAST| false | ******************************************************************************************************************************* >>> Queue stats on node 725cc669-5df8-11ef-a607-0a580a81024a, url=tcp://prod-broker-ss-3.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616 |NAME |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL| | | | COUNT | COUNT | ADDED | COUNT | ACKED | COUNT | TYPE | | |queuea|queuea | 0 | 2 | 2 | 0 | 0 | 0 |ANYCAST| false | |queueb|queueb | 0 | 1 | 1 | 0 | 0 | 0 |ANYCAST| false | ******************************************************************************************************************************* >>> Queue stats on node 65b778ee-5df8-11ef-9b5a-0a580a810249, url=tcp://prod-broker-ss-2.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616 |NAME |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL| | | | COUNT | COUNT | ADDED | COUNT | ACKED | COUNT | TYPE | | |queuea|queuea | 0 | 2 | 2 | 0 | 0 | 0 |ANYCAST| false | |queueb|queueb | 0 | 1 | 1 | 0 | 0 | 0 |ANYCAST| false |
After scale-down 4 -> 2
2024-08-19T07:04:01.399Z DEBUG [default][ExecutorOperator:79] [mirror-dr-tests] dr-broker-ss-0 Running command: ./amq-broker/bin/artemis queue stat --user=admin --password=adminPass --maxRows=1000 --queueName=queue --maxColumnSize=-1 --url=tcp://dr-broker-ss-0:61616 --clustered 2024-08-19T07:04:04.664Z DEBUG [default][BundledArtemisClient:85] NOTE: Picked up JDK_JAVA_OPTIONS: -Dbroker.properties=/amq/extra/secrets/dr-broker-props/broker.properties Connection brokerURL = tcp://dr-broker-ss-0:61616 2024-08-19 07:03:57,592 DEBUG [org.apache.activemq.artemis.utils.UUIDGenerator] using hardware address a:58:a:ffffff81:2:70 2024-08-19 07:03:57,875 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] Setting up call broker::getNodeID::[] 2024-08-19 07:03:57,888 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] management result:: 3aaacf73-5df8-11ef-bc87-0a580a810246 2024-08-19 07:03:57,896 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] Setting up call broker::listNetworkTopology::[] 2024-08-19 07:03:57,900 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] management result:: [{"nodeID":"8b621c8e-5df8-11ef-94eb-0a580a81024b","live":"dr-broker-ss-2:61616","primary":"dr-broker-ss-2:61616"},{"nodeID":"3aaacf73-5df8-11ef-bc87-0a580a810246","live":"dr-broker-ss-0.dr-broker-hdls-svc.mirror-dr-tests.svc.cluster.local:61616","primary":"dr-broker-ss-0.dr-broker-hdls-svc.mirror-dr-tests.svc.cluster.local:61616"},{"nodeID":"963060c1-5df8-11ef-b231-0a580a81024c","live":"dr-broker-ss-3:61616","primary":"dr-broker-ss-3:61616"},{"nodeID":"81ae9745-5df8-11ef-98d2-0a580a800224","live":"dr-broker-ss-1.dr-broker-hdls-svc.mirror-dr-tests.svc.cluster.local:61616","primary":"dr-broker-ss-1.dr-broker-hdls-svc.mirror-dr-tests.svc.cluster.local:61616"}] ******************************************************************************************************************************* >>> Queue stats on node 3aaacf73-5df8-11ef-bc87-0a580a810246, url=tcp://dr-broker-ss-0:61616 |NAME |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL| | | | COUNT | COUNT | ADDED | COUNT | ACKED | COUNT | TYPE | | |queuea|queuea | 0 | 6 | 6 | 0 | 0 | 0 |ANYCAST| false | |queueb|queueb | 0 | 3 | 3 | 0 | 0 | 0 |ANYCAST| false | ******************************************************************************************************************************* >>> Queue stats on node 8b621c8e-5df8-11ef-94eb-0a580a81024b, url=tcp://dr-broker-ss-2:61616 2024-08-19 07:03:57,951 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: dr-broker-ss-2: Name or service not known at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?] ... Connection failed::Failed to create session factory
But sometimes (when stars align properly and order of broker pods) it's working "as expected"
2024-08-19T07:02:46.156Z DEBUG [default][ExecutorOperator:79] [mirror-prod-tests] prod-broker-ss-0 Running command: ./amq-broker/bin/artemis queue stat --user=admin --password=adminPass --maxRows=1000 --queueName=queue --maxColumnSize=-1 --url=tcp://prod-broker-ss-0:61616 --clustered 2024-08-19T07:02:49.183Z DEBUG [default][BundledArtemisClient:85] NOTE: Picked up JDK_JAVA_OPTIONS: -Dbroker.properties=/amq/extra/secrets/prod-broker-props/broker.properties Connection brokerURL = tcp://prod-broker-ss-0:61616 2024-08-19 07:02:42,075 DEBUG [org.apache.activemq.artemis.utils.UUIDGenerator] using hardware address a:58:a:ffffff81:2:71 2024-08-19 07:02:42,360 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] Setting up call broker::getNodeID::[] 2024-08-19 07:02:42,372 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] management result:: 473bab17-5df8-11ef-a427-0a580a810247 2024-08-19 07:02:42,378 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] Setting up call broker::listNetworkTopology::[] 2024-08-19 07:02:42,382 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] management result:: [{"nodeID":"473bab17-5df8-11ef-a427-0a580a810247","live":"prod-broker-ss-0.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616","primary":"prod-broker-ss-0.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616"},{"nodeID":"5aefc4ad-5df8-11ef-8dd0-0a580a810248","live":"prod-broker-ss-1.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616","primary":"prod-broker-ss-1.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616"},{"nodeID":"725cc669-5df8-11ef-a607-0a580a81024a","live":"prod-broker-ss-3:61616","primary":"prod-broker-ss-3:61616"},{"nodeID":"65b778ee-5df8-11ef-9b5a-0a580a810249","live":"prod-broker-ss-2:61616","primary":"prod-broker-ss-2:61616"}] ******************************************************************************************************************************* >>> Queue stats on node 473bab17-5df8-11ef-a427-0a580a810247, url=tcp://prod-broker-ss-0:61616 |NAME |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL| | | | COUNT | COUNT | ADDED | COUNT | ACKED | COUNT | TYPE | | |queuea|queuea | 0 | 6 | 6 | 0 | 0 | 0 |ANYCAST| false | |queueb|queueb | 0 | 3 | 3 | 0 | 0 | 0 |ANYCAST| false | ******************************************************************************************************************************* >>> Queue stats on node 5aefc4ad-5df8-11ef-8dd0-0a580a810248, url=tcp://prod-broker-ss-1.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616 |NAME |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL| | | | COUNT | COUNT | ADDED | COUNT | ACKED | COUNT | TYPE | | |queuea|queuea | 0 | 2 | 2 | 0 | 0 | 0 |ANYCAST| false | |queueb|queueb | 0 | 1 | 1 | 0 | 0 | 0 |ANYCAST| false | ******************************************************************************************************************************* >>> Queue stats on node 725cc669-5df8-11ef-a607-0a580a81024a, url=tcp://prod-broker-ss-3:61616 2024-08-19 07:02:42,466 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: prod-broker-ss-3: Name or service not known at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?] ...
- is related to
-
ENTMQBR-9354 Mirroring deployment - message migration duplicates on scaledown
- Backlog
- relates to
-
ENTMQBR-9091 Broker keeps on connecting to old cluster instance after scaling down
- Closed