Loading...

Type: Bug
Resolution: Unresolved
Priority: Normal
Fix Version/s: None
Affects Version/s: None
Component/s: operator
Labels:
None

Blocked:
False
Blocked Reason:
None
Ready:
False
Intelligence Requested:
Market:

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

On OCP, when we deploy for example 4 broker pods, we can easily see all queue stats using --clustered option of artemis queue stat. But after scale-down scenario, URLs seems to no working at all. (4 -> 2) we expect to work 0 and 1 pod urls, but they are not reachable anymore.

Example before scale-down

2024-08-19T07:01:03.022Z DEBUG [default][ExecutorOperator:79] [mirror-prod-tests] prod-broker-ss-0 Running command: ./amq-broker/bin/artemis queue stat --user=admin --password=adminPass --maxRows=1000 --queueName=queue --maxColumnSize=-1 --url=tcp://prod-broker-ss-0:61616 --clustered
2024-08-19T07:01:06.062Z DEBUG [default][BundledArtemisClient:85] NOTE: Picked up JDK_JAVA_OPTIONS: -Dbroker.properties=/amq/extra/secrets/prod-broker-props/broker.properties
Connection brokerURL = tcp://prod-broker-ss-0:61616
2024-08-19 07:00:58,849 DEBUG [org.apache.activemq.artemis.utils.UUIDGenerator] using hardware address a:58:a:ffffff81:2:71
2024-08-19 07:00:59,168 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] Setting up call broker::getNodeID::[]
2024-08-19 07:00:59,182 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] management result:: 473bab17-5df8-11ef-a427-0a580a810247
2024-08-19 07:00:59,194 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] Setting up call broker::listNetworkTopology::[]
2024-08-19 07:00:59,200 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] management result:: [{"nodeID":"473bab17-5df8-11ef-a427-0a580a810247","live":"prod-broker-ss-0.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616","primary":"prod-broker-ss-0.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616"},{"nodeID":"5aefc4ad-5df8-11ef-8dd0-0a580a810248","live":"prod-broker-ss-1.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616","primary":"prod-broker-ss-1.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616"},{"nodeID":"725cc669-5df8-11ef-a607-0a580a81024a","live":"prod-broker-ss-3.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616","primary":"prod-broker-ss-3.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616"},{"nodeID":"65b778ee-5df8-11ef-9b5a-0a580a810249","live":"prod-broker-ss-2.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616","primary":"prod-broker-ss-2.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616"}]
*******************************************************************************************************************************
>>> Queue stats on node 473bab17-5df8-11ef-a427-0a580a810247, url=tcp://prod-broker-ss-0:61616
|NAME  |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL|
|      |       | COUNT  | COUNT | ADDED  |  COUNT   | ACKED  |  COUNT  | TYPE  |        |
|queuea|queuea |   0    |   2   |   2    |    0     |   0    |    0    |ANYCAST| false  |
|queueb|queueb |   0    |   1   |   1    |    0     |   0    |    0    |ANYCAST| false  |
*******************************************************************************************************************************
>>> Queue stats on node 5aefc4ad-5df8-11ef-8dd0-0a580a810248, url=tcp://prod-broker-ss-1.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616
|NAME  |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL|
|      |       | COUNT  | COUNT | ADDED  |  COUNT   | ACKED  |  COUNT  | TYPE  |        |
|queuea|queuea |   0    |   2   |   2    |    0     |   0    |    0    |ANYCAST| false  |
|queueb|queueb |   0    |   1   |   1    |    0     |   0    |    0    |ANYCAST| false  |
*******************************************************************************************************************************
>>> Queue stats on node 725cc669-5df8-11ef-a607-0a580a81024a, url=tcp://prod-broker-ss-3.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616
|NAME  |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL|
|      |       | COUNT  | COUNT | ADDED  |  COUNT   | ACKED  |  COUNT  | TYPE  |        |
|queuea|queuea |   0    |   2   |   2    |    0     |   0    |    0    |ANYCAST| false  |
|queueb|queueb |   0    |   1   |   1    |    0     |   0    |    0    |ANYCAST| false  |
*******************************************************************************************************************************
>>> Queue stats on node 65b778ee-5df8-11ef-9b5a-0a580a810249, url=tcp://prod-broker-ss-2.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616
|NAME  |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL|
|      |       | COUNT  | COUNT | ADDED  |  COUNT   | ACKED  |  COUNT  | TYPE  |        |
|queuea|queuea |   0    |   2   |   2    |    0     |   0    |    0    |ANYCAST| false  |
|queueb|queueb |   0    |   1   |   1    |    0     |   0    |    0    |ANYCAST| false  |

After scale-down 4 -> 2

2024-08-19T07:04:01.399Z DEBUG [default][ExecutorOperator:79] [mirror-dr-tests] dr-broker-ss-0 Running command: ./amq-broker/bin/artemis queue stat --user=admin --password=adminPass --maxRows=1000 --queueName=queue --maxColumnSize=-1 --url=tcp://dr-broker-ss-0:61616 --clustered
2024-08-19T07:04:04.664Z DEBUG [default][BundledArtemisClient:85] NOTE: Picked up JDK_JAVA_OPTIONS: -Dbroker.properties=/amq/extra/secrets/dr-broker-props/broker.properties
Connection brokerURL = tcp://dr-broker-ss-0:61616
2024-08-19 07:03:57,592 DEBUG [org.apache.activemq.artemis.utils.UUIDGenerator] using hardware address a:58:a:ffffff81:2:70
2024-08-19 07:03:57,875 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] Setting up call broker::getNodeID::[]
2024-08-19 07:03:57,888 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] management result:: 3aaacf73-5df8-11ef-bc87-0a580a810246
2024-08-19 07:03:57,896 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] Setting up call broker::listNetworkTopology::[]
2024-08-19 07:03:57,900 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] management result:: [{"nodeID":"8b621c8e-5df8-11ef-94eb-0a580a81024b","live":"dr-broker-ss-2:61616","primary":"dr-broker-ss-2:61616"},{"nodeID":"3aaacf73-5df8-11ef-bc87-0a580a810246","live":"dr-broker-ss-0.dr-broker-hdls-svc.mirror-dr-tests.svc.cluster.local:61616","primary":"dr-broker-ss-0.dr-broker-hdls-svc.mirror-dr-tests.svc.cluster.local:61616"},{"nodeID":"963060c1-5df8-11ef-b231-0a580a81024c","live":"dr-broker-ss-3:61616","primary":"dr-broker-ss-3:61616"},{"nodeID":"81ae9745-5df8-11ef-98d2-0a580a800224","live":"dr-broker-ss-1.dr-broker-hdls-svc.mirror-dr-tests.svc.cluster.local:61616","primary":"dr-broker-ss-1.dr-broker-hdls-svc.mirror-dr-tests.svc.cluster.local:61616"}]
*******************************************************************************************************************************
>>> Queue stats on node 3aaacf73-5df8-11ef-bc87-0a580a810246, url=tcp://dr-broker-ss-0:61616
|NAME  |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL|
|      |       | COUNT  | COUNT | ADDED  |  COUNT   | ACKED  |  COUNT  | TYPE  |        |
|queuea|queuea |   0    |   6   |   6    |    0     |   0    |    0    |ANYCAST| false  |
|queueb|queueb |   0    |   3   |   3    |    0     |   0    |    0    |ANYCAST| false  |
*******************************************************************************************************************************
>>> Queue stats on node 8b621c8e-5df8-11ef-94eb-0a580a81024b, url=tcp://dr-broker-ss-2:61616
2024-08-19 07:03:57,951 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection
java.net.UnknownHostException: dr-broker-ss-2: Name or service not known
	at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
...
Connection failed::Failed to create session factory

But sometimes (when stars align properly and order of broker pods) it's working "as expected"

2024-08-19T07:02:46.156Z DEBUG [default][ExecutorOperator:79] [mirror-prod-tests] prod-broker-ss-0 Running command: ./amq-broker/bin/artemis queue stat --user=admin --password=adminPass --maxRows=1000 --queueName=queue --maxColumnSize=-1 --url=tcp://prod-broker-ss-0:61616 --clustered
2024-08-19T07:02:49.183Z DEBUG [default][BundledArtemisClient:85] NOTE: Picked up JDK_JAVA_OPTIONS: -Dbroker.properties=/amq/extra/secrets/prod-broker-props/broker.properties
Connection brokerURL = tcp://prod-broker-ss-0:61616
2024-08-19 07:02:42,075 DEBUG [org.apache.activemq.artemis.utils.UUIDGenerator] using hardware address a:58:a:ffffff81:2:71
2024-08-19 07:02:42,360 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] Setting up call broker::getNodeID::[]
2024-08-19 07:02:42,372 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] management result:: 473bab17-5df8-11ef-a427-0a580a810247
2024-08-19 07:02:42,378 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] Setting up call broker::listNetworkTopology::[]
2024-08-19 07:02:42,382 DEBUG [org.apache.activemq.artemis.api.core.management.SimpleManagement] management result:: [{"nodeID":"473bab17-5df8-11ef-a427-0a580a810247","live":"prod-broker-ss-0.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616","primary":"prod-broker-ss-0.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616"},{"nodeID":"5aefc4ad-5df8-11ef-8dd0-0a580a810248","live":"prod-broker-ss-1.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616","primary":"prod-broker-ss-1.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616"},{"nodeID":"725cc669-5df8-11ef-a607-0a580a81024a","live":"prod-broker-ss-3:61616","primary":"prod-broker-ss-3:61616"},{"nodeID":"65b778ee-5df8-11ef-9b5a-0a580a810249","live":"prod-broker-ss-2:61616","primary":"prod-broker-ss-2:61616"}]
*******************************************************************************************************************************
>>> Queue stats on node 473bab17-5df8-11ef-a427-0a580a810247, url=tcp://prod-broker-ss-0:61616
|NAME  |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL|
|      |       | COUNT  | COUNT | ADDED  |  COUNT   | ACKED  |  COUNT  | TYPE  |        |
|queuea|queuea |   0    |   6   |   6    |    0     |   0    |    0    |ANYCAST| false  |
|queueb|queueb |   0    |   3   |   3    |    0     |   0    |    0    |ANYCAST| false  |
*******************************************************************************************************************************
>>> Queue stats on node 5aefc4ad-5df8-11ef-8dd0-0a580a810248, url=tcp://prod-broker-ss-1.prod-broker-hdls-svc.mirror-prod-tests.svc.cluster.local:61616
|NAME  |ADDRESS|CONSUMER|MESSAGE|MESSAGES|DELIVERING|MESSAGES|SCHEDULED|ROUTING|INTERNAL|
|      |       | COUNT  | COUNT | ADDED  |  COUNT   | ACKED  |  COUNT  | TYPE  |        |
|queuea|queuea |   0    |   2   |   2    |    0     |   0    |    0    |ANYCAST| false  |
|queueb|queueb |   0    |   1   |   1    |    0     |   0    |    0    |ANYCAST| false  |
*******************************************************************************************************************************
>>> Queue stats on node 725cc669-5df8-11ef-a607-0a580a81024a, url=tcp://prod-broker-ss-3:61616
2024-08-19 07:02:42,466 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection
java.net.UnknownHostException: prod-broker-ss-3: Name or service not known
	at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
...

is related to

ENTMQBR-9354 Mirroring deployment - message migration duplicates on scaledown

Backlog

relates to

ENTMQBR-9091 Broker keeps on connecting to old cluster instance after scaling down

Closed

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates