-
Bug
-
Resolution: Won't Do
-
Minor
-
None
-
AMQ 7.10.0.GA
-
None
-
False
-
None
-
False
When using Artemis CLI tools version 7.9.4 with AMQ 7.10 broker deploy in OCP occasionally they fail to perform a failover and clinet insist on connection to a broker that does not exist any more instead of failing over.
Received text sized at 17 2022-09-09 14:36:25,719 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[1] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:36:26,014 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[2] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:36:26,321 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[3] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:36:26,640 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[4] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:36:26,969 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[5] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:36:27,409 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[6] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:36:27,981 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[7] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:36:28,875 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[8] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:36:30,434 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[9] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:36:33,243 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[10] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:36:33,244 WARN [org.apache.qpid.jms.provider.failover.FailoverProvider] Failed to connect after: 10 attempt(s) continuing to retry. 2022-09-09 14:36:38,642 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[11] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:36:49,165 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[12] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:37:09,931 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[13] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:37:40,238 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[14] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:38:10,484 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[15] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:38:40,743 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[16] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:39:11,094 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[17] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:39:41,363 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[18] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:40:11,653 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[19] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:40:41,930 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[20] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:40:41,931 WARN [org.apache.qpid.jms.provider.failover.FailoverProvider] Failed to connect after: 20 attempt(s) continuing to retry. 2022-09-09 14:41:12,204 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[21] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:41:42,558 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[22] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed 2022-09-09 14:42:12,836 INFO [org.apache.qpid.jms.provider.failover.FailoverProvider] Connection attempt:[23] to: amqps://amq-broker-artemis-2-svc-rte-amq-broker.apps.toross-03300984.emeashift.support:443 failed
I could not reproduce the issue when using AMQ 7.10 CLI tools with AMQ 7.10.
I've attached two shell scripts that generate and consume messages and broker CR yaml file.
To reproduce the issue:
- deploy a AMQ 7.10 cluster of 3 brokers in OCP
- start message producer and message consumer as per attached shell scripts
- scaledown the cluster from 3 broker s to 1 by replaying the broker CR with new replicas count. One of the clients would get stuck as in the above log fragment.
The does not reproduce every time and only with AMQ 7.9.4 CLI tools and AMQ 7.10 broker.
