Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-9091

Broker keeps on connecting to old cluster instance after scaling down

XMLWordPrintable

      When a cluster composed of two nodes is downscaled to one single broker (i.e. one POD on OpenShift), the remaining one tries to reconnect to the other broker which has been already removed.

      Following logs is filling first pod log:

      2024-04-30 10:48:21,949 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection
      java.net.UnknownHostException: middleware-amq-redhat-ss-1: Name or service not known
          at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
          at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?]
          at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?]
          at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?]
          at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
          at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?]
          at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
          at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?]
          at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
          at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41) ~[netty-resolver-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61) ~[netty-resolver-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53) ~[netty-resolver-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55) ~[netty-resolver-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31) ~[netty-resolver-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106) ~[netty-resolver-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:220) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:189) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:175) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:990) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:516) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
      2024-04-30 10:48:21,954 WARN  [org.apache.activemq.artemis.core.server] AMQ224091: Bridge ClusterConnectionBridge@143e5ce3 [name=$.artemis.internal.sf.middleware-amq7-redhat-cluster-connection.eabfd4a1-06cc-11ef-959d-0a580afc39b4, queue=QueueImpl[name=$.artemis.internal.sf.middleware-amq7-redhat-cluster-connection.eabfd4a1-06cc-11ef-959d-0a580afc39b4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@2236d61a targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@143e5ce3 [name=$.artemis.internal.sf.middleware-amq7-redhat-cluster-connection.eabfd4a1-06cc-11ef-959d-0a580afc39b4, queue=QueueImpl[name=$.artemis.internal.sf.middleware-amq7-redhat-cluster-connection.eabfd4a1-06cc-11ef-959d-0a580afc39b4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@2236d61a targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=middleware-amq-redhat-ss-1], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1917555614[nodeUUID=2d9f7b73-fb17-11ee-bfee-0a580afc3b49, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=middleware-amq-redhat-ss-0-middleware-amq-redhat-hdls-svc-zh-swdelivery-des-svc-cluster-local, address=, server=ActiveMQServerImpl::name=amq-broker])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=middleware-amq-redhat-ss-1], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying

      Happens only with: 
      messageMigration: true
      persistenceEnabled: true
       

              gaohoward Howard Gao
              rhn-support-abelkour Mohamed Amine Belkoura
              Mikhail Krutov Mikhail Krutov
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: