Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-9091

Broker keeps on connecting to old cluster instance after scaling down

XMLWordPrintable

      When a cluster composed of two nodes is downscaled to one single broker (i.e. one POD on OpenShift), the remaining one tries to reconnect to the other broker which has been already removed.

      Following logs is filling first pod log:

      2024-04-30 10:48:21,949 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection
      java.net.UnknownHostException: middleware-amq-redhat-ss-1: Name or service not known
          at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
          at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?]
          at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?]
          at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?]
          at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
          at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?]
          at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
          at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?]
          at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
          at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41) ~[netty-resolver-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61) ~[netty-resolver-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53) ~[netty-resolver-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55) ~[netty-resolver-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31) ~[netty-resolver-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106) ~[netty-resolver-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:220) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:189) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:175) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:990) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:516) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) ~[netty-transport-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.100.Final-redhat-00001.jar:4.1.100.Final-redhat-00001]
          at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
      2024-04-30 10:48:21,954 WARN  [org.apache.activemq.artemis.core.server] AMQ224091: Bridge ClusterConnectionBridge@143e5ce3 [name=$.artemis.internal.sf.middleware-amq7-redhat-cluster-connection.eabfd4a1-06cc-11ef-959d-0a580afc39b4, queue=QueueImpl[name=$.artemis.internal.sf.middleware-amq7-redhat-cluster-connection.eabfd4a1-06cc-11ef-959d-0a580afc39b4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@2236d61a targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@143e5ce3 [name=$.artemis.internal.sf.middleware-amq7-redhat-cluster-connection.eabfd4a1-06cc-11ef-959d-0a580afc39b4, queue=QueueImpl[name=$.artemis.internal.sf.middleware-amq7-redhat-cluster-connection.eabfd4a1-06cc-11ef-959d-0a580afc39b4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@2236d61a targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=middleware-amq-redhat-ss-1], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1917555614[nodeUUID=2d9f7b73-fb17-11ee-bfee-0a580afc3b49, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=middleware-amq-redhat-ss-0-middleware-amq-redhat-hdls-svc-zh-swdelivery-des-svc-cluster-local, address=, server=ActiveMQServerImpl::name=amq-broker])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=middleware-amq-redhat-ss-1], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying

      Happens only with: 
      messageMigration: true
      persistenceEnabled: true
       

            gaohoward Howard Gao
            rhn-support-abelkour Mohamed Amine Belkoura
            Mikhail Krutov Mikhail Krutov
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: