Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-2038

After message migration broker log is periodically spammed with UnknownHost exception

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • AMQ 7.2.1 Image 1.0.GA
    • container image
    • None
    • Hide

      1. Deploy broker cluster with 2 replicas
      2. Deploy scaledown controller
      3. Scale down to 1 replica
      4. After migration monitor the broker-0 log -> observer the log spam
      5. Scale up to 2 replicas -> observer the broker-1 log shows 2 cluster connections

      Show
      1. Deploy broker cluster with 2 replicas 2. Deploy scaledown controller 3. Scale down to 1 replica 4. After migration monitor the broker-0 log -> observer the log spam 5. Scale up to 2 replicas -> observer the broker-1 log shows 2 cluster connections

    Description

      After the scale down and up procedure, there seems to be a leftover reference to a draining broker pod that isn't reachable anymore.

      The log is flooded with UnknownHost exception with reference to broker-1 that was actually a draining pod.

      I've tried to collect logs from all phases.

      From the broker-0 logs it can be observed that despite the UnknowHost exception after scale-up new cluster connection is formed successfully. However, it's hard to spot from e.g. tailing a log via oc logs -f.

      There're broker-1 logs from before, int, after phase (before scale down, intermediate draining, after scale up).

      Notice that after scale up broker-1 reports 2 cluster connections instead of 1.

      2018-10-10 13:07:38,982 INFO  [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.6.3.redhat-00004 [broker, nodeID=73e3486f-cc8d-11e8-be93-0a580a830141] 
2018-10-10 13:07:39,234 INFO  [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge@3bfa875b [name=$.artemis.internal.sf.my-cluster.cd8fa160-cc8c-11e8-b394-0a580a810189, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.cd8fa160-cc8c-11e8-b394-0a580a810189, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=73e3486f-cc8d-11e8-be93-0a580a830141], temp=false]@72c48ccc targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@3bfa875b [name=$.artemis.internal.sf.my-cluster.cd8fa160-cc8c-11e8-b394-0a580a810189, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.cd8fa160-cc8c-11e8-b394-0a580a810189, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=73e3486f-cc8d-11e8-be93-0a580a830141], temp=false]@72c48ccc targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=broker-0-broker-amq-headless-dsimansk-amq-svc-cluster-local], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1631227617[nodeUUID=73e3486f-cc8d-11e8-be93-0a580a830141, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=broker-1-broker-amq-headless-dsimansk-amq-svc-cluster-local, address=, server=ActiveMQServerImpl::serverUUID=73e3486f-cc8d-11e8-be93-0a580a830141])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=broker-0-broker-amq-headless-dsimansk-amq-svc-cluster-local], discoveryGroupConfiguration=null]] is connected
2018-10-10 13:07:39,314 INFO  [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge@75ae581f [name=$.artemis.internal.sf.my-cluster.de43c141-cc8c-11e8-8b54-0a580a820163, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.de43c141-cc8c-11e8-8b54-0a580a820163, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=73e3486f-cc8d-11e8-be93-0a580a830141], temp=false]@269057c targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@75ae581f [name=$.artemis.internal.sf.my-cluster.de43c141-cc8c-11e8-8b54-0a580a820163, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.de43c141-cc8c-11e8-8b54-0a580a820163, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=73e3486f-cc8d-11e8-be93-0a580a830141], temp=false]@269057c targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=broker-1], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1631227617[nodeUUID=73e3486f-cc8d-11e8-be93-0a580a830141, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=broker-1-broker-amq-headless-dsimansk-amq-svc-cluster-local, address=, server=ActiveMQServerImpl::serverUUID=73e3486f-cc8d-11e8-be93-0a580a830141])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=broker-1], discoveryGroupConfiguration=null]] is connected

      Attachments

        1. broker-0-spam.log
          1.41 MB
        2. broker-0-spam-after.log
          2.65 MB
        3. broker-1-after.log
          12 kB
        4. broker-1-before.log
          10 kB
        5. broker-1-int.log
          42 kB

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. CLOUD-2934 [AMQ7] after message migration broker log is periodically spammed with UnknownHost exception

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is blocked by

          Bug - A problem that impairs or prevents the functions of the product. ENTMQBR-2042 Cluster connection non-default configuration elements ignored

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed

          Activity

            People

              rhn-support-rkieley Roderick Kieley
              rhn-support-rkieley Roderick Kieley
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: