Uploaded image for project: 'Cloud Enablement'
  1. Cloud Enablement
  2. CLOUD-2934

[AMQ7] after message migration broker log is periodically spammed with UnknownHost exception

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • AMQ72 1.1.0.GA
    • AMQ72 1.0.0.GA
    • AMQ7
    • None
    • Hide

      1. Deploy broker cluster with 2 replicas
      2. Deploy scaledown controller
      3. Scale down to 1 replica
      4. After migration monitor the broker-0 log -> observer the log spam
      5. Scale up to 2 replicas -> observer the broker-1 log shows 2 cluster connections

      Show
      1. Deploy broker cluster with 2 replicas 2. Deploy scaledown controller 3. Scale down to 1 replica 4. After migration monitor the broker-0 log -> observer the log spam 5. Scale up to 2 replicas -> observer the broker-1 log shows 2 cluster connections

      After the scale down and up procedure, there seems to be a leftover reference to a draining broker pod that isn't reachable anymore.

      The log is flooded with UnknownHost exception with reference to broker-1 that was actually a draining pod.

      I've tried to collect logs from all phases.

      From the broker-0 logs it can be observed that despite the UnknowHost exception after scale-up new cluster connection is formed successfully. However, it's hard to spot from e.g. tailing a log via oc logs -f.

      There're broker-1 logs from before, int, after phase (before scale down, intermediate draining, after scale up).

      Notice that after scale up broker-1 reports 2 cluster connections instead of 1.

      2018-10-10 13:07:38,982 INFO  [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.6.3.redhat-00004 [broker, nodeID=73e3486f-cc8d-11e8-be93-0a580a830141] 
2018-10-10 13:07:39,234 INFO  [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge@3bfa875b [name=$.artemis.internal.sf.my-cluster.cd8fa160-cc8c-11e8-b394-0a580a810189, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.cd8fa160-cc8c-11e8-b394-0a580a810189, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=73e3486f-cc8d-11e8-be93-0a580a830141], temp=false]@72c48ccc targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@3bfa875b [name=$.artemis.internal.sf.my-cluster.cd8fa160-cc8c-11e8-b394-0a580a810189, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.cd8fa160-cc8c-11e8-b394-0a580a810189, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=73e3486f-cc8d-11e8-be93-0a580a830141], temp=false]@72c48ccc targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=broker-0-broker-amq-headless-dsimansk-amq-svc-cluster-local], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1631227617[nodeUUID=73e3486f-cc8d-11e8-be93-0a580a830141, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=broker-1-broker-amq-headless-dsimansk-amq-svc-cluster-local, address=, server=ActiveMQServerImpl::serverUUID=73e3486f-cc8d-11e8-be93-0a580a830141])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=broker-0-broker-amq-headless-dsimansk-amq-svc-cluster-local], discoveryGroupConfiguration=null]] is connected
2018-10-10 13:07:39,314 INFO  [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge@75ae581f [name=$.artemis.internal.sf.my-cluster.de43c141-cc8c-11e8-8b54-0a580a820163, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.de43c141-cc8c-11e8-8b54-0a580a820163, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=73e3486f-cc8d-11e8-be93-0a580a830141], temp=false]@269057c targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@75ae581f [name=$.artemis.internal.sf.my-cluster.de43c141-cc8c-11e8-8b54-0a580a820163, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.de43c141-cc8c-11e8-8b54-0a580a820163, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=73e3486f-cc8d-11e8-be93-0a580a830141], temp=false]@269057c targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=broker-1], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1631227617[nodeUUID=73e3486f-cc8d-11e8-be93-0a580a830141, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=broker-1-broker-amq-headless-dsimansk-amq-svc-cluster-local, address=, server=ActiveMQServerImpl::serverUUID=73e3486f-cc8d-11e8-be93-0a580a830141])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=broker-1], discoveryGroupConfiguration=null]] is connected

        1. broker-0-spam.log
          1.41 MB
        2. broker-0-spam-after.log
          2.65 MB
        3. broker-1-after.log
          12 kB
        4. broker-1-before.log
          10 kB
        5. broker-1-int.log
          42 kB

              rhn-support-rkieley Roderick Kieley
              dsimansk@redhat.com David Simansky
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: