Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-2180

SSL NetworkConnector Blocks Shutdown when Both Sides of Network Failover Simultaneously

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Critical Critical
    • None
    • JBoss A-MQ 6.3
    • broker, networks
    • None
    • Hide

      (Work in Progress)

      Set up two brokers with tcp+ssl transports for broker to broker connections, JDBC persistence adapters and each with their own schema in the database

      Clients are XA-transacted MDBs running in JBoss EAP (A servlet is used to produce a message load)

      Set up a load in both directions by producing and consuming from the same queue on both sides of the NOB.

      While production is in-progress, stop the RDBMS to force locker failures on both sides of the NOB at once.

      Sample Transport and NOB config:

              <sslContext>
                  <sslContext
      		 keyStore="/opt/app/SSL/test.keystore"
                       keyStorePassword="${keystore.password}"
      		 trustStore="/opt/app/SSL/cacerts.jks"
                       trustStorePassword="${truststore.password}" />
              </sslContext>
      
              <transportConnectors>
                  <transportConnector name="nio+ssl" uri="nio+ssl://0.0.0.0:61617?transport.enabledProtocols=TLSv1.2&amp;maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
                 <transportConnector name="ssl2" uri="ssl://0.0.0.0:61618?transport.enabledProtocols=TLSv1.2&amp;maximumConnections=200&amp;wireFormat.maxFrameSize=104857600&amp;transport.soTimeout=60000&amp;transport.soLinger=0"/>
              </transportConnectors>
      
      	<networkConnectors>
       	 <networkConnector name="NETWORK_TO_PAIR_2"
       	 uri="masterslave:(ssl://amq03.example.com:61618?soWriteTimeout=20000,ssl://amq04.example.com:61618?soWriteTimeout=20000)"
       	 networkTTL="2"
       	 conduitSubscriptions="false"
       	 suppressDuplicateQueueSubscriptions="true"
       	 userName="BrokerGroup"
       	 password="${brokernetworking.password}" />
       	</networkConnectors>
      
      Show
      (Work in Progress) Set up two brokers with tcp+ssl transports for broker to broker connections, JDBC persistence adapters and each with their own schema in the database Clients are XA-transacted MDBs running in JBoss EAP (A servlet is used to produce a message load) Set up a load in both directions by producing and consuming from the same queue on both sides of the NOB. While production is in-progress, stop the RDBMS to force locker failures on both sides of the NOB at once. Sample Transport and NOB config: <sslContext> <sslContext keyStore= "/opt/app/SSL/test.keystore" keyStorePassword= "${keystore.password}" trustStore= "/opt/app/SSL/cacerts.jks" trustStorePassword= "${truststore.password}" /> </sslContext> <transportConnectors> <transportConnector name= "nio+ssl" uri= "nio+ssl://0.0.0.0:61617?transport.enabledProtocols=TLSv1.2&amp;maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600" /> <transportConnector name= "ssl2" uri= "ssl://0.0.0.0:61618?transport.enabledProtocols=TLSv1.2&amp;maximumConnections=200&amp;wireFormat.maxFrameSize=104857600&amp;transport.soTimeout=60000&amp;transport.soLinger=0" /> </transportConnectors> <networkConnectors> <networkConnector name= "NETWORK_TO_PAIR_2" uri= "masterslave:(ssl://amq03.example.com:61618?soWriteTimeout=20000,ssl://amq04.example.com:61618?soWriteTimeout=20000)" networkTTL= "2" conduitSubscriptions= "false" suppressDuplicateQueueSubscriptions= "true" userName= "BrokerGroup" password= "${brokernetworking.password}" /> </networkConnectors>

      In testing a simple network consisting of two brokers with JDBC persistence stores, each broker with an ssl broker to broker transport and a half-duplex network connector to the other node, we found that if we force both brokers to lose their lock while under a load (for example by shutting down the RDBMS feeding both broker nodes), we can get a deadlock on the transport with both sides trying to write to the other and the SSLSocketImpl holding one lock and trying to get a lock to write a close_notify. This deadlock persists until the brokers are forcibly shut down with warnings in the log on one side like:

      07:44:06,477 | INFO  | erService[amq01] | TransportConnection              | 162 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630310-03 | The connection to 'tcp://xx.xxx.xxx.165:46870' is taking a long time to shutdown.
      

      and on the other side, like:

      10:18:06,877 | ERROR | [amq04] Task-772 | TransportConnector               | 162 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630310-03 | Could not accept connection from tcp://xx.xxx.xxx.228:40754: org.apache.activemq.broker.BrokerStoppedException: Broker BrokerService[amq04] is being stopped
      

        1. tdump02-1.txt
          115 kB
          Gary Tully
        2. tdump04-1.txt
          133 kB
          Gary Tully

              gtully@redhat.com Gary Tully
              rhn-support-dhawkins Duane Hawkins
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: