Uploaded image for project: 'JBoss Enterprise Application Platform 4 and 5'
  1. JBoss Enterprise Application Platform 4 and 5
  2. JBPAPP-6522

HornetQ core bridge will hang when target server is killed while messages are being delivered

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • EAP_EWP 5.1.2 CR2
    • EAP_EWP 5.1.0
    • HornetQ
    • JBoss-EAP-5.1-GA, HornetQ build : 2.2.3.GA (HQ_2_2_3_GA_EAP, 121)

    • Hide
      • Setup two server, ServerA (source) and ServerB (target).
      • Create a queue on ServerA, queue/A at hornetq-jmx.xml :
        <queue name="A">
        <entry name="/queue/A"/>
        </queue>
      • Create an address on ServerA, at hornetq-configuration.xml :
        <queues>
        <queue name="jms.queue.A">
        <address>jms.queue.A</address>
        </queue>
        </queues>
      • Set destination full policy to PAGING at both servers at hornetq-configuration.xml :
        <address-settings>
        <address-setting match="#">
        <dead-letter-address>jms.queue.DLQ</dead-letter-address>
        <expiry-address>jms.queue.ExpiryQueue</expiry-address>
        <redelivery-delay>0</redelivery-delay>
        <max-size-bytes>-1</max-size-bytes>
        <message-counter-history-day-limit>10</message-counter-history-day-limit>
        <page-size-bytes>104857600</page-size-bytes>
        <address-full-policy>PAGE</address-full-policy>
        </address-setting>
        </address-settings>
      • Create "connector" on ServerA at hornetq-configuration.xml, which is pointed towards the ServerB :
        <connector name="remote">
        <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
        <param key="host" value="192.168.0.8"/>
        <param key="port" value="5445"/>
        </connector>
      • Create a bridge on the hornetq-configuration.xml at ServerA :
        <bridges>
        <bridge name="second-bridge">
        <queue-name>jms.queue.A</queue-name>
        <forwarding-address>jms.queue.B</forwarding-address>
        <retry-interval>1000</retry-interval>
        <retry-interval-multiplier>1.0</retry-interval-multiplier>
        <reconnect-attempts>-1</reconnect-attempts>
        <use-duplicate-detection>true</use-duplicate-detection>
        <static-connectors>
        <connector-ref>remote</connector-ref>
        </static-connectors>
        </bridge>
        </bridges>
      • Create a queue, queue/B on ServerB at hornetq-jms.xml :
        <queue name="A">
        <entry name="/queue/A"/>
        </queue>
      • Create an address on ServerB, at hornetq-configuration.xml :
        <queues>
        <queue name="jms.queue.B">
        <address>jms.queue.B</address>
        </queue>
        </queues>
      • Start ServerB then ServerA
      • Send ~5 bytemessages to destination queue/A at ServerA
      • Make sure the core-bridge is delivering these messages by opening up the "data/large-messages". You could see the files are growing as the messages are being delivered by the core-bridge.
      • Kill the ServerB ("target" server) using kill -9 <pid>
      • Start the ServerB again. Nothing happens at this stage.
      • Try sending textmessages over this bridge, they won't get delivered. You could also notice "data/large-messages" is not growing. Leave the servers idle for hours, the bridge would remain stuck.
      • The core bridge remains stuck until ServerA is restarted.
      • The core bridge starts moving messages from ServerA to ServerN once it's restarted, however partial messages would remain at the "data/large-messages" directory at ServerB.
      Show
      Setup two server, ServerA (source) and ServerB (target). Create a queue on ServerA, queue/A at hornetq-jmx.xml : <queue name="A"> <entry name="/queue/A"/> </queue> Create an address on ServerA, at hornetq-configuration.xml : <queues> <queue name="jms.queue.A"> <address>jms.queue.A</address> </queue> </queues> Set destination full policy to PAGING at both servers at hornetq-configuration.xml : <address-settings> <address-setting match="#"> <dead-letter-address>jms.queue.DLQ</dead-letter-address> <expiry-address>jms.queue.ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <max-size-bytes>-1</max-size-bytes> <message-counter-history-day-limit>10</message-counter-history-day-limit> <page-size-bytes>104857600</page-size-bytes> <address-full-policy>PAGE</address-full-policy> </address-setting> </address-settings> Create "connector" on ServerA at hornetq-configuration.xml, which is pointed towards the ServerB : <connector name="remote"> <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class> <param key="host" value="192.168.0.8"/> <param key="port" value="5445"/> </connector> Create a bridge on the hornetq-configuration.xml at ServerA : <bridges> <bridge name="second-bridge"> <queue-name>jms.queue.A</queue-name> <forwarding-address>jms.queue.B</forwarding-address> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1.0</retry-interval-multiplier> <reconnect-attempts>-1</reconnect-attempts> <use-duplicate-detection>true</use-duplicate-detection> <static-connectors> <connector-ref>remote</connector-ref> </static-connectors> </bridge> </bridges> Create a queue, queue/B on ServerB at hornetq-jms.xml : <queue name="A"> <entry name="/queue/A"/> </queue> Create an address on ServerB, at hornetq-configuration.xml : <queues> <queue name="jms.queue.B"> <address>jms.queue.B</address> </queue> </queues> Start ServerB then ServerA Send ~5 bytemessages to destination queue/A at ServerA Make sure the core-bridge is delivering these messages by opening up the "data/large-messages". You could see the files are growing as the messages are being delivered by the core-bridge. Kill the ServerB ("target" server) using kill -9 <pid> Start the ServerB again. Nothing happens at this stage. Try sending textmessages over this bridge, they won't get delivered. You could also notice "data/large-messages" is not growing. Leave the servers idle for hours, the bridge would remain stuck. The core bridge remains stuck until ServerA is restarted. The core bridge starts moving messages from ServerA to ServerN once it's restarted, however partial messages would remain at the "data/large-messages" directory at ServerB.
    • Release Notes, Compatibility/Configuration
    • Hide
      It was discovered that the HornetQ core bridge service was hanging if the target server was terminated while large *bytemessages* of approximately 270MB were being delivered. The contents of the failed messages were not cleaned up from the target server's <filename>data/large-messages</filename> directory.</para><para>When the server was terminated more than once during large *bytemessage* transmission, multiple files would persist in the<filename>data/large-messages</filename> directory. The only way to correct this issue was to restart the server that the bridge service was deployed on.</para> <para>The fix implements changes to the core bridge, which now restarts until it successfully reconnects to the target server. The bridge no longer hangs when the target server is terminated, which corrects the issue.
      Show
      It was discovered that the HornetQ core bridge service was hanging if the target server was terminated while large *bytemessages* of approximately 270MB were being delivered. The contents of the failed messages were not cleaned up from the target server's <filename>data/large-messages</filename> directory.</para><para>When the server was terminated more than once during large *bytemessage* transmission, multiple files would persist in the<filename>data/large-messages</filename> directory. The only way to correct this issue was to restart the server that the bridge service was deployed on.</para> <para>The fix implements changes to the core bridge, which now restarts until it successfully reconnects to the target server. The bridge no longer hangs when the target server is terminated, which corrects the issue.
    • Documented as Resolved Issue
    • ON_QA

    Description

      HornetQ core bridge gets hang if you kill the target server while large bytemessages are being delivered. The size of the file is around 270MB across a stable LAN. Actually there are two issues occur when the target server is killed :

      • The bridge service hangs. You have to restart the "source" server, where the bridge service is deployed to get it to work.
      • The contents of the failed messages never get cleaned up from the "target" server's "data/large-messages" directory. If you kill the server a few times, you could see a number of files at the "data/large-messages".

      Attachments

        Activity

          People

            csuconic@redhat.com Clebert Suconic
            rhn-support-tywickra Tyronne Wickramarathne
            Jared Morgan Jared Morgan (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: