Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-969

[AMQP] Test hangs when running qpid-interop-test against Artemis

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Major Major
    • None
    • None
    • amqp-protocol
    • None
    • Hide
      1. Download, build and install qpid-interop-test (https://git-wip-us.apache.org/repos/asf/qpid-interop-test.git). See QUICKSTART.md for instructions.
      2. Prepare the Artemis config file (add the pre-defined queues)
      3. Start Artemis (artemis run), note the queues deployed from the config file
      4. Set the environment for qpid-interop-test (source qpid-interop-test/build/config.sh
      5. Run the amqp_types_test, C++ shim only: python -m qpid_interop_test.amqp_types_test --include-shim ProtonCpp

      The test should run through all combinations of AMQP types and complete with a success message. However, it usually hangs on the start of the second test, occasionally, more tests will complete before it hangs.

      NOTE: There is a bug in qpid_interop_test in which the Receiver shim does not always terminate when ctl-c is used to stop the client, it will be necessary to use ps -eaf | grep Receiver to find and kill it before starting another test.

      Show
      Download, build and install qpid-interop-test ( https://git-wip-us.apache.org/repos/asf/qpid-interop-test.git ). See QUICKSTART.md for instructions. Prepare the Artemis config file (add the pre-defined queues) Start Artemis ( artemis run ), note the queues deployed from the config file Set the environment for qpid-interop-test ( source qpid-interop-test/build/config.sh Run the amqp_types_test, C++ shim only: python -m qpid_interop_test.amqp_types_test --include-shim ProtonCpp The test should run through all combinations of AMQP types and complete with a success message. However, it usually hangs on the start of the second test, occasionally, more tests will complete before it hangs. NOTE: There is a bug in qpid_interop_test in which the Receiver shim does not always terminate when ctl-c is used to stop the client, it will be necessary to use ps -eaf | grep Receiver to find and kill it before starting another test.
    • AMQ Broker 1833

      When running any of the qpid-interop-tests against the Artemis broker, it consistently works for the first test, but hangs on the second:

      [kvdr@localhost RedHat]$ python -m qpid_interop_test.amqp_types_test
      WARNING: AMQP DotNetLite shims not installed
      Test Broker: apache-activemq-artemis v.2.5.0-SNAPSHOT on <platform not found>
      
      test_binary_ProtonCpp->ProtonCpp (__main__.BinaryTestCase) ... ok
      test_binary_ProtonCpp->ProtonPython (__main__.BinaryTestCase) ...
      

      and behaves the same way for all of the interop tests.

      In all qpid-interop-tests, the client receiver is first connected followed immediately by the client sender on separate threads. This means that the client is not waiting for the broker to respond to the receiver's connection before establishing the senders connection.

      During a previous investigation of similar behavior, it was thought that the problem might lie in the auto-queue creation logic of the broker, and that pre-defining the queues would solve the problem (as a work-around). However, I have added all the queues used by qpid_interop_test.amqp_types_test to the broker.xml file (attached), and these are being deployed by the broker on startup:

      2018-01-08 11:15:56,138 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue jms.queue.qpid-interop.amqp_types_test.binary.ProtonCpp.ProtonCpp on address jms.queue.qpid-interop.amqp_types_test.binary.ProtonCpp.ProtonCpp
      2018-01-08 11:15:56,141 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue jms.queue.qpid-interop.amqp_types_test.binary.ProtonCpp.ProtonPython on address jms.queue.qpid-interop.amqp_types_test.binary.ProtonCpp.ProtonPython
      2018-01-08 11:15:56,144 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue jms.queue.qpid-interop.amqp_types_test.binary.ProtonCpp.RheaJs on address jms.queue.qpid-interop.amqp_types_test.binary.ProtonCpp.RheaJs
      2018-01-08 11:15:56,148 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue jms.queue.qpid-interop.amqp_types_test.binary.ProtonPython.ProtonCpp on address jms.queue.qpid-interop.amqp_types_test.binary.ProtonPython.ProtonCpp
      ...
      

      but appears to have no effect on this issue.

      These tests work correctly against all other brokers (ActiveMQ, Qpid-cpp and Disptach single-node).

      An analysis of the AMQP traffic for the qpid-cpp shim (see attached wireshark capture) shows that in each case, on the second test (or occasionally a few more tests will complete successfully before the hang), after the receiver sends its AMQP flow performative, the broker stops responding. It should at this point start transferring messages to the receiver.

      In the wireshark file attached, this can be clearly seen using only the Proton C++ shim. In this test, the first four tests succeeded, and the fifth hung. Looking at the fifth test (starting in frame 176):

      Frame 176: Sender connects to broker with AMQP open, begin, attach
      Frame 181: Receiver connects to broker with AMQP open, begin, attach
      Frame 183: Broker responds to sender with AMQP open
      Frame 185: Broker responds to sender with AMQP begin, attach, flow
      Frame 187: Sender sends 4 test messages to broker (AMQP transfer x4)
      Frame 188: Broker sends 4 dispositions (AMQP disposition x4)
      Frame 189: Sender closes (AMQP close)
      Frame 190: Broker closes (AMQP close)
      The traffic between the sender and broker appears normal. However, the receiver traffic from frame 181 has not yet been responded to in any way.
      Frame 194: Broker responds to receiver with AMQP open
      Frame 196: Broker responds to receiver with AMQP begin, attach
      Frame 198: Receiver sends broker AMQP flow
      The receipt of flow should allow the broker to start the transfer of its messages, but instead, it does not produce any further traffic.

      If I use the PN_TRACE_FRM=1 environment variable when running Artemis, I can see that the AMQP flow from the receiver to the broker is the last AMQP packet received at a Proton level.

        1. 01.artemis.cpp_client_only.ignore_flow.pcapng
          48 kB
          Kim van der Riet
        2. broker.xml
          50 kB
          Kim van der Riet

              mtaylor1@redhat.com Martyn Taylor (Inactive)
              kvanderr@redhat.com Kim van der Riet
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Estimated:
                  Original Estimate - 2 days
                  2d
                  Remaining:
                  Remaining Estimate - 2 days
                  2d
                  Logged:
                  Time Spent - Not Specified
                  Not Specified