Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-3376

AMQP Journal loading is triggering reencode

    XMLWordPrintable

Details

    • User Experience
    • +
    • Verified in a release
    • Hide
      • download and unzip the data folder in the attachment to replace the existing (empty) data folder of a new broker instance
      • replace the the existing GC configuration on artemis.profile with:
      # Java Opts
JAVA_ARGS=" -XX:+PrintClassHistogram -XX:+UseG1GC -Xms4G -Xmx4G -XX:+AlwaysPreTouch -Dhawtio.realm=activemq  -Dhawtio.offline=true -Dhawtio.role=amq -Dhawtio.rolePrincipalClasses=org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal -Djolokia.policyLocation=${ARTEMIS_INSTANCE_ETC_URI}jolokia-access.xml -Djon.id=amq"

      The important bits are -XX:+UseG1GC -Xms4G -Xmx4G -XX:+AlwaysPreTouch

      • start the broker
      • it goes OOM with a log similar to
      Exception in thread "Thread-0 (-scheduled-threads)" Exception in thread "ActiveMQ Artemis Server Shutdown Timer" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
2020-03-18 10:50:08,680 ERROR [org.apache.activemq.artemis.core.server] AMQ224097: Failed to start server: java.lang.OutOfMemoryError: Java heap space
      Show
      download and unzip the data folder in the attachment to replace the existing (empty) data folder of a new broker instance replace the the existing GC configuration on artemis.profile with: # Java Opts JAVA_ARGS= " -XX:+PrintClassHistogram -XX:+UseG1GC -Xms4G -Xmx4G -XX:+AlwaysPreTouch -Dhawtio.realm=activemq -Dhawtio.offline= true -Dhawtio.role=amq -Dhawtio.rolePrincipalClasses=org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal -Djolokia.policyLocation=${ARTEMIS_INSTANCE_ETC_URI}jolokia-access.xml -Djon.id=amq" The important bits are -XX:+UseG1GC -Xms4G -Xmx4G -XX:+AlwaysPreTouch start the broker it goes OOM with a log similar to Exception in thread " Thread -0 (-scheduled-threads)" Exception in thread "ActiveMQ Artemis Server Shutdown Timer" java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space 2020-03-18 10:50:08,680 ERROR [org.apache.activemq.artemis.core.server] AMQ224097: Failed to start server: java.lang.OutOfMemoryError: Java heap space

    Description

      The journal loading is seeing AMQP messages to be fully decoded while they shouldn't not; it seems causes by AMQPStandardMessage:: reloadPersistence:

         @Override
   public void reloadPersistence(ActiveMQBuffer record, CoreMessageObjectPools pools) {
      int size = record.readInt();
      byte[] recordArray = new byte[size];
      record.readBytes(recordArray);
      data = ReadableBuffer.ByteBufferReader.wrap(ByteBuffer.wrap(recordArray));

      // Message state is now that the underlying buffer is loaded, but the contents not yet scanned
      resetMessageData();
      modified = false;
      messageDataScanned = RELOAD_PERSISTENCE;
      // can happen when moved to a durable location.  We must re-encode here to
      // avoid a subsequent redelivery from suddenly appearing with a durable header
      // tag when the initial delivery did not.
      if (!isDurable()) {
         setDurable(true);
         reencode();
      }
   }

      Specifically by:

            if (!isDurable()) {
         setDurable(true);
         reencode();
      }

      Due to this, the journal loading of AMQP message is producing a big amount of garbage, leading to multi-seconds GC full GC pauses that could affect both HA and the user experience.

      Attachments

        Issue Links

          is caused by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. ENTMQBR-2883 Implement streaming of Large Messages over AMQP

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. ENTMQBR-2313 Fails to load journal after queue full

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              fnigro Francesco Nigro
              fnigro Francesco Nigro
              Tiago Bueno Tiago Bueno
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 day
                  1d