Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-3376

AMQP Journal loading is triggering reencode

XMLWordPrintable

    • User Experience
    • +
    • Verified in a release
    • Hide
      • download and unzip the data folder in the attachment to replace the existing (empty) data folder of a new broker instance
      • replace the the existing GC configuration on artemis.profile with:
      # Java Opts
JAVA_ARGS=" -XX:+PrintClassHistogram -XX:+UseG1GC -Xms4G -Xmx4G -XX:+AlwaysPreTouch -Dhawtio.realm=activemq  -Dhawtio.offline=true -Dhawtio.role=amq -Dhawtio.rolePrincipalClasses=org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal -Djolokia.policyLocation=${ARTEMIS_INSTANCE_ETC_URI}jolokia-access.xml -Djon.id=amq"

      The important bits are -XX:+UseG1GC -Xms4G -Xmx4G -XX:+AlwaysPreTouch

      • start the broker
      • it goes OOM with a log similar to
      Exception in thread "Thread-0 (-scheduled-threads)" Exception in thread "ActiveMQ Artemis Server Shutdown Timer" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
2020-03-18 10:50:08,680 ERROR [org.apache.activemq.artemis.core.server] AMQ224097: Failed to start server: java.lang.OutOfMemoryError: Java heap space
      Show
      download and unzip the data folder in the attachment to replace the existing (empty) data folder of a new broker instance replace the the existing GC configuration on artemis.profile with: # Java Opts JAVA_ARGS= " -XX:+PrintClassHistogram -XX:+UseG1GC -Xms4G -Xmx4G -XX:+AlwaysPreTouch -Dhawtio.realm=activemq -Dhawtio.offline= true -Dhawtio.role=amq -Dhawtio.rolePrincipalClasses=org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal -Djolokia.policyLocation=${ARTEMIS_INSTANCE_ETC_URI}jolokia-access.xml -Djon.id=amq" The important bits are -XX:+UseG1GC -Xms4G -Xmx4G -XX:+AlwaysPreTouch start the broker it goes OOM with a log similar to Exception in thread " Thread -0 (-scheduled-threads)" Exception in thread "ActiveMQ Artemis Server Shutdown Timer" java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space 2020-03-18 10:50:08,680 ERROR [org.apache.activemq.artemis.core.server] AMQ224097: Failed to start server: java.lang.OutOfMemoryError: Java heap space

      The journal loading is seeing AMQP messages to be fully decoded while they shouldn't not; it seems causes by AMQPStandardMessage:: reloadPersistence:

         @Override
   public void reloadPersistence(ActiveMQBuffer record, CoreMessageObjectPools pools) {
      int size = record.readInt();
      byte[] recordArray = new byte[size];
      record.readBytes(recordArray);
      data = ReadableBuffer.ByteBufferReader.wrap(ByteBuffer.wrap(recordArray));

      // Message state is now that the underlying buffer is loaded, but the contents not yet scanned
      resetMessageData();
      modified = false;
      messageDataScanned = RELOAD_PERSISTENCE;
      // can happen when moved to a durable location.  We must re-encode here to
      // avoid a subsequent redelivery from suddenly appearing with a durable header
      // tag when the initial delivery did not.
      if (!isDurable()) {
         setDurable(true);
         reencode();
      }
   }

      Specifically by:

            if (!isDurable()) {
         setDurable(true);
         reencode();
      }

      Due to this, the journal loading of AMQP message is producing a big amount of garbage, leading to multi-seconds GC full GC pauses that could affect both HA and the user experience.

        1. data_amqp.tar.xz
          47.92 MB
          Francesco Nigro

              fnigro Francesco Nigro
              fnigro Francesco Nigro
              Tiago Bueno Tiago Bueno
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 day
                  1d