Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-11518

[7.0] Messaging - max-size-bytes is not respected leaving server to crash on OutOfMemoryError

    XMLWordPrintable

Details

    • Hide

      Steps to reproduce:

      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout master

groovy -DEAP_VERSION=7.1.0.DR19 PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/
mvn clean test -Dtest=MessagePerformanceTestCase#testOOMEWithBigJournal -DfailIfNoTests=false -Deap=7x  | tee log
      Show
      Steps to reproduce: git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout master groovy -DEAP_VERSION=7.1.0.DR19 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=MessagePerformanceTestCase#testOOMEWithBigJournal -DfailIfNoTests= false -Deap=7x | tee log

    Description

      Customer Impact: Server can crash on OutOfMemoryError as maximum amount of memory allocated for each queue (defined by max-size-bytes attribute) is not respected. For administrator is not possible to estimate how much memory server can consume.

      Test Scenario:

      • Start EAP 7.1.0.DR19 with 2GB heap (-Xmx2g)
        • max-size-bytes is set to 1GB to all destinations
      • Send 1 million of 1KB messages to queue InQueue (so ~1GB of messages)

      Result: Server crashes on OOME during sending of messages.

      Server crashed on OOME when ~659k messages were sent which should correspond to ~659 MB of messages in heap. Size of Artemis journal is 831 MB. Paging did not trigger so all messages are in "journal" directory. If server is started with larger heap (-Xmx3g) then server starts but consumes slightly more than 2 GB of heap after loading journal.

      So every message in heap consumes more than double of it's size which is not the biggest issue. Worse thing is that max-size-bytes which is 1GB did not start paging when messages in memory for one queue exceeded 1GB of memory and let server crash on OutOfMemoryError.

      There are 2 things to fix:

      • trigger paging/block/fail/drop mode when memory allocated by one queue exceeded max-size-bytes
      • why messages consumes so much of memory (more than double of their payload)

      This is not regression.

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-11264 [7.1] Messaging - max-size-bytes is not respected leaving server to crash on OutOfMemoryError

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          relates to

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-11192 Random OutOfMemory in soak

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Activity

            People

              sgilda_jira Sande Gilda (Inactive)
              sgilda_jira Sande Gilda (Inactive)
              Miroslav Novak Miroslav Novak
              Miroslav Novak Miroslav Novak
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: