Uploaded image for project: 'Hawkular'
  1. Hawkular
  2. HAWKULAR-1130

Internal server errors under heavy load because of high GC activity

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 1.0.0.Alpha15
    • None
    • None
    • None
    • 0.0.17.Final-SNAPSHOT (commit ba27915a15d99ea06abf8087307f7f4996eda423)

      1. start cassandra node
      2. start hawkular services
      3. generate load (POST requests to hawkular metrics)

      Hawkular services JVM is spending all the time on Full GC under heavy load so some requests fail with 500 response because of timeout issues.

      Performance test is hitting hawkular metrics with POST requests. There is no other activity.

      I didn't search for exact commit but it was definitely caused by something after Oct 5. Build from Oct 5 passed.

      Here is thread dump - http://web.bc.jonqe.lab.eng.bos.redhat.com/jstack-issue.out
      Here is heap dump - http://web.bc.jonqe.lab.eng.bos.redhat.com/haw_ser_dump_issue.bin
      All logs including GC log are attached.
      I used just 512MB for heap in this test to reproduce it faster but it was visible even with 4GB for heap.

              Unassigned Unassigned
              fbrychta@redhat.com Filip Brychta
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: