Uploaded image for project: 'Hawkular'
  1. Hawkular
  2. HAWKULAR-1130

Internal server errors under heavy load because of high GC activity

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 1.0.0.Alpha15
    • None
    • None
    • None
      1. start cassandra node
      2. start hawkular services
      3. generate load (POST requests to hawkular metrics)

    Description

      Hawkular services JVM is spending all the time on Full GC under heavy load so some requests fail with 500 response because of timeout issues.

      Performance test is hitting hawkular metrics with POST requests. There is no other activity.

      I didn't search for exact commit but it was definitely caused by something after Oct 5. Build from Oct 5 passed.

      Here is thread dump - http://web.bc.jonqe.lab.eng.bos.redhat.com/jstack-issue.out
      Here is heap dump - http://web.bc.jonqe.lab.eng.bos.redhat.com/haw_ser_dump_issue.bin
      All logs including GC log are attached.
      I used just 512MB for heap in this test to reproduce it faster but it was visible even with 4GB for heap.

      Attachments

        Activity

          People

            Unassigned Unassigned
            fbrychta@redhat.com Filip Brychta
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: