Uploaded image for project: 'Teiid'
  1. Teiid
  2. TEIID-3106

BufferManager Cleaner consuming 90% of total CPU time

    Details

    • Steps to Reproduce:
      Hide

      There are no straight forward steps to reproduce as it is intermittent problem. Start and waiting for high CPU to occur is the only known step. It has been found to be happening ONLY on start-up though.

      Show
      There are no straight forward steps to reproduce as it is intermittent problem. Start and waiting for high CPU to occur is the only known step. It has been found to be happening ONLY on start-up though.

      Description

      BufferManager Cleaner thread is consuming almost all of the CPU utilized by the jboss process. Thread dump shows following information.

      3XMTHREADINFO "BufferManager Cleaner" J9VMThread:0x0000004C41CFEB00, j9thread_t:0x0000004C52B85AE0, java/lang/Thread:0x000000481C036E20, state:CW, prio=5
      3XMJAVALTHREAD (java/lang/Thread getId:0x76, isDaemon:true)
      3XMTHREADINFO1 (native thread ID:0x3AEC2600, native priority:0x5, native policy:UNKNOWN)
      3XMHEAPALLOC Heap bytes allocated since last GC cycle=2609184 (0x27D020)
      3XMTHREADINFO3 Java callstack:
      4XESTACKTRACE at java/util/concurrent/ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1070(Compiled Code))
      4XESTACKTRACE at java/util/concurrent/ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1659(Compiled Code))
      4XESTACKTRACE at org/teiid/common/buffer/impl/LrfuEvictionQueue.remove(LrfuEvictionQueue.java:60(Compiled Code))
      4XESTACKTRACE at org/teiid/common/buffer/impl/BufferManagerImpl.doEvictions(BufferManagerImpl.java:854(Compiled Code))
      5XESTACKTRACE (entered lock: org/teiid/common/buffer/CacheEntry@0x00000048393C2598, entry count: 1)
      4XESTACKTRACE at org/teiid/common/buffer/impl/BufferManagerImpl$Cleaner.run(BufferManagerImpl.java:108)
      4XESTACKTRACE at java/util/TimerThread.mainLoop(Timer.java:555)
      4XESTACKTRACE at java/util/TimerThread.run(Timer.java:505)

      When we added log statements around the BufferManagerImpl.doEvictions() it loops through the remove and firstEntry loop.

      Forum discussion link : https://community.jboss.org/message/901792

        Gliffy Diagrams

          Attachments

          1. BufferManagerImpl.java
            44 kB
          2. BufferManagerImpl.java
            43 kB
          3. CacheKey_Loog.patch
            2 kB
          4. complete_log.zip
            547 kB
          5. cpu monitoring.docx
            2.31 MB
          6. dump.zip
            73 kB
          7. in_local.docx
            1.20 MB
          8. LrfuEvictionQueue.java
            6 kB
          9. LrfuEvictionQueue.patch
            2 kB
          10. non_high_cpu.docx
            295 kB

            Activity

              People

              • Assignee:
                shawkins Steven Hawkins
                Reporter:
                deveshmishra Devesh Mishra
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: