Uploaded image for project: 'Teiid'
  1. Teiid
  2. TEIID-3106

BufferManager Cleaner consuming 90% of total CPU time

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 8.7.1, 8.9
    • 8.7
    • Query Engine
    • None
    • Hide

      There are no straight forward steps to reproduce as it is intermittent problem. Start and waiting for high CPU to occur is the only known step. It has been found to be happening ONLY on start-up though.

      Show
      There are no straight forward steps to reproduce as it is intermittent problem. Start and waiting for high CPU to occur is the only known step. It has been found to be happening ONLY on start-up though.

    Description

      BufferManager Cleaner thread is consuming almost all of the CPU utilized by the jboss process. Thread dump shows following information.

      3XMTHREADINFO "BufferManager Cleaner" J9VMThread:0x0000004C41CFEB00, j9thread_t:0x0000004C52B85AE0, java/lang/Thread:0x000000481C036E20, state:CW, prio=5
      3XMJAVALTHREAD (java/lang/Thread getId:0x76, isDaemon:true)
      3XMTHREADINFO1 (native thread ID:0x3AEC2600, native priority:0x5, native policy:UNKNOWN)
      3XMHEAPALLOC Heap bytes allocated since last GC cycle=2609184 (0x27D020)
      3XMTHREADINFO3 Java callstack:
      4XESTACKTRACE at java/util/concurrent/ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1070(Compiled Code))
      4XESTACKTRACE at java/util/concurrent/ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1659(Compiled Code))
      4XESTACKTRACE at org/teiid/common/buffer/impl/LrfuEvictionQueue.remove(LrfuEvictionQueue.java:60(Compiled Code))
      4XESTACKTRACE at org/teiid/common/buffer/impl/BufferManagerImpl.doEvictions(BufferManagerImpl.java:854(Compiled Code))
      5XESTACKTRACE (entered lock: org/teiid/common/buffer/CacheEntry@0x00000048393C2598, entry count: 1)
      4XESTACKTRACE at org/teiid/common/buffer/impl/BufferManagerImpl$Cleaner.run(BufferManagerImpl.java:108)
      4XESTACKTRACE at java/util/TimerThread.mainLoop(Timer.java:555)
      4XESTACKTRACE at java/util/TimerThread.run(Timer.java:505)

      When we added log statements around the BufferManagerImpl.doEvictions() it loops through the remove and firstEntry loop.

      Forum discussion link : https://community.jboss.org/message/901792

      Attachments

        1. BufferManagerImpl.java
          44 kB
        2. BufferManagerImpl.java
          43 kB
        3. CacheKey_Loog.patch
          2 kB
        4. complete_log.zip
          547 kB
        5. cpu monitoring.docx
          2.31 MB
        6. dump.zip
          73 kB
        7. in_local.docx
          1.20 MB
        8. LrfuEvictionQueue.java
          6 kB
        9. LrfuEvictionQueue.patch
          2 kB
        10. non_high_cpu.docx
          295 kB

        Activity

          People

            rhn-engineering-shawkins Steven Hawkins
            deveshmishra_jira Devesh Mishra (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: