Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-15344

Fuse/CXF (7.7) application on EAP 7.2.9: threads spin forever, after period of load

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • fuse-7.9-GA
    • fuse-7.7-GA
    • WildFly Camel
    • None
    • False
    • False
    • % %
    • Undefined

      An application that consumes SOAP HTTP requests using Camel and CXF is based on Fuse 7.7 and deployed on EAP 7.2.9. After a long period of high load (weeks), multiple EAP servers enter an 100%-CPU state, with many threads stuck with this thread stack:

       

      "I/O dispatcher 39" #885 prio=5 os_prio=0 tid=0x00000000093ae000 nid=0x6245 runnable [0x00007f0d5267a000]
         java.lang.Thread.State: RUNNABLE
      	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
      	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
      	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
      	- locked <0x0000000519e671a8> (a sun.nio.ch.Util$3)
      	- locked <0x0000000519e67198> (a java.util.Collections$UnmodifiableSet)
      	- locked <0x0000000519e67150> (a sun.nio.ch.EPollSelectorImpl)
      	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
      	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
      	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
      	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
      	at java.lang.Thread.run(Thread.java:748) 

       

      These threads persist in every subsequent thread dump, for as long as the customer can wait, even after all load is removed from the server. The problem can affect multiple EAP servers on the same occasion.

      Since there have been reported problems in this area with CXF on EAP in the past, the customer upgraded his JVM to 1.8.0_252 and his EAP from 7.2.8 to 7.2.9, with no improvement.

      The problem appears to have started after an upgrade from EAP 7.2.1. However, given how infrequent it is, it's hard to be certain exactly what the cause is.

       

       

       

        1. under_load.svg
          269 kB
        2. after_unload.svg
          269 kB
        3. Thread_Blocked.out
          23 kB

              ldemasi Luigi De Masi
              rhn-support-kboone Kevin Boone
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: