Uploaded image for project: 'XNIO'
  1. XNIO
  2. XNIO-335

XNIO I/O threads are taking high CPU (may be because its not closing CLOSE_WAIT TCP connections)

XMLWordPrintable

      Hello,

      XNIO I/O threads starts taking more CPU after an hour of deployment. Can you please help us with this issue:

      Details:
      1. We are using Spring boot + undertow (version details are mentioned below)
      2. App server is behind NGnix load balancer. We are using keep alive connections from NGnix to app server
      3. When we deploy the app server, everything is OK, CPU usage is low, app is able to close TCP connections.
      4. After an hour or so, the app server process starts taking up 25% of the CPU. Upon some inspection, I see that app is not able to close the CLOSE_WAIT connections (which may be the root cause )
      5. When we leave the app server up and running for few days, it takes up more than 50% of CPU and the number of CLOSE_WAIT connections grows.
      6. We are consistently able to reproduce with following versions:

      • spring-boot-starter-undertow: 2.1.1.RELEASE and 2.1.3.RELEASE
      • xnio-nio: 3.6.5.Final and 3.7.0.Final

      7. I have noticed one 502 (BAD GATEWAY) from the app server just around the time when it starts taking up high CPU and at that moment I see the CLOSE_WAIT connections (although this app server doesn't have much traffic). May be when XNIO I/O thread rejects the incoming request, it gets into this state?
      8. Please see attached images.

      Using following server options:

      • builder.setServerOption(UndertowOptions.NO_REQUEST_TIMEOUT, 30 * 1000);
      • builder.setServerOption(UndertowOptions.REQUEST_PARSE_TIMEOUT, 30 * 1000);
      • builder.setServerOption(UndertowOptions.IDLE_TIMEOUT, 60 * 1000);
      • server.connection-timeout=50000 (spring property)

      Please let me know if you need additional details.

      Thanks,
      Amar

        1. file_descriptors.png
          file_descriptors.png
          490 kB
        2. lsof_machine-10.png
          lsof_machine-10.png
          221 kB
        3. Thread Status - XNIO-1 IO-4.png
          Thread Status - XNIO-1 IO-4.png
          174 kB
        4. Thread Status - XNIO-1 IO-5.png
          Thread Status - XNIO-1 IO-5.png
          166 kB
        5. XNIO_ConnectionCount.png
          XNIO_ConnectionCount.png
          377 kB
        6. XNIO_threads_steadly_increasing_CPU_usage.png
          XNIO_threads_steadly_increasing_CPU_usage.png
          833 kB

              dlloyd@redhat.com David Lloyd
              amarsladdha_jira Amar Laddha (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: