Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-12074

EJB client 4.x freezes intermittently after some time

    XMLWordPrintable

Details

    • Regression, Blocks Testing

    Description

      Seen in our clustering failover tests for remote stateful EJBs.
      Setup: 4 node cluster, one node at the time is shut down gracefully/killed, while 500 standalone clients keep calling the application (number of clients was lowered from 2000 to reduce the log size).

      Description: the test continues normally for some time (3-4 minutes), then suddenly all the client threads get stuck on invoking remoteBean.getSerialAndIncrement() and never get a response.

      Note: we do not specify invocation timeout so it's set to its default value. We can set it to any value if needed.
      Edit: We cannot. See JBEAP-12075.

      NOTE: There is extra debugging enabled - each EJB client logs these two messages (in the beginning of the invocation and after it received a response):

      Invoking SFSB: getSerialAndIncrement()
Response acquired, serial: X delay: Y

      You can see the ID number of the EJB client in the log message: [Runner - XXX]

      Client log:
      http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/mvinkler_perflab_eap-7x-failover-ejb-ejbremote-jvmkill-dist-async/14/console-perf29/

      Snippet from client log - clients log "getSerialAndIncrement()" but no response comes after that.

      2017/07/12 03:32:37:039 EDT [DEBUG][Runner - 404] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Response acquired, serial: 49 delay: 2
2017/07/12 03:32:37:039 EDT [DEBUG][Runner - 279] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:039 EDT [DEBUG][Runner - 17] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Response acquired, serial: 50 delay: 2
2017/07/12 03:32:37:041 EDT [DEBUG][Runner - 89] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:041 EDT [DEBUG][Runner - 279] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Response acquired, serial: 49 delay: 2
2017/07/12 03:32:37:043 EDT [DEBUG][Runner - 89] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Response acquired, serial: 50 delay: 2
2017/07/12 03:32:37:093 EDT [DEBUG][Runner - 244] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:093 EDT [DEBUG][Runner - 256] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:111 EDT [DEBUG][Runner - 249] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:113 EDT [DEBUG][Runner - 249] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Response acquired, serial: 50 delay: 2
2017/07/12 03:32:37:127 EDT [DEBUG][Runner - 408] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:129 EDT [DEBUG][Runner - 408] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Response acquired, serial: 49 delay: 2
2017/07/12 03:32:37:156 EDT [DEBUG][Runner - 250] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:161 EDT [DEBUG][Runner - 251] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:161 EDT [DEBUG][Runner - 409] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:184 EDT [DEBUG][Runner - 252] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:306 EDT [DEBUG][Runner - 415] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:332 EDT [DEBUG][Runner - 414] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:338 EDT [DEBUG][Runner - 258] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:364 EDT [DEBUG][Runner - 259] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:367 EDT [DEBUG][Runner - 417] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:437 EDT [DEBUG][Runner - 420] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:464 EDT [DEBUG][Runner - 421] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:563 EDT [DEBUG][Runner - 423] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:599 EDT [DEBUG][Runner - 425] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:602 EDT [DEBUG][Runner - 426] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:629 EDT [DEBUG][Runner - 428] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:673 EDT [DEBUG][Runner - 269] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:692 EDT [DEBUG][Runner - 270] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:694 EDT [DEBUG][Runner - 429] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:721 EDT [DEBUG][Runner - 272] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:767 EDT [DEBUG][Runner - 274] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:791 EDT [DEBUG][Runner - 276] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:850 EDT [DEBUG][Runner - 278] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:893 EDT [DEBUG][Runner - 438] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:923 EDT [DEBUG][Runner - 439] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().
2017/07/12 03:32:37:923 EDT [DEBUG][Runner - 281] HOST perf29.mw.lab.eng.bos.redhat.com:rootProcess:c - Invoking SFSB: getSerialAndIncrement().

      This is very similar to JBEAP-10307 although there is no obvious connection between cluster topology change and the clients being stuck.

      Attachments

        1. JBEAP-12074_perf22_eap-7x-failover-ejb-ejbremote-jvmkill-dist-sync-7.zip
          991 kB
        2. jstack_threaddump_#3.zip
          183 kB
        3. jstack_threaddump_#4.zip
          166 kB
        4. jstack_threaddump_2.zip
          343 kB
        5. jstack_threaddump_24_run2.out
          1.10 MB
        6. jstack_threaddump_26_CR1.out
          2.15 MB
        7. jstack_threaddump_30_run1.out
          1.13 MB
        8. jstack_threaddump.zip
          461 kB
        9. jvmkill-repl-sync_threaddump.zip
          1010 kB
        10. stackTraceAfterLockup.txt
          548 kB
        11. threaddump_055e133.txt
          6.07 MB

        Issue Links

          is blocked by

          Bug - A problem that impairs or prevents the functions of the product. EJBCLIENT-255 RemoteEJBReceiver blocks when attempting to get the client service, instead of registering a notifier

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved
          is caused by

          Bug - A problem that impairs or prevents the functions of the product. EJBCLIENT-267 Invalid state possible in getResult

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Resolved

          Bug - A problem that impairs or prevents the functions of the product. REM3-301 Message tracker semaphore is misbehaving

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-13169 EJB client 4.x occasionally hangs

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Verified

          Bug - A problem that impairs or prevents the functions of the product. EJBCLIENT-266 EJB client should limit concurrent retries, as it can cause the worker pool to fill up resulting in a hang

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Resolved
          is incorporated by

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. JBEAP-12600 Upgrade to ejb-client 4.0.0.Final

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Verified
          relates to

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-12076 (7.1.z) Client gets stuck when invoking EJBs in multiple threads when using old remote-naming API

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed

          Activity

            People

              Unassigned Unassigned
              mvinkler1@redhat.com Michal Vinkler
              Daniel Cihak Daniel Cihak
              Daniel Cihak Daniel Cihak
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: