Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-24730

Clustering: FATAL error "thread pool is full" in ejb+servlet scenario

XMLWordPrintable

    • False
    • None
    • False
    • Hide

      1. create a four nodes cluster where each node is configured as in the following:

      embed-server --server-config=standalone-ha.xml
      if (outcome != success) of /subsystem=jgroups:read-attribute(name=default-stack)
      /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
      else
      /subsystem=jgroups:write-attribute(name=default-stack,value=tcp)
      /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
      end-if
      # doubled number of credits use by MFC and UFC between 7.4.x and 8.0.x - but this value is too high for the test environment
      /subsystem=jgroups/stack=tcp/protocol=UFC:map-put(name=properties, key=max_credits, value=2m)
      /subsystem=jgroups/stack=tcp/protocol=MFC:map-put(name=properties, key=max_credits, value=2m)
      /subsystem=infinispan/cache-container=web/replicated-cache=testReplWeb:add()
      /subsystem=infinispan/cache-container=web/replicated-cache=testReplWeb/component=locking:write-attribute(name=isolation, value=REPEATABLE_READ)
      /subsystem=infinispan/cache-container=web/replicated-cache=testReplWeb/component=transaction:write-attribute(name=mode, value=BATCH)
      /subsystem=infinispan/cache-container=web/replicated-cache=testReplWeb/store=file:add()
      /subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=testReplWeb)
      /subsystem=infinispan/cache-container=ejb/replicated-cache=testReplEjb:add()
      /subsystem=infinispan/cache-container=ejb/replicated-cache=testReplEjb/component=locking:write-attribute(name=isolation, value=REPEATABLE_READ)
      /subsystem=infinispan/cache-container=ejb/replicated-cache=testReplEjb/component=transaction:write-attribute(name=mode, value=BATCH)
      /subsystem=infinispan/cache-container=ejb/replicated-cache=testReplEjb/store=file:add()
      /subsystem=infinispan/cache-container=ejb:write-attribute(name=default-cache, value=testReplEjb)
      

      2. deploy clusterbench-ee10.ear

      3. shutdown and restart the first node

      4. shutdown and restart the second node; after the application clusterbench-ee10.ear is deployed we observed the following error:

      2023-04-06 07:44:24,091 FATAL [org.jgroups.protocols.TCP] (Connection.Receiver [10.0.153.16:7600 - 10.0.152.201:51623]-11,ejb,wildfly2) wildfly2: thread pool is full (max=100, active=100); thread dump (dumped once, until thread_dump is reset):
      "Reference Handler" #2 prio=0 tid=0x2 nid=NA runnable
         java.lang.Thread.State: RUNNABLE
      	at java.base@11.0.18/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
      	at java.base@11.0.18/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
      	at java.base@11.0.18/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
      
      Show
      1. create a four nodes cluster where each node is configured as in the following: embed-server --server-config=standalone-ha.xml if (outcome != success) of /subsystem=jgroups:read-attribute(name=default-stack) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) else /subsystem=jgroups:write-attribute(name=default-stack,value=tcp) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) end-if # doubled number of credits use by MFC and UFC between 7.4.x and 8.0.x - but this value is too high for the test environment /subsystem=jgroups/stack=tcp/protocol=UFC:map-put(name=properties, key=max_credits, value=2m) /subsystem=jgroups/stack=tcp/protocol=MFC:map-put(name=properties, key=max_credits, value=2m) /subsystem=infinispan/cache-container=web/replicated-cache=testReplWeb:add() /subsystem=infinispan/cache-container=web/replicated-cache=testReplWeb/component=locking:write-attribute(name=isolation, value=REPEATABLE_READ) /subsystem=infinispan/cache-container=web/replicated-cache=testReplWeb/component=transaction:write-attribute(name=mode, value=BATCH) /subsystem=infinispan/cache-container=web/replicated-cache=testReplWeb/store=file:add() /subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=testReplWeb) /subsystem=infinispan/cache-container=ejb/replicated-cache=testReplEjb:add() /subsystem=infinispan/cache-container=ejb/replicated-cache=testReplEjb/component=locking:write-attribute(name=isolation, value=REPEATABLE_READ) /subsystem=infinispan/cache-container=ejb/replicated-cache=testReplEjb/component=transaction:write-attribute(name=mode, value=BATCH) /subsystem=infinispan/cache-container=ejb/replicated-cache=testReplEjb/store=file:add() /subsystem=infinispan/cache-container=ejb:write-attribute(name=default-cache, value=testReplEjb) 2. deploy clusterbench-ee10.ear 3. shutdown and restart the first node 4. shutdown and restart the second node; after the application clusterbench-ee10.ear is deployed we observed the following error: 2023-04-06 07:44:24,091 FATAL [org.jgroups.protocols.TCP] (Connection.Receiver [10.0.153.16:7600 - 10.0.152.201:51623]-11,ejb,wildfly2) wildfly2: thread pool is full (max=100, active=100); thread dump (dumped once, until thread_dump is reset): "Reference Handler" #2 prio=0 tid=0x2 nid=NA runnable java.lang.Thread.State: RUNNABLE at java.base@11.0.18/java.lang.ref.Reference.waitForReferencePendingList(Native Method) at java.base@11.0.18/java.lang.ref.Reference.processPendingReferences(Reference.java:241) at java.base@11.0.18/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)

      We have a four nodes cluster where on each node we deploy clusterbench-ee10.ear ;

      When we introduce fail-over by shutdown/restart of EAP, we observe the following error:

      2023-04-06 07:44:24,091 FATAL [org.jgroups.protocols.TCP] (Connection.Receiver [10.0.153.16:7600 - 10.0.152.201:51623]-11,ejb,wildfly2) wildfly2: thread pool is full (max=100, active=100); thread dump (dumped once, until thread_dump is reset):
      "Reference Handler" #2 prio=0 tid=0x2 nid=NA runnable
         java.lang.Thread.State: RUNNABLE
      	at java.base@11.0.18/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
      	at java.base@11.0.18/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
      	at java.base@11.0.18/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
      
      

            pferraro@redhat.com Paul Ferraro
            tborgato@redhat.com Tommaso Borgato
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: