Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-12075

Wildfly Fails to stop due to Infinispan persisted cache not holding all permits on semaphore

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 17.0.0.Final
    • 15.0.1.Final
    • Clustering
    • None
    • Hide

      1. start server and access web application (http://localhost:8080/wildfly-helloworld/hello) so that it fills the caches.
      2. shutdown Wildfly server using jboss-cli.bat --connect --controller=localhost:9990 --command=:shutdown (this time it stops without problems)
      3. start server and access web application (http://localhost:8080/wildfly-helloworld/hello) so that it calls method that accesses the keySets and entrySets from caches
      4. shutdown Wildfly server using jboss-cli.bat --connect --controller=localhost:9990 --command=:shutdown (this time it doesn't stops)

      Show
      1. start server and access web application ( http://localhost:8080/wildfly-helloworld/hello ) so that it fills the caches. 2. shutdown Wildfly server using jboss-cli.bat --connect --controller=localhost:9990 --command=:shutdown (this time it stops without problems) 3. start server and access web application ( http://localhost:8080/wildfly-helloworld/hello ) so that it calls method that accesses the keySets and entrySets from caches 4. shutdown Wildfly server using jboss-cli.bat --connect --controller=localhost:9990 --command=:shutdown (this time it doesn't stops)
    • User Experience
    • Workaround Exists
    • Hide

      Either kill process if it takes to long or create a ShutdownHook that checks if a cache stop thread stays to much in waiting and force to release a permit:

      ((AbstractQueuedSynchronizer) LockSupport.getBlocker(thread)).releaseShared(1);
      Show
      Either kill process if it takes to long or create a ShutdownHook that checks if a cache stop thread stays to much in waiting and force to release a permit: ((AbstractQueuedSynchronizer) LockSupport.getBlocker(thread)).releaseShared(1);

    Description

      Wildfly 15.0.1 fails to stop on shutdown command due to Infinispan subsystem failing to stop it's persisted caches successfully due to a possible bug in Infinispan's PersistenceManagerImpl.publishEntries not releasing Semaphore permit after finishing, so that the stop method waits forever because it lacks one more permit.


      Use following application to reproduce the issue:
      web-java-wildfly.zip

      standalone Infinispan caches:

                  <cache-container name="testMetaData">
                <local-cache name="testCache">
                    <locking concurrency-level="3"/>
                    <transaction locking="OPTIMISTIC" mode="FULL_XA"/>
                    <object-memory size="1000000"/>
                    <file-store path="testCH/testCache" passivation="false" preload="false" purge="false"/>
                </local-cache>
                <local-cache name="testComponentCache">
                    <locking concurrency-level="3"/>
                    <transaction locking="OPTIMISTIC" mode="FULL_XA"/>
                    <object-memory size="1000000"/>
                    <file-store path="testCH/testComponentCache" passivation="false" preload="false" purge="false"/>
                </local-cache>
            </cache-container>

      11:02:00,103 INFO  [stdout] (default task-1) bla blanull
11:02:00,103 INFO  [stdout] (default task-1) bla blanull
11:03:17,748 INFO  [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0236: Suspending server with no timeout.
11:03:17,750 INFO  [org.jboss.as.ejb3] (management-handler-thread - 1) WFLYEJB0493: EJB subsystem suspension complete
11:03:17,752 INFO  [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
11:03:17,785 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb1
11:03:17,785 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-7) WFLYMAIL0002: Unbound mail session [java:/mail/myproject]
11:03:17,785 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb2
11:03:17,787 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb3
11:03:17,789 INFO  [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 88) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
11:03:17,790 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-3) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
11:03:17,790 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb4
11:03:17,797 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0019: Stopped Driver service with driver-name = ojdbc.jar
11:03:17,800 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 101) WFLYUT0022: Unregistered web context: '/wildfly-helloworld' from server 'default-server'
11:03:17,821 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 101) MODCLUSTER000002: Initiating mod_cluster shutdown
11:03:17,823 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0019: Host default-host stopping
11:03:17,840 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-3) WFLYJCA0119: Unbinding connection factory named java:/JmsXA to alias java:jboss/DefaultJMSConnectionFactory
11:03:17,841 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
11:03:17,842 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
11:03:17,842 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = h2
11:03:17,846 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 101) WFLYCLINF0003: Stopped client-mappings cache from ejb container
11:03:17,864 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0008: Undertow HTTP listener default suspending
11:03:17,864 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0008: Undertow AJP listener ajp suspending
11:03:17,870 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0028: Stopped deployment ojdbc.jar (runtime-name: ojdbc.jar) in 108ms
11:03:17,883 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0028: Stopped deployment wildfly-helloworld.war (runtime-name: wildfly-helloworld.war) in 120ms
11:03:17,875 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to 192.168.1.2:8009
11:03:17,874 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 192.168.1.2:8080
11:03:17,872 INFO  [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 104) AMQ151003: resource adaptor stopped
11:03:17,932 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 104) AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.6.3.jbossorg-00014 [7b985d04-761e-11e9-bb4e-0a0027000008] stopped, uptime 1 minute
11:03:17,933 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0008: Undertow HTTPS listener default-ssl suspending
11:03:17,934 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0007: Undertow HTTPS listener default-ssl stopped, was bound to 192.168.1.2:8443
11:03:17,935 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0004: Undertow 2.0.15.Final stopping

      Attachments

        1. forth_semaphore.png
          forth_semaphore.png
          69 kB
        2. third_semaphore.png
          third_semaphore.png
          70 kB
        3. web-java-wildfly.zip
          60 kB

        Issue Links

          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. ISPN-10206 Cache fails to stop due to unreleased persistence manager semaphore permits when using early terminal stream operations

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              blassmegod@yahoo.com Blass MeGod (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: