Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-10206

Cache fails to stop due to unreleased persistence manager semaphore permits when using early terminal stream operations

XMLWordPrintable

    • Hide

      1. start server and access web application (http://localhost:8080/wildfly-helloworld/hello) so that it fills the caches.
      2. shutdown Wildfly server using jboss-cli.bat --connect --controller=localhost:9990 --command=:shutdown (this time it stops without problems)
      3. start server and access web application (http://localhost:8080/wildfly-helloworld/hello) so that it calls method that accesses the keySets and entrySets from caches
      4. shutdown Wildfly server using jboss-cli.bat --connect --controller=localhost:9990 --command=:shutdown (this time it doesn't stops)

      Show
      1. start server and access web application ( http://localhost:8080/wildfly-helloworld/hello ) so that it fills the caches. 2. shutdown Wildfly server using jboss-cli.bat --connect --controller=localhost:9990 --command=:shutdown (this time it stops without problems) 3. start server and access web application ( http://localhost:8080/wildfly-helloworld/hello ) so that it calls method that accesses the keySets and entrySets from caches 4. shutdown Wildfly server using jboss-cli.bat --connect --controller=localhost:9990 --command=:shutdown (this time it doesn't stops)
    • User Experience
    • Workaround Exists
    • Hide

      Either kill process if it takes to long or create a ShutdownHook that checks if a cache stop thread stays to much in waiting and force to release a permit:

      ((AbstractQueuedSynchronizer) LockSupport.getBlocker(thread)).releaseShared(1);
      Show
      Either kill process if it takes to long or create a ShutdownHook that checks if a cache stop thread stays to much in waiting and force to release a permit: ((AbstractQueuedSynchronizer) LockSupport.getBlocker(thread)).releaseShared(1);

      Wildfly 15.0.1 fails to stop on shutdown command due to Infinispan subsystem failing to stop it's persisted caches successfully due to a possible bug in Infinispan's PersistenceManagerImpl.publishEntries not releasing Semaphore permit after finishing, so that the stop method waits forever because it lacks one more permit.

      To clarify this issue only happens when invoking an operation that uses an early terminating stream operation that doesn't fully iterate over the store contents. Examples would be `Cache.isEmpty`


      Use following application to reproduce the issue:
      [^web-java-wildfly.zip]

      standalone Infinispan caches:

                  <cache-container name="testMetaData">
                <local-cache name="testCache">
                    <locking concurrency-level="3"/>
                    <transaction locking="OPTIMISTIC" mode="FULL_XA"/>
                    <object-memory size="1000000"/>
                    <file-store path="testCH/testCache" passivation="false" preload="false" purge="false"/>
                </local-cache>
                <local-cache name="testComponentCache">
                    <locking concurrency-level="3"/>
                    <transaction locking="OPTIMISTIC" mode="FULL_XA"/>
                    <object-memory size="1000000"/>
                    <file-store path="testCH/testComponentCache" passivation="false" preload="false" purge="false"/>
                </local-cache>
            </cache-container>

      11:02:00,103 INFO  [stdout] (default task-1) bla blanull
11:02:00,103 INFO  [stdout] (default task-1) bla blanull
11:03:17,748 INFO  [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0236: Suspending server with no timeout.
11:03:17,750 INFO  [org.jboss.as.ejb3] (management-handler-thread - 1) WFLYEJB0493: EJB subsystem suspension complete
11:03:17,752 INFO  [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
11:03:17,785 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb1
11:03:17,785 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-7) WFLYMAIL0002: Unbound mail session [java:/mail/myproject]
11:03:17,785 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb2
11:03:17,787 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb3
11:03:17,789 INFO  [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 88) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
11:03:17,790 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-3) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
11:03:17,790 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb4
11:03:17,797 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0019: Stopped Driver service with driver-name = ojdbc.jar
11:03:17,800 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 101) WFLYUT0022: Unregistered web context: '/wildfly-helloworld' from server 'default-server'
11:03:17,821 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 101) MODCLUSTER000002: Initiating mod_cluster shutdown
11:03:17,823 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0019: Host default-host stopping
11:03:17,840 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-3) WFLYJCA0119: Unbinding connection factory named java:/JmsXA to alias java:jboss/DefaultJMSConnectionFactory
11:03:17,841 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
11:03:17,842 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
11:03:17,842 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = h2
11:03:17,846 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 101) WFLYCLINF0003: Stopped client-mappings cache from ejb container
11:03:17,864 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0008: Undertow HTTP listener default suspending
11:03:17,864 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0008: Undertow AJP listener ajp suspending
11:03:17,870 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0028: Stopped deployment ojdbc.jar (runtime-name: ojdbc.jar) in 108ms
11:03:17,883 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0028: Stopped deployment wildfly-helloworld.war (runtime-name: wildfly-helloworld.war) in 120ms
11:03:17,875 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to 192.168.1.2:8009
11:03:17,874 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 192.168.1.2:8080
11:03:17,872 INFO  [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 104) AMQ151003: resource adaptor stopped
11:03:17,932 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 104) AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.6.3.jbossorg-00014 [7b985d04-761e-11e9-bb4e-0a0027000008] stopped, uptime 1 minute
11:03:17,933 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0008: Undertow HTTPS listener default-ssl suspending
11:03:17,934 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0007: Undertow HTTPS listener default-ssl stopped, was bound to 192.168.1.2:8443
11:03:17,935 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0004: Undertow 2.0.15.Final stopping

              wburns@redhat.com Will Burns
              pferraro@redhat.com Paul Ferraro
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: