Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-18420

(7.3.z) Clustering: server not shutting down in response to management operation 'shutdown'

    XMLWordPrintable

Details

    • Bug
    • Resolution: Obsolete
    • Critical
    • None
    • 7.3.0.GA.CR1
    • Clustering
    • None

    Description

      The error is observed in Clustering tests with the following configuration: 4 EAP nodes Cluster where session data is offloaded to an Oracle 12c relational Database;

      The 4 EAP nodes are configured as follows:

      embed-server --server-config=standalone-ha.xml
/subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload:add()
/subsystem=infinispan/cache-container=web/invalidation-cache==offload/component=transaction:write-attribute(name=mode, value=BATCH)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/component=locking:write-attribute(name=isolation, value=REPEATABLE_READ)
data-source add --name=testDS --jndi-name=java:jboss/datasources/testDS --driver-name=oracle-connector.jar --connection-url=jdbc:oracle:thin:@oracle-12cr2.rhev-ci-vms.eng.rdu2.redhat.com:1521:dballo --enabled=true --jta=true --use-java-context=true --transaction-isolation=TRANSACTION_READ_COMMITTED --min-pool-size=1 --max-pool-size=5 --pool-prefill=true --user-name=dballo13 --password=dballo13 --prepared-statements-cache-size=32 --share-prepared-statements=true
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc:add(data-source=testDS,fetch-state=false,passivation=false,purge=false,shared=true,dialect=ORACLE){allow-resource-service-restart=true}
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=prefix,value=s)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=id-column.name,value=id)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=id-column.type,value=VARCHAR(255))
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=data-column.name,value=datum)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=data-column.type,value=BLOB)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=timestamp-column.name,value=version)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=timestamp-column.type,value=NUMBER)
/subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=offload)
/subsystem=transactions:write-attribute(name=node-identifier,value=wildfly1)

      The sequence of steps performed in the test is the following:

      • EAP on node 1 is started
      • EAP on node 2 is started
      • EAP on node 3 is started
      • EAP on node 4 is started
      • EAP on node 1 is stopped
      • EAP on node 1 is started
      • EAP on node 2 is stopped
      • EAP on node 2 is started
      • EAP on node 3 is stopped
      • EAP on node 3 is started: 2020-01-07 13:39:58,771 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service org.wildfly.undertow.listener.default: org.jboss.msc.service.StartException in service org.wildfly.undertow.listener.default: Address already in use /10.16.176.62:8080
      • EAP on node 4 is stopped
      • EAP on node 4 is started: 2020-01-07 13:39:59,046 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service org.wildfly.undertow.listener.https: org.jboss.msc.service.StartException in service org.wildfly.undertow.listener.https: Address already in use /10.16.176.62:8443

      Observing EAP logs on node 3, it looks like the shut-down sequence is not completed:

      2020-01-07 13:38:04,799 INFO  [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
2020-01-07 13:38:04,826 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-7) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
2020-01-07 13:38:04,828 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting JGroups channel ejb
2020-01-07 13:38:04,828 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel ejb
2020-01-07 13:38:04,828 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0010: Unbound data source [java:jboss/datasources/testDS]
2020-01-07 13:38:04,831 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 87) WFLYUT0022: Unregistered web context: '/web-logger' from server 'default-server'
2020-01-07 13:38:04,833 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 97) WFLYUT0022: Unregistered web context: '/clusterbench-granular' from server 'default-server'
2020-01-07 13:38:04,837 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 95) WFLYUT0022: Unregistered web context: '/clusterbench-passivating' from server 'default-server'
2020-01-07 13:38:04,842 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 98) WFLYUT0022: Unregistered web context: '/clusterbench' from server 'default-server'
2020-01-07 13:38:04,859 INFO  [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 99) ISPN000029: Passivating all entries to disk
2020-01-07 13:38:04,861 INFO  [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 99) ISPN000030: Passivated 0 entries in 2 milliseconds
2020-01-07 13:38:04,864 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 99) [Context=cbnc.ear/clusterbench-ejb.jar/infinispan] ISPN100008: Updating cache members list [wildfly1, wildfly2], topology id 40
2020-01-07 13:38:04,866 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 99) [Context=cbnc.ear/clusterbench-ejb.jar/infinispan] ISPN100002: Starting rebalance with members [wildfly1, wildfly2], phase READ_OLD_WRITE_ALL, topology id 41
2020-01-07 13:38:04,874 WARN  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 101) [Context=cbnc.ear.c.war] ISPN000312: Lost data because of graceful leaver wildfly4
2020-01-07 13:38:04,876 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 101) [Context=cbnc.ear.c.war] ISPN100008: Updating cache members list [wildfly1, wildfly2], topology id 20
2020-01-07 13:38:04,876 WARN  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 97) [Context=cbnc.ear.b.war] ISPN000312: Lost data because of graceful leaver wildfly4
2020-01-07 13:38:04,878 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 99) WFLYCLINF0003: Stopped cbnc.ear/clusterbench-ejb.jar/infinispan cache from ejb container
2020-01-07 13:38:04,879 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 101) [Context=cbnc.ear.c.war] ISPN100008: Updating cache members list [wildfly1, wildfly2], topology id 21
2020-01-07 13:38:04,879 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 97) [Context=cbnc.ear.b.war] ISPN100008: Updating cache members list [wildfly1, wildfly2], topology id 20
2020-01-07 13:38:04,880 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 101) WFLYCLINF0003: Stopped cbnc.ear.c.war cache from web container
2020-01-07 13:38:04,881 INFO  [org.infinispan.CLUSTER] (remote-thread--p8-t8) [Context=cbnc.ear/clusterbench-ejb.jar/infinispan] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 42
2020-01-07 13:38:04,882 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 97) [Context=cbnc.ear.b.war] ISPN100008: Updating cache members list [wildfly1, wildfly2], topology id 21
2020-01-07 13:38:04,883 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 97) WFLYCLINF0003: Stopped cbnc.ear.b.war cache from web container
2020-01-07 13:38:04,884 INFO  [org.infinispan.CLUSTER] (remote-thread--p8-t8) [Context=cbnc.ear/clusterbench-ejb.jar/infinispan] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 43
2020-01-07 13:38:04,887 INFO  [org.infinispan.CLUSTER] (remote-thread--p8-t8) [Context=cbnc.ear/clusterbench-ejb.jar/infinispan] ISPN100010: Finished rebalance with members [wildfly1, wildfly2], topology id 44
2020-01-07 13:38:04,888 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0028: Stopped deployment web-logger.war (runtime-name: web-logger.war) in 84ms
2020-01-07 13:38:05,316 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
2020-01-07 13:38:05,318 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 87) MODCLUSTER000002: Initiating mod_cluster shutdown
2020-01-07 13:38:05,320 WARN  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 101) [Context=cbnc.ear.a.war] ISPN000312: Lost data because of graceful leaver wildfly4
2020-01-07 13:38:05,322 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 101) [Context=cbnc.ear.a.war] ISPN100008: Updating cache members list [wildfly1, wildfly2], topology id 20
2020-01-07 13:38:05,331 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 101) [Context=cbnc.ear.a.war] ISPN100008: Updating cache members list [wildfly1, wildfly2], topology id 21
2020-01-07 13:38:05,331 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 96) [Context=default-server] ISPN100008: Updating cache members list [wildfly1, wildfly2], topology id 36
2020-01-07 13:38:05,334 WARN  [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 101) ISPN000100: Stopping, but there are 0 local transactions and 10 remote transactions that did not finish in time.
2020-01-07 13:38:05,337 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 96) WFLYCLINF0003: Stopped default-server cache from web container
2020-01-07 13:38:05,339 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
2020-01-07 13:38:05,342 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-6) WFLYJCA0019: Stopped Driver service with driver-name = h2
2020-01-07 13:38:05,355 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 96) [Context=client-mappings] ISPN100008: Updating cache members list [wildfly1, wildfly2], topology id 40
2020-01-07 13:38:05,356 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 96) WFLYCLINF0003: Stopped client-mappings cache from ejb container
2020-01-07 13:38:05,359 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel ejb
2020-01-07 13:38:05,375 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 101) WFLYCLINF0003: Stopped cbnc.ear.a.war cache from web container
2020-01-07 13:38:05,378 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel ejb
2020-01-07 13:38:05,390 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ejb.jar) in 585ms
2020-01-07 13:38:05,392 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0208: Stopped subdeployment (runtime-name: a.war) in 583ms
2020-01-07 13:38:05,392 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0208: Stopped subdeployment (runtime-name: c.war) in 587ms
2020-01-07 13:38:05,392 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: b.war) in 587ms
2020-01-07 13:38:05,394 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0028: Stopped deployment cbnc.ear (runtime-name: cbnc.ear) in 590ms

2020-01-07 13:39:56,113 INFO  [org.jboss.modules] (main) JBoss Modules version 1.9.1.Final-redhat-00001
2020-01-07 13:39:56,539 INFO  [org.jboss.msc] (main) JBoss MSC version 1.4.11.Final-redhat-00001
2020-01-07 13:39:56,547 INFO  [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final-redhat-00001
2020-01-07 13:39:56,674 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: JBoss EAP 7.3.0.GA (WildFly Core 10.1.0.Final-redhat-00001) starting

      Attachments

        Activity

          People

            rhn-support-iweiss Ingo Weiss
            tborgato@redhat.com Tommaso Borgato
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: