Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-25812

(8.0.z) [CLUSTERING] shutdown doesn't work intermittently when using a malfuctioning DB

XMLWordPrintable

    • Critical

      Scenario: we have a 4 nodes cluster using an ORACLE RAC 19 database to offload session data;
      Each node is configured as in the following:

      embed-server --server-config=standalone-ha.xml
      /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
      /subsystem=infinispan/cache-container=web/invalidation-cache=offload:add()
      data-source add --name=testDS --jndi-name=java:jboss/datasources/testDS --driver-name=oracle-connector.jar --connection-url=jdbc:oracle:thin:@(DESCRIPTION=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-19c-rac-01.mwqe.upshift.rdu2.redhat.com)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-19c-rac-02.mwqe.upshift.rdu2.redhat.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dballo))) --enabled=true --jta=true --use-java-context=true --transaction-isolation=TRANSACTION_READ_COMMITTED --min-pool-size=1 --max-pool-size=5 --pool-prefill=true --user-name=dballo03 --password=dballo03 --prepared-statements-cache-size=32 --share-prepared-statements=true
      /subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc:add(data-source=testDS,fetch-state=false,passivation=false,purge=false,shared=true,dialect=ORACLE){allow-resource-service-restart=true}
      /subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=id-column.name,value=id)
      /subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=data-column.name,value=datum)
      /subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=timestamp-column.name,value=version)
      /subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=id-column.type,value=VARCHAR2(255))
      /subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=timestamp-column.type,value=NUMBER)
      /subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=data-column.type,value=BLOB)
      /subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=prefix, value=i){allow-resource-service-restart=true}
      /subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=offload)
      /subsystem=transactions:write-attribute(name=node-identifier,value=wildfly1)
      

      Each node is failed in a sequence by being shut down and subsequently restarted;
      At the end of the test, all node are shut down in the following order node1, node2,node3,node4;

      The issue is that node2 did not shut down after receiving the shut down command;

      Complete logs in RUN-119.zip;

      Thread dump of the node that hung in stack.log;

      The issue is very difficult to reproduce: we just observed it once so far;

        1. jstack-mariadb-galera#129.zip
          61 kB
        2. mariadb-galera#129-wildfly.zip
          121 kB
        3. MARIGALERA.zip
          105 kB
        4. RUN-119.zip
          135 kB
        5. stack.log
          47 kB
        6. wildfly-db-session-shutdown-repl-mariadb-galera#119.zip
          126 kB

            pferraro@redhat.com Paul Ferraro
            tborgato@redhat.com Tommaso Borgato
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: