Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-10749

Artemis with JDBC store on slow or high latency database crashed on: Critical IO Error. Failed to process JDBC Record statements: java.sql.BatchUpdateException: IO Error: Socket read timed out

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 12.0.0.Final, 13.0.0.Final, 14.0.0.Final, 15.0.0.Beta1
    • JMS
    • None

      Artemis configured with JDBC store is not robust if database is slow/under high load and when there is higher latency to database. Artemis has tendency to shutdown itself on "Critical IO Error" under really low load. It is visible when EAP QE moved Jenkins to Central CI where is higher latency to used Oracle12cR2 database. In following test scenario:

      • Start 2 WF/EAP server in cluster. There is queue InQueue and OutQueue deployed on both of the servers.
      • Send 10 000 small messages to InQueue to server 1
      • Then deploy MDB on server 2. MDB consumes messages from InQueue and resends to OutQueue
      • Wait until MDB processes all messages and check that 10 000 messages is in OutQueue

      Result:
      Sometimes happens that Artemis in server 1 crashes due to following exception:

      20:13:45,641 WARN  [org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl] (Thread-17 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@4de066d8)) IO Error: Socket read 
timed out: java.sql.BatchUpdateException: IO Error: Socket read timed out
        at oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:10032)
        at oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1364)
        at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9839)
        at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:234)
        at org.jboss.jca.adapters.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:1180)
        at org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl.sync(JDBCJournalImpl.java:228) [artemis-jdbc-store-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl$JDBCJournalSync.run(JDBCJournalImpl.java:991) [artemis-jdbc-store-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$2.run(ActiveMQScheduledComponent.java:284) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:122) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_171]

20:13:45,652 WARN  [org.apache.activemq.artemis.core.server] (Thread-17 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@4de066d8)) AMQ222010: Critical IO Error, shutting down the server. file=NULL, message=Critical IO Error.  Failed to process JDBC Record statements: java.sql.BatchUpdateException: IO Error: Socket read timed out
        at oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:10032)
        at oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1364)
        at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9839)
        at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:234)
        at org.jboss.jca.adapters.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:1180)
        at org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl.sync(JDBCJournalImpl.java:228) [artemis-jdbc-store-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl$JDBCJournalSync.run(JDBCJournalImpl.java:991) [artemis-jdbc-store-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$2.run(ActiveMQScheduledComponent.java:284) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:122) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_171]

20:13:45,655 WARN  [org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl] (Thread-17 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@4de066d8)) java.sql.SQLRecoverableException: Closed Connection

      This thing can be very frustrating for customers who would rely on JDBC store in production. We should provide more robust solution and guide how to configure JDBC store to avoid this situation.

      This problem will arise also when moving to OpenShift which runs in cloud environment with unreliable and slow network to database like Amazon AWS, MS Azure or OpenStack.

              ehugonne1@redhat.com Emmanuel Hugonnet
              mnovak1@redhat.com Miroslav Novak
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: