Uploaded image for project: 'JBoss Enterprise Application Platform 4 and 5'
  1. JBoss Enterprise Application Platform 4 and 5
  2. JBPAPP-6656

ConcurrentModificationException during failover

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • EAP_EWP 5.1.1 ER3
    • EAP_EWP 5.1.1 ER2
    • Messaging
    • None
    • NEW

    Description

      Hi,

      I tried following test scenario:

      1. Start cluster of 4 nodes - A,B,C,D
      2. Start 4 Producers and 4 Consumers on one clustered queue - clients use ClusteredConnectionFactory with client_ack session.
      3. Kill server A
      4. Kill server B
      5. Kill server C
      6. Start A
      7. Kill D
      8. Stop producers

      Clients should be reconnected to other nodes in cluster but during killing server sometimes happens that client won't get reconnected to other node and ends with exception. I see following exception in server log:

      [JBoss] 12:17:44,257 ERROR [ExceptionUtil] SessionEndpoint[we-7mhcheog-1-y9b6heog-sfream-y1i2ga] recoverDeliveries [8f-pmhcheog-1-y9b6heog-sfream-y1i2ga]
      [JBoss] java.util.ConcurrentModificationException
      [JBoss] at java.util.LinkedHashMap$LinkedHashIterator.remove(LinkedHashMap.java:364)
      [JBoss] at org.jboss.messaging.core.impl.MessagingQueue.removeStrandedReferences(MessagingQueue.java:355)
      [JBoss] at org.jboss.jms.server.endpoint.ServerSessionEndpoint.recoverDeliveries(ServerSessionEndpoint.java:667)
      [JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$recoverDeliveries$aop(SessionAdvised.java:150)
      [JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised$recoverDeliveries_4196799148363996265.invokeTarget(SessionAdvised$recoverDeliveries_4196799148363996265.java)
      [JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
      [JBoss] at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
      [JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
      [JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised.recoverDeliveries(SessionAdvised.java)
      [JBoss] at org.jboss.jms.wireformat.SessionRecoverDeliveriesRequest.serverInvoke(SessionRecoverDeliveriesRequest.java:96)
      [JBoss] at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:165)
      [JBoss] at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)
      [JBoss] at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:791)
      [JBoss] at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
      [JBoss] at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:586)
      [JBoss] at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
      [JBoss] 12:17:44,257 ERROR [ExceptionUtil] SessionEndpoint[ye-7mhcheog-1-y9b6heog-sfream-y1i2ga] recoverDeliveries [6f-pmhcheog-1-y9b6heog-sfream-y1i2ga]
      [JBoss] java.util.ConcurrentModificationException
      [JBoss] at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
      [JBoss] at java.util.LinkedHashMap$ValueIterator.next(LinkedHashMap.java:388)
      [JBoss] at org.jboss.messaging.core.impl.MessagingQueue.removeStrandedReferences(MessagingQueue.java:347)
      [JBoss] at org.jboss.jms.server.endpoint.ServerSessionEndpoint.recoverDeliveries(ServerSessionEndpoint.java:667)
      [JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$recoverDeliveries$aop(SessionAdvised.java:150)
      [JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised$recoverDeliveries_4196799148363996265.invokeTarget(SessionAdvised$recoverDeliveries_4196799148363996265.java)
      [JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
      [JBoss] at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
      [JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
      [JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised.recoverDeliveries(SessionAdvised.java)
      [JBoss] at org.jboss.jms.wireformat.SessionRecoverDeliveriesRequest.serverInvoke(SessionRecoverDeliveriesRequest.java:96)
      [JBoss] at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:165)
      [JBoss] at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)
      [JBoss] at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:791)
      [JBoss] at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
      [JBoss] at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:586)
      [JBoss] at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
      [JBoss] 12:17:44,257 ERROR [ExceptionUtil] SessionEndpoint[0f-7mhcheog-1-y9b6heog-sfream-y1i2ga] recoverDeliveries [9f-pmhcheog-1-y9b6heog-sfream-y1i2ga]
      [JBoss] java.util.ConcurrentModificationException
      [JBoss] at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
      [JBoss] at java.util.LinkedHashMap$ValueIterator.next(LinkedHashMap.java:388)
      [JBoss] at org.jboss.messaging.core.impl.MessagingQueue.removeStrandedReferences(MessagingQueue.java:347)
      [JBoss] at org.jboss.jms.server.endpoint.ServerSessionEndpoint.recoverDeliveries(ServerSessionEndpoint.java:667)
      [JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$recoverDeliveries$aop(SessionAdvised.java:150)
      [JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised$recoverDeliveries_4196799148363996265.invokeTarget(SessionAdvised$recoverDeliveries_4196799148363996265.java)
      [JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
      [JBoss] at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
      [JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
      [JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised.recoverDeliveries(SessionAdvised.java)
      [JBoss] at org.jboss.jms.wireformat.SessionRecoverDeliveriesRequest.serverInvoke(SessionRecoverDeliveriesRequest.java:96)
      [JBoss] at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:165)
      [JBoss] at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)
      [JBoss] at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:791)
      [JBoss] at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
      [JBoss] at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:586)
      [JBoss] at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)

      It seems that there is access to "recoveryMap = Collections.synchronizedMap(new LinkedHashMap());" during iteration.

      Could you look at it please?

      Thank you,

      Mirek

      Attachments

        1. client_exception
          6 kB
        2. jboss-messaging.jar
          1.15 MB
        3. jboss-messaging-client.jar
          578 kB

        Issue Links

          is duplicated by

          Bug - A problem that impairs or prevents the functions of the product. JBPAPP-6685 MessagingJMSException during failover

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. JBPAPP-6685 MessagingJMSException during failover

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Activity

            People

              gaohoward Howard Gao
              mnovak1@redhat.com Miroslav Novak
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: