Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-26042

(8.0.z) Hotrod : Cache inconsistency

XMLWordPrintable

    • False
    • None
    • False
    • +
    • Known Issue
    • Hide

      prepare datagrid nodes

      DATAGRID_ZIP=redhat-datagrid-8.4.5-server.zip
      DATAGRID_FOLDER=redhat-datagrid-8.4.5-server
      
      rm -rdf ${DATAGRID_FOLDER}_1
      rm -rdf ${DATAGRID_FOLDER}_2
      
      unzip -q $DATAGRID_ZIP
      cp -r $DATAGRID_FOLDER ${DATAGRID_FOLDER}_2
      mv $DATAGRID_FOLDER ${DATAGRID_FOLDER}_1
      
      ${DATAGRID_FOLDER}_1/bin/cli.sh user create admin -p pass.1234
      ${DATAGRID_FOLDER}_2/bin/cli.sh user create admin -p pass.1234
      

      start datagrid node 1:

      DATAGRID_FOLDER=redhat-datagrid-8.4.5-server
      ./${DATAGRID_FOLDER}_1/bin/server.sh --bind-address=127.0.0.1 --server-config=infinispan.xml --cluster-stack=tcp --cluster-name=cluster_234.99.54.25 --node-name=jdg1 -Dinfinispan.socket.binding.port-offset=100
      

      start datagrid node 2:

      DATAGRID_FOLDER=redhat-datagrid-8.4.5-server
      ./${DATAGRID_FOLDER}_2/bin/server.sh --bind-address=127.0.0.1 --server-config=infinispan.xml --cluster-stack=tcp --cluster-name=cluster_234.99.54.25 --node-name=jdg2 -Dinfinispan.socket.binding.port-offset=200
      

      prepare eap nodes

      EAP_ZIP=jboss-eap-8.0.0.GA-redhat-20231106.zip
      EAP_FOLDER=jboss-eap-8.0
      
      rm -rdf ${EAP_FOLDER}_1
      rm -rdf ${EAP_FOLDER}_2
      
      unzip -q $EAP_ZIP
      cp -r $EAP_FOLDER ${EAP_FOLDER}_2
      mv $EAP_FOLDER ${EAP_FOLDER}_1
      
      cat <<EOF > $PWD/test1.cli
      embed-server --server-config=standalone-ha.xml
      if (outcome != success) of /subsystem=jgroups:read-attribute(name=default-stack)
        /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
      else
        /subsystem=jgroups:write-attribute(name=default-stack,value=tcp)
        /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
      end-if
      /subsystem=transactions:write-attribute(name=node-identifier,value=wildfly1)
      /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-jdg-server1:add(host=127.0.0.1, port=11322)
      /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-jdg-server2:add(host=127.0.0.1, port=11422)
      
      batch
      /subsystem=infinispan/remote-cache-container=session_data_cc:add(default-remote-cluster=jdg-server-cluster, protocol-version=3.1, statistics-enabled=true, properties={infinispan.client.hotrod.auth_username=admin, infinispan.client.hotrod.auth_password=pass.1234})
      /subsystem=infinispan/remote-cache-container=session_data_cc/remote-cluster=jdg-server-cluster:add(socket-bindings=[remote-jdg-server1,remote-jdg-server2])
      run-batch
      if (outcome == success) of /subsystem=infinispan/remote-cache-container=session_data_cc:read-attribute(name=modules)
          /subsystem=infinispan/remote-cache-container=session_data_cc:write-attribute(name=modules,value=[org.wildfly.clustering.web.hotrod])
      else
          /subsystem=infinispan/remote-cache-container=session_data_cc:write-attribute(name=module,value=org.wildfly.clustering.web.hotrod)
      end-if
      /subsystem=distributable-web/hotrod-session-management=sm_offload:add(remote-cache-container=session_data_cc, granularity=ATTRIBUTE)
      /subsystem=distributable-web/hotrod-session-management=sm_offload/affinity=local:add()
      /subsystem=distributable-web/hotrod-session-management=sm_offload_granular:add(remote-cache-container=session_data_cc, granularity=ATTRIBUTE)
      /subsystem=distributable-web/hotrod-session-management=sm_offload_granular/affinity=local:add()
      if (outcome == success) of /subsystem=infinispan/remote-cache-container=session_data_cc:read-children-resources(child-type=near-cache)
      /subsystem=infinispan/remote-cache-container=session_data_cc/near-cache=invalidation:add(max-entries=1000)
      end-if
      /subsystem=distributable-web:write-attribute(name=default-session-management,value=sm_offload_granular)
      if (outcome == success) of /subsystem=infinispan/remote-cache-container=session_data_cc:read-attribute(name=marshaller)
      /subsystem=infinispan/remote-cache-container=session_data_cc:write-attribute(name=marshaller,value=PROTOSTREAM)
      end-if
      EOF
      
      ./${EAP_FOLDER}_1/bin/jboss-cli.sh --file=$PWD/test1.cli
      
      cat <<EOF > $PWD/test2.cli
      embed-server --server-config=standalone-ha.xml
      if (outcome != success) of /subsystem=jgroups:read-attribute(name=default-stack)
        /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
      else
        /subsystem=jgroups:write-attribute(name=default-stack,value=tcp)
        /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
      end-if
      /subsystem=transactions:write-attribute(name=node-identifier,value=wildfly2)
      /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-jdg-server1:add(host=127.0.0.1, port=11322)
      /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-jdg-server2:add(host=127.0.0.1, port=11422)
      
      batch
      /subsystem=infinispan/remote-cache-container=session_data_cc:add(default-remote-cluster=jdg-server-cluster, protocol-version=3.1, statistics-enabled=true, properties={infinispan.client.hotrod.auth_username=admin, infinispan.client.hotrod.auth_password=pass.1234})
      /subsystem=infinispan/remote-cache-container=session_data_cc/remote-cluster=jdg-server-cluster:add(socket-bindings=[remote-jdg-server1,remote-jdg-server2])
      run-batch
      if (outcome == success) of /subsystem=infinispan/remote-cache-container=session_data_cc:read-attribute(name=modules)
          /subsystem=infinispan/remote-cache-container=session_data_cc:write-attribute(name=modules,value=[org.wildfly.clustering.web.hotrod])
      else
          /subsystem=infinispan/remote-cache-container=session_data_cc:write-attribute(name=module,value=org.wildfly.clustering.web.hotrod)
      end-if
      /subsystem=distributable-web/hotrod-session-management=sm_offload:add(remote-cache-container=session_data_cc, granularity=ATTRIBUTE)
      /subsystem=distributable-web/hotrod-session-management=sm_offload/affinity=local:add()
      /subsystem=distributable-web/hotrod-session-management=sm_offload_granular:add(remote-cache-container=session_data_cc, granularity=ATTRIBUTE)
      /subsystem=distributable-web/hotrod-session-management=sm_offload_granular/affinity=local:add()
      if (outcome == success) of /subsystem=infinispan/remote-cache-container=session_data_cc:read-children-resources(child-type=near-cache)
      /subsystem=infinispan/remote-cache-container=session_data_cc/near-cache=invalidation:add(max-entries=1000)
      end-if
      /subsystem=distributable-web:write-attribute(name=default-session-management,value=sm_offload_granular)
      if (outcome == success) of /subsystem=infinispan/remote-cache-container=session_data_cc:read-attribute(name=marshaller)
      /subsystem=infinispan/remote-cache-container=session_data_cc:write-attribute(name=marshaller,value=PROTOSTREAM)
      end-if
      EOF
      
      ./${EAP_FOLDER}_2/bin/jboss-cli.sh --file=$PWD/test2.cli
      
      export CLUSTERBENCH_EAR=clusterbench-ee10.ear
      cp $CLUSTERBENCH_EAR ${EAP_FOLDER}_1/standalone/deployments/
      cp $CLUSTERBENCH_EAR ${EAP_FOLDER}_2/standalone/deployments/
      

      start eap node 1:

      EAP_FOLDER=jboss-eap-8.0
      ./${EAP_FOLDER}_1/bin/standalone.sh -b=127.0.0.1 -bmanagement=127.0.0.1 -bprivate=127.0.0.1 --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.118 -Dprogram.name=wildfly1 -Djboss.node.name=wildfly1 -Djboss.socket.binding.port-offset=100
      

      start eap node 2:

      EAP_FOLDER=jboss-eap-8.0
      ./${EAP_FOLDER}_2/bin/standalone.sh -b=127.0.0.1 -bmanagement=127.0.0.1 -bprivate=127.0.0.1 --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.118 -Dprogram.name=wildfly1 -Djboss.node.name=wildfly2 -Djboss.socket.binding.port-offset=200
      

      invoke clusterbench on node 1

      curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8180/clusterbench/session
      curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8180/clusterbench/session
      curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8180/clusterbench/session
      curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8180/clusterbench/session
      

      You can see the ouput is 0111 which means session is established but serial isn't incremented!
      If you use your favourite browser the resut is the same.

      invoke clusterbench on node 2

      curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8280/clusterbench/session
      curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8280/clusterbench/session
      curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8280/clusterbench/session
      curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8280/clusterbench/session
      

      You can see the ouput is 1111 which means session is propagated to the other node but serial isn't incremented!
      If you use your favourite browser the resut is the same.

      Show
      prepare datagrid nodes DATAGRID_ZIP=redhat-datagrid-8.4.5-server.zip DATAGRID_FOLDER=redhat-datagrid-8.4.5-server rm -rdf ${DATAGRID_FOLDER}_1 rm -rdf ${DATAGRID_FOLDER}_2 unzip -q $DATAGRID_ZIP cp -r $DATAGRID_FOLDER ${DATAGRID_FOLDER}_2 mv $DATAGRID_FOLDER ${DATAGRID_FOLDER}_1 ${DATAGRID_FOLDER}_1/bin/cli.sh user create admin -p pass.1234 ${DATAGRID_FOLDER}_2/bin/cli.sh user create admin -p pass.1234 start datagrid node 1: DATAGRID_FOLDER=redhat-datagrid-8.4.5-server ./${DATAGRID_FOLDER}_1/bin/server.sh --bind-address=127.0.0.1 --server-config=infinispan.xml --cluster-stack=tcp --cluster-name=cluster_234.99.54.25 --node-name=jdg1 -Dinfinispan.socket.binding.port-offset=100 start datagrid node 2: DATAGRID_FOLDER=redhat-datagrid-8.4.5-server ./${DATAGRID_FOLDER}_2/bin/server.sh --bind-address=127.0.0.1 --server-config=infinispan.xml --cluster-stack=tcp --cluster-name=cluster_234.99.54.25 --node-name=jdg2 -Dinfinispan.socket.binding.port-offset=200 prepare eap nodes EAP_ZIP=jboss-eap-8.0.0.GA-redhat-20231106.zip EAP_FOLDER=jboss-eap-8.0 rm -rdf ${EAP_FOLDER}_1 rm -rdf ${EAP_FOLDER}_2 unzip -q $EAP_ZIP cp -r $EAP_FOLDER ${EAP_FOLDER}_2 mv $EAP_FOLDER ${EAP_FOLDER}_1 cat <<EOF > $PWD/test1.cli embed-server --server-config=standalone-ha.xml if (outcome != success) of /subsystem=jgroups:read-attribute(name=default-stack) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) else /subsystem=jgroups:write-attribute(name=default-stack,value=tcp) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) end-if /subsystem=transactions:write-attribute(name=node-identifier,value=wildfly1) /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-jdg-server1:add(host=127.0.0.1, port=11322) /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-jdg-server2:add(host=127.0.0.1, port=11422) batch /subsystem=infinispan/remote-cache-container=session_data_cc:add(default-remote-cluster=jdg-server-cluster, protocol-version=3.1, statistics-enabled=true, properties={infinispan.client.hotrod.auth_username=admin, infinispan.client.hotrod.auth_password=pass.1234}) /subsystem=infinispan/remote-cache-container=session_data_cc/remote-cluster=jdg-server-cluster:add(socket-bindings=[remote-jdg-server1,remote-jdg-server2]) run-batch if (outcome == success) of /subsystem=infinispan/remote-cache-container=session_data_cc:read-attribute(name=modules) /subsystem=infinispan/remote-cache-container=session_data_cc:write-attribute(name=modules,value=[org.wildfly.clustering.web.hotrod]) else /subsystem=infinispan/remote-cache-container=session_data_cc:write-attribute(name=module,value=org.wildfly.clustering.web.hotrod) end-if /subsystem=distributable-web/hotrod-session-management=sm_offload:add(remote-cache-container=session_data_cc, granularity=ATTRIBUTE) /subsystem=distributable-web/hotrod-session-management=sm_offload/affinity=local:add() /subsystem=distributable-web/hotrod-session-management=sm_offload_granular:add(remote-cache-container=session_data_cc, granularity=ATTRIBUTE) /subsystem=distributable-web/hotrod-session-management=sm_offload_granular/affinity=local:add() if (outcome == success) of /subsystem=infinispan/remote-cache-container=session_data_cc:read-children-resources(child-type=near-cache) /subsystem=infinispan/remote-cache-container=session_data_cc/near-cache=invalidation:add(max-entries=1000) end-if /subsystem=distributable-web:write-attribute(name=default-session-management,value=sm_offload_granular) if (outcome == success) of /subsystem=infinispan/remote-cache-container=session_data_cc:read-attribute(name=marshaller) /subsystem=infinispan/remote-cache-container=session_data_cc:write-attribute(name=marshaller,value=PROTOSTREAM) end-if EOF ./${EAP_FOLDER}_1/bin/jboss-cli.sh --file=$PWD/test1.cli cat <<EOF > $PWD/test2.cli embed-server --server-config=standalone-ha.xml if (outcome != success) of /subsystem=jgroups:read-attribute(name=default-stack) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) else /subsystem=jgroups:write-attribute(name=default-stack,value=tcp) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) end-if /subsystem=transactions:write-attribute(name=node-identifier,value=wildfly2) /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-jdg-server1:add(host=127.0.0.1, port=11322) /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-jdg-server2:add(host=127.0.0.1, port=11422) batch /subsystem=infinispan/remote-cache-container=session_data_cc:add(default-remote-cluster=jdg-server-cluster, protocol-version=3.1, statistics-enabled=true, properties={infinispan.client.hotrod.auth_username=admin, infinispan.client.hotrod.auth_password=pass.1234}) /subsystem=infinispan/remote-cache-container=session_data_cc/remote-cluster=jdg-server-cluster:add(socket-bindings=[remote-jdg-server1,remote-jdg-server2]) run-batch if (outcome == success) of /subsystem=infinispan/remote-cache-container=session_data_cc:read-attribute(name=modules) /subsystem=infinispan/remote-cache-container=session_data_cc:write-attribute(name=modules,value=[org.wildfly.clustering.web.hotrod]) else /subsystem=infinispan/remote-cache-container=session_data_cc:write-attribute(name=module,value=org.wildfly.clustering.web.hotrod) end-if /subsystem=distributable-web/hotrod-session-management=sm_offload:add(remote-cache-container=session_data_cc, granularity=ATTRIBUTE) /subsystem=distributable-web/hotrod-session-management=sm_offload/affinity=local:add() /subsystem=distributable-web/hotrod-session-management=sm_offload_granular:add(remote-cache-container=session_data_cc, granularity=ATTRIBUTE) /subsystem=distributable-web/hotrod-session-management=sm_offload_granular/affinity=local:add() if (outcome == success) of /subsystem=infinispan/remote-cache-container=session_data_cc:read-children-resources(child-type=near-cache) /subsystem=infinispan/remote-cache-container=session_data_cc/near-cache=invalidation:add(max-entries=1000) end-if /subsystem=distributable-web:write-attribute(name=default-session-management,value=sm_offload_granular) if (outcome == success) of /subsystem=infinispan/remote-cache-container=session_data_cc:read-attribute(name=marshaller) /subsystem=infinispan/remote-cache-container=session_data_cc:write-attribute(name=marshaller,value=PROTOSTREAM) end-if EOF ./${EAP_FOLDER}_2/bin/jboss-cli.sh --file=$PWD/test2.cli export CLUSTERBENCH_EAR=clusterbench-ee10.ear cp $CLUSTERBENCH_EAR ${EAP_FOLDER}_1/standalone/deployments/ cp $CLUSTERBENCH_EAR ${EAP_FOLDER}_2/standalone/deployments/ start eap node 1: EAP_FOLDER=jboss-eap-8.0 ./${EAP_FOLDER}_1/bin/standalone.sh -b=127.0.0.1 -bmanagement=127.0.0.1 -bprivate=127.0.0.1 --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.118 -Dprogram.name=wildfly1 -Djboss.node.name=wildfly1 -Djboss.socket.binding.port-offset=100 start eap node 2: EAP_FOLDER=jboss-eap-8.0 ./${EAP_FOLDER}_2/bin/standalone.sh -b=127.0.0.1 -bmanagement=127.0.0.1 -bprivate=127.0.0.1 --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.118 -Dprogram.name=wildfly1 -Djboss.node.name=wildfly2 -Djboss.socket.binding.port-offset=200 invoke clusterbench on node 1 curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8180/clusterbench/session curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8180/clusterbench/session curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8180/clusterbench/session curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8180/clusterbench/session You can see the ouput is 0111 which means session is established but serial isn't incremented! If you use your favourite browser the resut is the same. invoke clusterbench on node 2 curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8280/clusterbench/session curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8280/clusterbench/session curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8280/clusterbench/session curl -b /tmp/cookies1 -c /tmp/cookies1 http://localhost:8280/clusterbench/session You can see the ouput is 1111 which means session is propagated to the other node but serial isn't incremented! If you use your favourite browser the resut is the same.

      Using two clustered datagrid nodes of version 8.4.5 and a cluster or two nodes of EAP 8.0.0 (most recent nightly build - closest thing we have to CR2) the following error is observed when accessing /clusterbench/session more than twice .... The serial does not increase above one:

      2023-11-08 14:47:49,835 ERROR o.j.e.c.j.ClusteringHTTPRequestSampler: Invalid serial: Expected 2, received 1 (previous: JSessionId{sessionId='R0iancqo7ooafzgjzsHovXYYjo8NRqnrrrrRl2Yw', jvmRoutes=[wildfly1]} current: JSessionId{sessionId='R0iancqo7ooafzgjzsHovXYYjo8NRqnrrrrRl2Yw', jvmRoutes=[wildfly1]})
      samples sequence:
      14:47:41.742 serial 0 JSessionId{sessionId='R0iancqo7ooafzgjzsHovXYYjo8NRqnrrrrRl2Yw', jvmRoutes=[wildfly1]}
      14:47:45.816 serial 1 JSessionId{sessionId='R0iancqo7ooafzgjzsHovXYYjo8NRqnrrrrRl2Yw', jvmRoutes=[wildfly1]}
      14:47:49.835 serial 1 JSessionId{sessionId='R0iancqo7ooafzgjzsHovXYYjo8NRqnrrrrRl2Yw', jvmRoutes=[wildfly1]}
      

      Note that the session ID and the node handling the requests do not change;

      Find attached complete EAP and JDG logs and config files;

        1. clusterbench-ee10.ear
          75 kB
          Tommaso Borgato

            pferraro@redhat.com Paul Ferraro
            tborgato@redhat.com Tommaso Borgato
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: