Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-21757

[JDG-4351][JBMAR-235] camel-infinispan requires jboss-marshalling update from 2.0.9.Final to 2.0.11.Final onwards

XMLWordPrintable

    • False
    • None
    • False
    • % %
    • Todo
    • Workaround Exists
    • Hide

      You can workaround this issue by replacing jboss-marshalling with a newer version (2.0.11.Final-redhat-00001 or later) manually by custom build setup in your pom.xml.
      See https://access.redhat.com/solutions/7025567 (or README.md in the attached reproducer-camel-infinispan.zip) for details about the workaround.

      Show
      You can workaround this issue by replacing jboss-marshalling with a newer version (2.0.11.Final-redhat-00001 or later) manually by custom build setup in your pom.xml. See https://access.redhat.com/solutions/7025567 (or README.md in the attached reproducer-camel-infinispan.zip) for details about the workaround.
    • Very Likely

      Due to the following known issue in JBoss Marshalling before 2.0.11.Final, there is a possibility of performance issue for camel-infinispan.

      https://issues.redhat.com/browse/JDG-4351
      https://issues.redhat.com/browse/JBMAR-235

      The following is pseudo-code to reproduce the issue. Note that it's required to use the same RemoteCacheManager object and execute all the following operations in the same thread.

      - PUTALL with Map object of a certain size  // no performance issue
      - PUT with a huge size List object
      - PUTALL with Map object of a certain size  // performance issue occurs.
      

      To implement this pseudo-code, the reproducer uses a loop to execute 3 PUTALL operations with a Map object of a certain size, 1 PUT operation with a huge size of List object, then 3 PUTALL operations with Map object of a certain size again. For example:

      public class MyRouteBuilder extends RouteBuilder {
      
          @Override
          public void configure() throws Exception {
              Map<String, String> map = new HashMap<>();
              for(int i = 0; i < 3000; i++) {
                  map.put("k" + i, "v" + i); 
              }
      
              List<String> list = new ArrayList<>();
              for(int i = 0; i < 1000000; i++) {
                  list.add(new String("foo" + i));
              }
      
              from("timer://foo?repeatCount=1")
                  .log("----------------- JBoss Marshalling Version = " + Version.getVersionString())
                  .setHeader("counter", simple("1"))
                  .loopDoWhile(simple("${header.counter} <= 7"))
                      .choice()
                      .when(simple("${header.counter} == '4'"))
                          .setHeader(InfinispanConstants.OPERATION, constant(InfinispanOperation.PUT))
                          .setHeader(InfinispanConstants.KEY).constant("hugelist")
                          .setHeader(InfinispanConstants.VALUE).constant(list)
                          .log(">>> operation ${header.counter} start --- put() huge list")
                      .otherwise()
                          .setHeader(InfinispanConstants.OPERATION, constant(InfinispanOperation.PUTALL))
                          .setHeader(InfinispanConstants.MAP).constant(map)
                          .log(">>> operation ${header.counter} start --- putAll()")
                      .end()
                      .to("infinispan:distributed?hosts=127.0.0.1:11222")
                      .log("<<< operation ${header.counter} end")
                      .setHeader("counter", simple("${header.counter}++"))
                  .end();
      
          }   
      
      }
      

      Also there are some "Fuse on Openshift" users, this reproducer works for "Fuse on Openshift" too.


      Steps to reproduce the performance issue with the attached reproducer:

      See also README.md in the attached reproducer-camel-infinispan.zip

      First, set up Red Hat Data Grid (RHDG) Server:

      1. Download and unzip a binary from our internal file server: http://download.devel.redhat.com/released/JBoss-middleware/datagrid/8.4.2/redhat-datagrid-8.4.2-server.zip

      2.Modify server/conf/infinispan.xml to create distributed-cache named "distributed" and disable security for testing purpose

      --- server/conf/infinispan.xml.BAK	2023-05-23 11:47:34.319251572 +0900
      +++ server/conf/infinispan.xml	2023-07-24 08:38:13.993779347 +0900
      @@ -7,9 +7,13 @@
       
          <cache-container name="default" statistics="true">
             <transport cluster="${infinispan.cluster.name:cluster}" stack="${infinispan.cluster.stack:tcp}" node-name="${infinispan.node.name:}"/>
      -      <security>
      -         <authorization/>
      -      </security>
      +      <!-- <security> -->
      +      <!--    <authorization/> -->
      +      <!-- </security> -->
      +      <distributed-cache-configuration name="dist-cache-template">
      +         <encoding media-type="application/x-jboss-marshalling"/>
      +      </distributed-cache-configuration>
      +      <distributed-cache name="distributed" configuration="dist-cache-template" statistics="true"/>
          </cache-container>
       
          <server xmlns="urn:infinispan:server:14.0">
      @@ -48,6 +52,7 @@
                </security-realms>
             </security>
       
      -      <endpoints socket-binding="default" security-realm="default" />
      +      <!-- <endpoints socket-binding="default" security-realm="default" /> -->
      +      <endpoints socket-binding="default"/>
          </server>
       </infinispan>
      

      3. Start RHDG Server. Make sure to use JDK 11.

      $ echo $JAVA_HOME
      /usr/lib/jvm/java-11-openjdk
      $ ./bin/server.sh
      

      Then, run camel route with camel-infinispan that accesses the above RHDG Server

      1. Download and unzip the attached reproducer-camel-infinispan.zip
      2. Build the maven project

      mvn clean install
      mvn karaf:assembly
      

      3. Start the build application

      $ ./target/assembly/bin/fuse
      

      4. Check the log (console output or karaf.log).
      You can see the first 3 putAll() operations (1-3) completed within 1 second, but the last 3 putAll() operations (5-7) after put() huge list took aroundabout 15-20 seconds which is very slow comparing with the first 3 operations.

      2023-07-24 18:52:40,933 | INFO  |  Event Dispatcher: 1 | o.i.c.h.RemoteCacheManager       | 127 - org.infinispan.client-hotrod - 9.4.19.Final-redhat-00001 | ISPN004021: Infinispan version: 9.4.19.Final-redhat-00001
      2023-07-24 18:52:40,964 | INFO  |  Event Dispatcher: 1 | o.a.c.b.BlueprintCamelContext    | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | Route: route1 started and consuming from: timer://foo?repeatCount=1
      2023-07-24 18:52:40,967 | INFO  |  Event Dispatcher: 1 | o.a.c.b.BlueprintCamelContext    | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | Total 1 routes, of which 1 are started
      2023-07-24 18:52:40,969 | INFO  |  Event Dispatcher: 1 | o.a.c.b.BlueprintCamelContext    | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | Apache Camel 2.23.2.fuse-7_11_1-00015-redhat-00002 (CamelContext: log-example-context) started in 1.714 seconds
      2023-07-24 18:52:41,993 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | ----------------- JBoss Marshalling Version = 2.0.9.Final
      2023-07-24 18:52:42,000 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | ----------------- Infinispan Version = 9.4.19.Final-redhat-00001
      2023-07-24 18:52:42,018 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | >>> operation 1 start --- putAll()
      2023-07-24 18:52:42,068 | INFO  | lient-async-pool-1-1 | o.i.c.h.i.p.Codec21              | 127 - org.infinispan.client-hotrod - 9.4.19.Final-redhat-00001 | ISPN004006: Server sent new topology view (id=727952508, age=0) containing 1 addresses: [127.0.0.1:11222]
      2023-07-24 18:52:42,264 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | <<< operation 1 end
      2023-07-24 18:52:42,269 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | >>> operation 2 start --- putAll()
      2023-07-24 18:52:42,316 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | <<< operation 2 end
      2023-07-24 18:52:42,318 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | >>> operation 3 start --- putAll()
      2023-07-24 18:52:42,367 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | <<< operation 3 end
      2023-07-24 18:52:42,369 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | >>> operation 4 start --- put() huge list
      2023-07-24 18:52:45,031 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | <<< operation 4 end
      2023-07-24 18:52:45,033 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | >>> operation 5 start --- putAll()
      2023-07-24 18:53:06,940 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | <<< operation 5 end
      2023-07-24 18:53:06,942 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | >>> operation 6 start --- putAll()
      2023-07-24 18:53:24,899 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | <<< operation 6 end
      2023-07-24 18:53:24,902 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | >>> operation 7 start --- putAll()
      2023-07-24 18:53:44,345 | INFO  | ead #1 - timer://foo | route1                           | 119 - org.apache.camel.camel-core - 2.23.2.fuse-7_11_1-00015-redhat-00002 | <<< operation 7 end
      

              ggrzybek Grzegorz Grzybek
              rhn-support-hfuruich Hisao Furuichi
              Federico Mariani Federico Mariani
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: