Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-5031

Scaling up 8.3 cluster with payload is slower compared to 8.2

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • RHDG 8.3 CD
    • RHDG 8.3 CD
    • Openshift Images
    • None
    • False
    • False
    • Hide

      Create a cluster:

      kind: Infinispan
apiVersion: infinispan.org/v1
metadata: 
  name: infinispan
  namespace: datagrid-suite-upgrade
spec: 
  expose: 
    type: Route
  replicas: 3
  security: 
    endpointAuthentication: false
    endpointEncryption: 
      type: None
  container: 
    cpu: 2000m
  service: 
    type: DataGrid

      Once up create persisted cache:

      {
  "distributed-cache": {
    "mode":"SYNC", 
    "encoding": {
      "media-type": "application/json"
    },
    "persistence": {
       "file-store":{}
    },
    "statistics":true
  }
}

      Upload data (ideally from within the container to speed it up):

      for x in {1..5000}
do
        curl -X PUT -d "{\"value\":\"${x}\"}" -H "Content-Type: text/plain" http://infinispan:11222/rest/v2/caches/testcache/key-${x}
done

      Scale cluster down to 1 and then back to 3. You'll notice second node is taking time to boot.

      Show
      Create a cluster: kind: Infinispan apiVersion: infinispan.org/v 1 metadata: name: infinispan namespace: datagrid-suite-upgrade spec: expose: type: Route replicas: 3 security: endpointAuthentication: false endpointEncryption: type: None container: cpu: 2000 m service: type: DataGrid Once up create persisted cache: { "distributed-cache" : { "mode" : "SYNC" , "encoding" : { "media-type" : "application/json" }, "persistence" : { "file-store" :{} }, "statistics" : true } } Upload data (ideally from within the container to speed it up): for x in {1..5000} do curl -X PUT -d "{\" value\ ":\" ${x}\ "}" -H "Content-Type: text/plain" http: //infinispan:11222/ rest /v2/caches/testcache/key-${x} done Scale cluster down to 1 and then back to 3. You'll notice second node is taking time to boot.

    Description

      If there cache with data exist and user scales up a cluster then the time to bring up another node is way higher compared to 8.2 and may even result in Initial state transfer timeout.

      While 8.2 clusters with 5000 persisted json entries is brough up in constant time, time required by 8.3 is at least 3 times longer potentially timing out.

      Attachments

        1. 1014523.txt
          178 kB
        2. infinispan-0-infinispan.log
          22 kB
        3. infinispan-1-infinispan.log
          5 kB
        4. log-0-8.2
          8.65 MB
        5. log-0-full
          11.84 MB
        6. log-restared-8.2
          3.38 MB
        7. log-restart-full
          3.83 MB

        Issue Links

          is caused by

          Bug - A problem that impairs or prevents the functions of the product. JGRP-2597 LEAVE-RSP is sent to leaver after leaver has left

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. ISPN-13625 Scaling up 8.3 cluster with payload is slower compared to 8.2

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. JDG-5026 [Operator] Cluster is unable to start after graceful shutdown on OCP4.9

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Activity

            People

              wburns@redhat.com Will Burns
              pdrobek@redhat.com Pavel Drobek
              Pavel Drobek Pavel Drobek
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: