Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-14737

SoftIndexFileStore Index can become corrupted

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • 15.0.0.Dev01, 14.0.8.Final
    • None
    • None
    • None
    • Hide

      Have a 5 Node VM cluster with (64 Gb Ram and 8 vCPUs) on Azure and 2 TB SSD attached to each VM.

      Installed ISPN from https://download.eng.bos.redhat.com/rcm-guest/staging/jboss-dg/JDG-8.4.x-CD/ on March 23rd.

      As you will see infinispan.xml, all the clusters are being set up with 58gb off-heap and SYNC as a replication strategy for data.

      Once the cluster is up, I used LB to start the benchmark utility using the below command:

      nohup ./redis-benchmark -t set,get -d 9000 -r 100,000,000 -n 150000000 -h 20.253.122.250 -p 11222 --user developer -a developer &

      And soon I started seeing errors in the log file within a few mins.

       

      NOTE: I checked infinispan.xml for all 5 nodes and noticed that I had SYNC on 1 VM but ASYNC on the other 4 VMs. Thought of sharing it if that helps.

       

       

       

       

      server.zip

      Show
      Have a 5 Node VM cluster with (64 Gb Ram and 8 vCPUs) on Azure and 2 TB SSD attached to each VM. Installed ISPN from https://download.eng.bos.redhat.com/rcm-guest/staging/jboss-dg/JDG-8.4.x-CD/ on March 23rd. As you will see infinispan.xml, all the clusters are being set up with 58gb off-heap and SYNC as a replication strategy for data. Once the cluster is up, I used LB to start the benchmark utility using the below command: nohup ./redis-benchmark -t set,get -d 9000 -r 100,000,000 -n 150000000 -h 20.253.122.250 -p 11222 --user developer -a developer & And soon I started seeing errors in the log file within a few mins.   NOTE : I checked infinispan.xml for all 5 nodes and noticed that I had SYNC on 1 VM but ASYNC on the other 4 VMs. Thought of sharing it if that helps.         server.zip

    Description

      Getting this error in my 5-node cluster created on Azure VMs. Before starting redis-benchmark cli test I clean up my data directory using /bin/cli.sh and restarted the cluster. Attaching the complete server folder with logs and thread dumps.

      Attachments

        1. jstack_threaddump.out
          5.09 MB
        2. server.zip
          1.45 MB

        Issue Links

          Activity

            People

              rh-ee-jbolina Jose Bolina
              ravirhsharma Ravi Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: