Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-14545

SIFS Compactor does not properly shut down but the index thinks it is okay


      The SIFS Compactor doesn't attempt to shutdown in an orderly fashion. Here are some glaring issues.

      1. It sets the logFile to null without verifying the compaction background thread has been completed.
      2. It does not update its log files with the log file it was creating and thus it is not part of the file stats on restart.
      3. If a file needs to be compacted at startup (server was shut down before compaction could complete) then it is possible to get a NPE from the key not being in the index because the index isn't started yet!

      These can have random affects, but usually what happens is a user will see an ERROR in the log about an orphaned data file. This in itself is not an issue, but it can sometimes cause the index to then become corrupted and later some operations may cause index lookup failures.

      The error usually encountered in log file is as follows:

      2023-02-27 19:31:17,633 ERROR (blocking-thread-infini11.xxx.de-p3-t8) [org.infinispan.persistence.sifs.Compactor] ISPN029021: File id 44879 encountered an exception while compacting, file may be orphaned java.lang.IllegalStateException: Error reading header from 16214:15006755 | 0
      at org.infinispan.persistence.sifs.IndexNode$LeafNode.getHeaderAndKey(IndexNode.java:1113)
      at org.infinispan.persistence.sifs.IndexNode$LeafNode.loadHeaderAndKey(IndexNode.java:1093)
      at org.infinispan.persistence.sifs.IndexNode$ReadOperation$4.apply(IndexNode.java:277)
      at org.infinispan.persistence.sifs.IndexNode$ReadOperation$4.apply(IndexNode.java:274)
      at org.infinispan.persistence.sifs.IndexNode.applyOnLeaf(IndexNode.java:317)
      at org.infinispan.persistence.sifs.Index.getInfo(Index.java:240)

      This can be worked around by ensuring your SIFS always has purgeOnStartup as the server is started with empty data and index and can't be corrupted between runs.

            wburns@redhat.com Will Burns
            wburns@redhat.com Will Burns
            0 Vote for this issue
            2 Start watching this issue