While InfiniSpan is very fast in-memory, once passivation is turned on, things get slow for me.

On my i5-2400 with a Crucial C300 SSD, it takes about 11 ms to passivate a 2964 B value. This comes down to a write speed of 250 to 300 KB/s. The disk can write up to 180 MB/s so it's not the bottleneck.

Clearly, passivation should be faster than that.

Profiling with YourKit shows most of the hot spots are centered around FileCacheStore, the JBoss Marshalling library and various bucket and ImmortalCacheEntry methods. Locking also seems to play a role. Here's a screenshot:

To make profiling easier for you, I've taken my test case and turned it into a small Maven project. Simply running mvn install should run the test, ready for profiling.

Of course, it could also be that my configuration is suboptimal. However, in my experiments, many configurations were unusable (hitting the file handle limit), which is not a good thing.

The project is available at http://johannburkard.de/resources/Johann/infinispan-performance.zip

I've also found one or two other things I think are more costly than they would need to be

In FileCacheStore

         byte[] buffer = new byte[streamBufferSize];
...
               fileInStream = new FileInputStream(file);
...
               bis = new BufferedInputStream(fileInStream);
...
                  bytesRead = bis.read(buffer, 0, streamBufferSize);

wrapping the FileInputStream in a BufferedInputStream is unnecessary because you already have a buffer in byte[] buffer. I think I have seen this before in InfiniSpan, so you might want to check for occurrences of BufferedInputStream or BufferedOutputStream.

Another one

         if (bytes.length == 0) {
            // Short circuit
            if (f.exists()) f.delete();
            return;
         }

f.exists() is essentially a syscall. Just calling f.delete() is enough.

      if (!root.exists()) {
         if (!root.mkdirs()) {
            log.problemsCreatingDirectory(root);
         }
      }
      if (!root.exists()) {
         throw new ConfigurationException("Directory " + root.getAbsolutePath() + " does not exist and cannot be created!");
      }

This could also be shortened to

      if (!root.mkdirs()) {
         throw new ConfigurationException("Directory " + root.getAbsolutePath() + " does not exist and cannot be created!");
      }