SoftIndexFileStore not removing log files with expiration

I have a cache with the following configuration:

ConfigurationBuilder configurationBuilder = new ConfigurationBuilder();
configurationBuilder
        .encoding()
            .mediaType(MediaType.APPLICATION_SERIALIZED_OBJECT_TYPE)
        .memory()
                .maxSize("25MB")
        .expiration()
                .lifespan(2, TimeUnit.HOURS)
        .persistence()
                .passivation(true)
                .addSoftIndexFileStore()
                        .dataLocation(cacheDirectory)
                        .indexLocation(cacheDirectory);

manager.defineConfiguration(Caches.SCHEDULES, configurationBuilder.build()); 

I would expect this cache to remove entries after 2 hours and then perform compaction and log file deletion as necessary.

The cache seems to generate about 1 log file every 5-10 minutes, so 16 MB of data in this interval, the first couple of log files were:

-rw-r--r-- 1 uva_prd uva_prd 16713133 Sep 28 21:43 ispn12.5
-rw-r--r-- 1 uva_prd uva_prd 16772504 Sep 28 21:36 ispn12.4
-rw-r--r-- 1 uva_prd uva_prd 16741848 Sep 28 21:30 ispn12.3
-rw-r--r-- 1 uva_prd uva_prd 16662625 Sep 28 21:22 ispn12.2
-rw-r--r-- 1 uva_prd uva_prd 16738889 Sep 28 21:15 ispn12.1
-rw-r--r-- 1 uva_prd uva_prd 16771927 Sep 28 21:08 ispn12.0 

The application started around 20:50 with an completely empty cache folder (no files present).

Trace level logging on the Compactor shows this: https://gist.github.com/MikeN123/4908f0eb6730a150c6e9bccd58a884d5

Only in the last log file entries are expired, other log files are skipped. For some reason log file 16 is still an active log file as well, even when it's days old, I have no idea why.

The cache is only used for 1 Spring-based cache with an @Cacheable(sync = true) annotation.

Looking at a heap dump I see for Compactor.fileStats that every entry has:

• markedForDeletion = false
• nextExpirationTime = -1
• scheduled = true

The inactive logs have:

• completed = true
• total = 16776575

The active logs (last one + log file 16) have:

• completed = false
• total = -1

The value for free varies between the files with older files having higher values, close to total.

We have had a discussion on the chat about this, and one of the issues may be that scheduled is never set to false.

I don't fully understand why log file 16 stays active as well. I have no Infinispan warnings in the logs.

I would expect the log files to be compacted and removed after 2 hours. Instead it seems to keep generating log files until the disk is full. I have been unable to test this with 14.0.0 yet, but have tested with the latest 13.x.