When <scanAnnotatedClasses/> is used, optaplanner users org.reflections to scan the classpath/modulepath.

org.reflections is probably not JDK 9 compatible, it has many issues reported around that. Maybe the latest version is (recently a lot of PR's were merged). Geoffrey might get contributor rights to help increase the quality of that library: https://github.com/ronmamo/reflections/pull/183#issuecomment-343664184

There's also an alternative library (scanannotation is dead), but org.reflections is very good, so this will just be trading one problem for another: https://github.com/lukehutch/fast-classpath-scanner

If org.reflections isn't JDK9 compatible, let's pin-point why and see if we can easily fix it (and contribute it back).