The file sources need to have consistent implementation of getFiles and should provide additional information such as the full file path and if it's a directory. For example currently it's possible to return directories in the file source if you specify a non-glob directory path, but that information is not exposed.

The glob paths should all support multiple, not just a single * in the file name.

At a first glance we may not implement these changes for ftp, but just throw appropriate exceptions - it currently only knows about returning a single file. It would be possible to eventually add logic for that though as it supports similar file visiting logic as Java.

So the focus will be on file, hdfs, and s3 - which invinci_1 is already looking into.