Right now we have a few problems with our code.

Firstly we had unique file names between uploads from ingress posts. This meant that data with the same usage times/partition dates would not be overwritten, Causing duplicate data.

Secondly if a customer had a single day span multiple files our download process would only take the data from the last file processed, because we break files out into daily archives we wrongly expected a single file to contain everything for that day. This meant we would overwrite files and drop line items between file batches.