https://redhat-internal.slack.com/archives/C024BNW4HT8/p1696607836771969

We received 2 payloads from a cluster. The first one contains data for today (2023-Oct-06). The second one contains all of todays data, plus the last 3 hours of data from yesterday (2023-Oct-05) appended to the bottom of all the reports(???) I don’t have a clue how we would have older data appended to a report file (it could potentially happen if a CR is created and allowed to generate data, then deleted, then another CR is created with the collect_previous_data flag set to True without cleaning the PVC first).

The way that we get start end dates into the operator manifest is to read the first and last rows of the report as the start and end dates, respectively. We’re assuming the reports in a cluster are appended to in sequential order. Because we do this, that second payload we received had a start of 2023-10-06T00:00:00Z and end of 2023-10-05T23:59:59Z which is not fantastic at all.

We should fix the operator to read each line of the report and get the min and max dates.

We should also update koku to read the start/end dates from the reports themselves instead of the payload manifest, especially since customers can modify payloads before they are sent.