After ingestion of OCP nise-generated report with overlapping data for one day, we see that the total costs for that specific accumulate instead of dropping the data for the day and repopulating it with data from the new payload.
After reviewing the metadata of parquet files for this specific day, we noticed that the date from report date start field does not match the date of the data, which is the source of the bug (See attached screenshot from minio)
Steps to reproduce:
- ingest multiple days of data, ending during previous day (in my case, I generated data ending 13:00 UTC on the previous day)
- Ingest full day of data, starting at 00:00 the previous day, ending on the current day
- During data generation, use the --daily-reports flag in Nise
- See that costs for previous day have accumulated
- 0.0.0.0:8000/api/cost-management/v1/reports/openshift/costs/?group_by[cluster]=*
- is related to
-
COST-3572 handle daily data from operator
- Closed