AWS introduced CUR 2.0 which collapses some columns into a new column with key-value pairs of the collapsed columns. We should update our pipeline to accept these new reports.

Things we need to be aware of:

• New permissions on the customer side needs to include the additional entries for data exports
  • "bcm-data-exports:ListExports*"

• • "bcm-data-exports:GetExport*"

• Koku side
  • Checking for data exports on source creation needs to be updated to use the data exports client
  • AWS downloader needs updating
    • Setting correct paths for downloading files
    • creation of manifest entries needs to be updated
  • Trino columns to need tweaking
    • product/operatingSystem and product/memory columns would be rolled into key/value pairs in a new product column
      • These need to be correctly pulled out and normalised during our post processor flow

 
Here are some relevant links:

• Current CUR columns: https://docs.aws.amazon.com/cur/latest/userguide/data-dictionary.html
• CUR 2.0 columns: https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2.html

Design doc