Uploaded image for project: 'Cost Management'
  1. Cost Management
  2. COST-2682

Optimization GCP Downloader: manifests

XMLWordPrintable

    • Icon: Epic Epic
    • Resolution: Done
    • Icon: Critical Critical
    • 2022Q3
    • None
    • None
    • None
    • Optimization GCP Downloader: manifests
    • False
    • None
    • False
    • 0% To Do, 0% In Progress, 100% Done

      GCP is different from our other providers since we build the csv files ourselves. Which should be handled differently in the day to day flow versus the initial downloaded. Therefore, the new implementation will check each day in the scan range to see if there is new data to be downloaded. We do this by collecting the partition date in GCP and mapping it to the last time data was exported to that partition: 

      SELECT DATE(_PARTITIONTIME), max(export_time) FROM `cloud-billing-292519.billing_datset.gcp_billing_export_resource_v1_018984_D0AAA4_940B88` WHERE DATE(_PARTITIONTIME) BETWEEN "2022-05-17" AND "2022-05-19" GROUP BY DATE(_PARTITIONTIME) – "2022-05-19"-2022-05-19 19:40:16.385000 UTC – Partitition_date:max(export_time)  

      Therefore, the new optimization will have a manifest per downloaded.

              myersco Cody Myers
              myersco Cody Myers
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: