Uploaded image for project: 'Cost Management'
  1. Cost Management
  2. COST-2682

Optimization GCP Downloader: manifests

XMLWordPrintable

    • Icon: Epic Epic
    • Resolution: Done
    • Icon: Critical Critical
    • 2022Q3
    • None
    • None
    • None

      GCP is different from our other providers since we build the csv files ourselves. Which should be handled differently in the day to day flow versus the initial downloaded. Therefore, the new implementation will check each day in the scan range to see if there is new data to be downloaded. We do this by collecting the partition date in GCP and mapping it to the last time data was exported to that partition: 

      SELECT DATE(_PARTITIONTIME), max(export_time) FROM `cloud-billing-292519.billing_datset.gcp_billing_export_resource_v1_018984_D0AAA4_940B88` WHERE DATE(_PARTITIONTIME) BETWEEN "2022-05-17" AND "2022-05-19" GROUP BY DATE(_PARTITIONTIME) – "2022-05-19"-2022-05-19 19:40:16.385000 UTC – Partitition_date:max(export_time)  

      Therefore, the new optimization will have a manifest per downloaded.

            myersco Cody Myers
            myersco Cody Myers
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: