Uploaded image for project: 'Cost Management'
  1. Cost Management
  2. COST-5129

Split OCP on Cloud parquet processing the download flow into a separate task

XMLWordPrintable

    • Icon: Epic Epic
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • None
    • OCP on Cloud parquet processing split
    • 35
    • False
    • None
    • False
    • To Do
    • 36% To Do, 0% In Progress, 64% Done

      OCP on Cloud parquet files are currently created during the download flow, this processing should be split into a separate celery task that runs Trino SQL. Large impact to the data processing pipeline, no impact to customer facing API/UI, update and document existing masu endpoint to use new flow.

       

      This should be done one provider at a time in order to reduce risk.

      Design Doc

       

      This will be released one provider at a time to reduce risk of a larger change. AWS will be the first.  Targeting stage by 2024/08/15.

       

            myersco Cody Myers
            cgoodfre Corey Goodfred (Inactive)
            Daniel Chorvatovic Daniel Chorvatovic
            Votes:
            0 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated: