Uploaded image for project: 'FlightPath'
  1. FlightPath
  2. FLPATH-2946

trino - partition deletion, parquet metadata

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • None

      The parquet files include metadata that is used to delete existing data. The existing data refers to the same day that the current upload is refering to. the current upload is newer and therefore existing data should be removed. This functionality is handled by prepare_parquet_s3

      There is another set of tables that are created via SQL files and used by report_db_accessor_base.py and its successors. For these tables we will not use partitioning at the moment.

              ydayagi yaron dayagi
              ydayagi yaron dayagi
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: