PO/Tech Lead NOTE: Developer Pairing Opportunity

Implement and configure Snowpipe pipelines for dev through production environments to automate data ingestion from S3 buckets into Snowflake.

WIP MR: https://gitlab.cee.redhat.com/dataverse/dataverse-config/dataproduct-config/-/merge_requests/5237/diffs 

Create Pipes for each s3 bucket:

• ocp_cluster_data
• sku_product_tag
  _ subscriptions _This is the floorist export
• tally

(not yet implemented in floorist)

• tally_snapshots
• tally_measurements
  _ subscription _This is the raw table
• subscription_measurements
• hosts
• host_tally_buckets

Acceptance Criteria:

• For environments dev
• When new data is added to the s3 buckets, then the snowflake stages are populated.
• Snowflake Snowpipe DB- We need to verify record count matches Floorist export