Uploaded image for project: 'Cost Management'
  1. Cost Management
  2. COST-4208

DB Access Simplification (Masu simplification)

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Done
    • Icon: Normal Normal
    • 2023Q4
    • None
    • None
    • False
    • None
    • False
    • COST-4218 - Data pipeline stability, reliability and observability enhancements
    • COST-4218Data pipeline stability, reliability and observability enhancements

      Target Dates

      Dev Target Date: 24-Nov-2023
      Test Target Date: ?

       

      Design Document

      1. Given that Masu was a separate repo in the beginning, we have many layers of code used to access database models (db accessors). We should remove all of the db-accessors because we now have direct access to the django models. Some (provider related) of these will be very easy, others (report db accessors) will be much more involved. For the more involved cases, we can use model managers to provide us with similar functionality that the accessors provide.

      The following classes should be removed entirely.

      KokuDBAccess
      AccountAliasAccessor
      CustomerDBAccessor
      ProviderAuthDBAccessor
      ProviderBillingSourceDBAccessor
      ProviderCollector
      ProviderDBAccessor
      ReportingCommonDBAccessor
      
      CURAccountsInterface
      CURAccountsDB
      AccountsAccessor
      

      The reporting-db-accessors are more complicated to try and remove entirely. These are responsible for the majority of data processing (inserting data into the correct tables). There are some things we can do to make understanding how they work a little better though.

      • remove _get_db_obj_query in favor of using the django model directly. Example:
          def get_cost_entry_bills_query_by_provider(self, provider_uuid):
              """Return all cost entry bills for the specified provider."""
              with schema_context(self.schema):
                  return AWSCostEntryBill.objects.filter(provider_id=provider_uuid)
      
      • Remove unused functions (there are many of them. these are easy to find by searching for methods and noticing that they are only used in tests)

              mskarbek Michael Skarbek
              mskarbek Michael Skarbek
              Daniel Chorvatovic Daniel Chorvatovic
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

                Created:
                Updated:
                Resolved: