Reconcile cluster k8s resources with 3scale using the 3scale API.

Include APIaaP objects

Some considerations for the implementation:

• Some types have owner and cannot be shared. For instance Products. Product belong to one account provider and cannot be shared between account providers. Same for limits. Limits are owned by some application plan and cannot be shared by multiple application plans.
• Objects can be invalid. Invalid objects will be marked as such, updated status with error message and will be filtered out by operator controllers. User should delete them and do not try to create new objects with the same spec. Invalidating an object A will trigger invalidation cascade to all objects with references to the object A. The alternative to invalidate objects is, object removal. Invalidating objects approach lets the user know what happened.
  • For instance, ActiveDocs belonging to a account provider and product, and those account provider and product are not related.
• Backend usage: 1 product cannot be using same backend multiple times. Unique key is composition of <product_id, backend_id>.
  • Proposed solution: In the spec of the product, there is a map called backend usages with the key being backend_id and the value being the backend usage object reference. This way it is ensured 1 product not using same backend multiple times.
• Deleteion cascade: Implement using ownerreferences? Ownerreferences does not work for Applicationa. Appilcation object has two owners, application plan and account. If any of those owners are deleted, application should be deleted as well (orphan object). K8s garbage collector works only when ALL the owners have been deleted. Hibrid solution? ownerreferences for simple relations, operator based deletetion cascade for complex relations like the applications?
• Orphan objects: when some controller finds an orphan object (some referenced object does not exist), it will wait for the referenced object to exist. The controller will start retry with exponential back-off pattern until the referenced object is found (which could never happen). This way it is allowed temporary inconsistency, like for instance create product resource before backend resource. Product object will be ofphan until backend is found.
• Objects lifecycle
  • Based on status.Conditions k8s common pattern
  • states:
    • InSync: Sync process going on
    • Orphan: Temporarily invalid. Waiting for some resource to exist.
    • Synced: Sync'ed
    • Invalid: Invalid object. Spec should be changed.
• Implement 3scale state cache in status? Avoid too many calls to 3scale API for every reconciliation loop. Assume 3scale state (read from 3scale API endpoints) does not change if 3scale operator does not change it (using 3scale API delete, put, post methods)
• If Account Provider object is deleted, the products, backends, accounts, activedocs will not be deleted from 3scale. It is just considered "disconnected". Current PoC implements deletion of all services from 3scale when "binding" object is deleted (finalizers are used for that regard)
• 3scale API does not allow deleting a backend if it is being used by a Product. This constraint is not easy to implement as k8s resources. If the user deletes a backend. All products using that deleted backend will be marked as orphan and will not sync until reference is found or backend usage is deleted.
• 3scale does not allow updating the ownership of some object types. For instance an application plan owned by a product cannot be changed and owned by another product. This kind of relations will be implemented by defining objects inside owner spec instead of having their own CRD. For instance application plans will not have their own CRD.
  • Detailed explanation: When there is a relation 1:N from A -> B, A instance owns multiple B instances. The owner of those B instance cannot be changed to some other C object. It is required to enforce ownership as fixed, not changeable. The reconciliation logic for the scenario where ownership can be updated is very complex. For example: Product A owns several Metrics and Application plans. One of these plans, plan P, owns a limit, L, related to some metric M. If application plan P's owner is changed to product B the entire system will be inconsistent. As the application plan P would be owned by B and it would own limits with metrics owned by a different product A. This scenario is not allowed in 3scale.
• Relations 1:N will be implemented using one of the following options, depending on each relationship type:
  • B instance contains a reference to A. This way you make sure B instance is not owned by multiple A's. But this solution has a big weakness. There is nothing like relational integrity, that prevents the user to update the reference hold by B to A and change to a reference to C or even something not existing. Any change has to be validated by the controller. Example: Product 1 -> N ActriveDocs
  • A maintains a list/map of B references. Then 1:N relation has to be validated by the controller making sure B instance is not owned by several A instances. Example: Product 1 -> N Backend (backend usage relationship)
  • A maintaing a list/map of B objects as subresources. B object type will not have their own CRD. Ownership is fixed. Example: Product 1 -> N Application plans
• 3scale entities are linked to a Provider Account. This provider account will be referenced by the credentials secret to sync with 3scale using REST API. Each object will need provider account to sync. The look up process can be as follows:
  • If provider_account_reference attribute is found, referenced provider account will be used.
  • If no provider_account_reference attribute is found, some secret with hardcoded name will be looked up in the namespace. For example, `default_provider_account` secret. If found, that one will be used.
  • If no provider_account_reference attribute is found AND provider account default secret in the namespace is not found either, then, 3scale default provider account (3scale-admin) will be looked up using system-seed secret in the current namespace.
  • If nothing is successfully found, the object will be marked as orphan.

Design proposal:

Account
--------------
provider_account_reference: REF
applications: 
  app_id: 
    attributes: {}

ActiveDocs
--------------
provider_account_reference: REF
product_reference: REF
attributes: {}

Backend
--------------
name: Operated Backend A                                         
privateBaseURL: https://example.com:8443                         
systemName: OperatedBackendA                                     
metrics:
  hits:                                                          
    description: Number of API hits                              
    friendlyName: Hits                                           
    unit: "hit"                                                  
methods:                                                         
  method01:                                                      
    friendlyName: Method01                                       
mappingRules:                                                    
  - httpMethod: GET                                              
    pattern: "/pets"                                             
    metricMethodRef: metric02                                    
  - httpMethod: POST                                             
    pattern: "/pets"                                             
    metricMethodRef: hits 

Product
--------------
provider_account_reference: REF
deployment:                                                 
  apicastSelfManaged:                                       
    stagingPublicBaseURL: "https://staging2.example.com"    
    productionPublicBaseURL: "https://prod2.example.com"    
    authentication:                                         
      appKeyAppID:  {}          
metrics:                                                            
  hits:                                   
    description: Number of API hits       
    friendlyName: Hits
    unit: "hit"                           
methods:                                  
  method01:                               
    friendlyName: Method01                
mappingRules:                         
  - httpMethod: GET                   
    pattern: "/pets"                  
    metricMethodRef: method01         
  - httpMethod: GET                   
    pattern: "/pets/id"               
    metricMethodRef: method01
backendUsages:      
  backendA:         
    path: /A        
applicationPlans:                  
  plan01:                          
    name: "My Plan 01"
    limits: 
      - period: month            
        value: 100               
        metricMethodRef:         
          systemName: hits       
      - period: month            
        value: 200               
        metricMethodRef:         
           systemName: hits       
           backend: backendA       
    pricingRules:                  
      - from: 1                    
        to: 100                    
        pricePerUnit: "15.45"      
        metricMethodRef:           
          systemName: hits         
      - from: 1                    
        to: 100                    
        pricePerUnit: "15.45"      
        metricMethodRef:           
          systemName: hits         
          backend: backendA