Create a Proof of Concept (PoC) for a Leases Watcher

• Add a Pipelines Leases Controller: Introduce a new controller within the Pipelines operator to monitor the leases held by Pipelines controllers.
• Monitor Lease Distribution: The controller will track the distribution of leases across the controller replicas to ensure they are evenly allocated.
• Ensure Equal Distribution: The number of leases should align with the number of buckets specified in the configuration and be distributed evenly among all replicas.
• Rebalancing Mechanism: If the distribution is unequal, the controller will rebalance the leases by removing a lease from one controller to allow another replica to acquire it.
• Load Balancing Formula:
• • Let n represent the total number of buckets (and leases).
  • Let m represent the number of replicas (Pipelines controller instances).
  • Each replica should approximately hold n/m leases to evenly distribute the load across all replicas.

Test Cases for Leases Watcher PoC

1. Test 1: Correct Initial Lease Distribution

• • Objective: Ensure that when n replicas and m buckets are created, the leases are evenly distributed.
  • Check: Verify that the number of leases held by each replica is approximately n/m.

1. Test 2: Stability of Lease Rebalancing

• • Objective: Confirm that once leases are balanced, the controller doesn’t continuously trigger rebalancing.
  • Check: Verify that after an initial rebalancing, the system reaches a stable state without unnecessary lease adjustments.

1. Test 3: Reconciliation Stability During Rebalancing

• • Objective: Ensure that ongoing rebalancing does not interfere with active reconciliation requests.
  • Check: Verify that reconciliation requests are processed without loss or delay during the lease rebalancing phase.

1. Test 4: Lease Redistribution on Scaling Events

• • Objective: Ensure that scaling up or down the number of replicas triggers a proper redistribution of leases.
  • Check: After scaling, verify that leases are evenly redistributed across the new number of replicas.

1. Test 5: Lease Cleanup on Replica Shutdown

• • Objective: Ensure that when a replica is taken down, its leases are properly cleaned up.
  • Check: Verify that the leases from the downed replica are released and reassigned to the remaining replicas.