Enable HPA for Backend Listener, Backend worker and Apicast.

Decided to wait for this issue to be fixed. Expected fixed OCP release: 4.7

Enable HPA as opt-in for:

• Backend Listener
• Backend worker
• Apicast production (not for apicast staging for now)

Regarding backend components, few notes. Although async mode is not a must, makes easier and robust to configure autoscaling. Thus, async mode will be enabled to enable HPA. Notes about enabling async and hpa for backend components: https://github.com/3scale/apisonator/blob/master/docs/openshift_horizontal_scaling.md. Currently async mode has one limitation: Cannot use Redis logical databases. Therefore cannot be used with current 3scale default deployment where backend uses two logical redis databases from the same instance. Either we deploy 2 redis instances for the default deployment mode (which is for dev testing and demos) or HPA is only enabled when external databases are being selected.

On OCP 4.3, HPA does not work well with init containers. We are using init containers in apicast and backend-worker (currently 3scale 2.9.1) . -A fix has been merged and according to generated errata (linked in BZ ticket), the fix is included in 4.5.1. -

https://access.redhat.com/solutions/4661461

https://bugzilla.redhat.com/show_bug.cgi?id=1749468 

 Design proposal:

• New APIMAnager CRD field of object type: spec.backend.listenerSpec.hpa.

spec:
  backend:
    listenerSpec:
      hpa:
        minPods: X
        maxPods: Y
        cpuPercent: Z          

• New APIMAnager CRD field of object type: spec.backend.workerSpec.hpa.

spec:
  backend:
    workerSpec:
      hpa:
        minPods: X
        maxPods: Y
        cpuPercent: Z          

• New APIMAnager CRD field of object type: spec.apicast.productionSpec.hpa.

spec:
  apicast:
    productionSpec:
      hpa:
        minPods: X
        maxPods: Y
        cpuPercent: Z          

• hpa objects and the replicas sibling fields are mutually exclusive. Existence of both fields in the CR will make the operator raise error and invalidate CR.
• HPA feature will only be enabled on "External Databases" deployment mode. As it only makes sense on production base deployments. There is a constraint: Backend redis URLS can not reference two logical databases of the same redis instance. It is either the same logical database or two independent redis instances.
• Default values will be defined for all hpa objects. As for minPods, could be 1 or 2 if HA is wanted. To be defined. As for maxPods, it is hard to say. Maybe the default value could be calculated from the operator taking into account the namespace limits. As for the cpuPercent, it should be something close to 80% or 90% to respect resource requests values.

 
Useful links:

• HPA in openshift 4.5
• HorizontalPodAutoscaler autoscaling/v1 spec
• Kubernetes Horizontal Pod Autoscaler