Scope
Upstream Istio added support for external control plane deployment model in 1.8 onwoards. That is, the control plane (mainly istiod) can be deployed in another Kubernetes cluster than where the mesh is configured and the proxies are running. This is supported with code change and also manifest template change.
The scope of this story is to add the external control plane capability to Maistra as an optional deployment model.
Principles
Operator based Kubernetes resource management is usually built around one or more custom Kubernetes resource definitions and instances, whereas a single instance of a particular service is represented by a single custom resource instance. In case of Maistra, the main mesh resource is the ServiceMeshControlPlane (SMCP). That is, a *single* SMCP shall trigger the mesh installation/(re)configuration in every case.
Main problem statement
The split deployment inevitably contains resource management in two distinct Kubernetes clusters. The control plane cluster is hidden from the user from management point of view (it falls to the cloud vendor's or service provider's ownership). Only the necessary data plane components shall have hardened network access towards the services, exposed on the control plane cluster. The solution shall take care to not expose any credentials of the control plane to the user's cluster.
Change areas
Maistra custom resource source
As a single SMCP shall trigger everything, the solution shall make the operator capable to contact both clusters during the resource reconciliation. As the SMCP contains every configuration, including the global mesh settings (subject to change directly by the user), the SMCP and other Maistra CRD storage is going to be the data-plane cluster.
Remote operation for the operator
As the operator pod shall have access to both clusters and the control-plane API credentials must not be exposed to the mesh user, the operator shall run on the control plane cluster. It's primary API access (which is used by the operator SDK framework for watching CRs) however, should remain the data-plane cluster.
The working assumption is to leverage the out-of-cluster (local) environment configuration for the operator to run remotely from the control-plane side, offered by Operator SDK. In this mode basically the operator is able to function without any assumption on it's environment (does not attempt to read it's own pod, attempt leader election, etc.).
Webhook management
Maistra operator contains it's own mutating and validating webhooks and their entire management. Since it will operate remotely for the data-plane cluster, the webhook configuration shall follow this approach (i.e. use URL instead of in-cluster Service reference). As integrator may implement the networking between the CP and DP side differently, the webhook management is going to be disabled as part of this change set. That is, the webhook management (including custom TLS CA management) is going to be conditionally disabled in the operator, so that it will not conflict with the integrators own tooling. Integrator may use OLM to manage the webhooks, or other custom automation. This change will have it's own disable option as a new OLM annotation. Note: Disabling webhook management in the operator is going to provide platform independency for Maistra (as it is currently using Openshift specific services for webhook cert management).
Secondary API access
Having said that the operator will run on the CP side, operating on the DP side mainly, there shall be a secondary API connection for it, which will be configured for the local, CP API. Conditional logic is going to be made to install a selected subset of the charts on each side. The Kubernetes resource tracking and owner reference management on the control-plane side is subject to change as required (i.e. there won't be any SMCP on the CP side which currently is the owner of downstream resources).
Data-plane deployment
Istio 1.8 introduced split deployment option with the addition of a standalone chart for the data-plane side. It is called as `istiod-remote`. As part of this change set, the chart is going to be supported by Maistra operator.
Control-plane deployment
Control-plane side mainly means the deployment of `istio-discovery`.
Other optional components
Observability charts/components are possibly split across the control- and data-plane in multiple ways. Currently under consideration, whether they are going to be deployed with Maistra operator or separately.
Option control for split deployment
Since the primary API access is going to be one specific data-plane Kubernetes API for Maistra operator, it is not able to cover a multi-tenant environment with a single instance, assuming the integrator is operating a multi-tenant control-plane. That said, the operator may not be prepared for multiple mesh management, which means a global, static option may be used to start the operator up in the new mode for split deployment. The new option may be a new OLM annotation.
Maistra operator API changes
There are certain parameters which are configured through the `techPreview` section in order to have the split deployment work. Some of them may need be promoted into real SMCP parameters to let either the data-plane user or the integrator to configure them as supported features/parameters.
Story
Minor
