Deploy multi-homed OLM operator pod without patches

1. Proposed title of this feature request

Multi-homed OLM operator pods using Multus

2. What is the nature and description of the request?

https://access.redhat.com/support/cases/#/case/03168849/discussion?commentId=a0a2K00000eg12HQAQ

3. Why does the customer need this? (List the business requirements here)

The customer is running multi-homed workload in different VLANs using Multus. Some of this workload are operators that need to access resources located in the VLAN as well. At the moment there is no lean way of achieving this goal other than patching the CSV after the operator has been installed.

4. List any affected packages or components.

OLM (https://github.com/openshift/operator-framework-olm)

// Additional Info :
When rolling out an operator with OLM [1] this is done by creating both, an operator group and a subscription. Once the subscription has been created, OLM will create a CSV which the holds the application metadata. According to the specification in the subscriptions `.spec.config` field and the values provided by the CSV the operator deployment will be rolled out. For our usecase we would like to deploy an multi-homed operator pod using Multus [2], as the operator needs the control a bunch of hosts in another network. Therefore the field `metadata.annoations.k8s.v1.cni.cncf.io/networks` needs to be set for the operator pods so that additional network interfaces can be attached to the pod.

I have tried to resolve the issue but the only way that seems to work is by manually patching the CSV so that the section defining the deployment of the operator pod contains the `metadata.annoations.k8s.v1.cni.cncf.io/networks` annotation. Since the name of the CSV is not static (my-operator.1.0.0-202203111032-<hash>) and might change with every update it is also not possible to provide a static patch. Instead a script needs to be run, that first determines the name of the CSV and then applies the patch to the resource. This is not feasible for a GitOps workflow.

Another possible solution I investigated was the usage of the subscription `spec.config` field [3]. But it seems that there is no way to specify an annotation there. Even a proposal to enhance the config field [4] by using a `PodSpec` does not solve the problem, as `PodSpec` can not be used to define annotations. Instead I would like to follow a similar approach as described in the proposal but instead of using a `PodSpec` I would rather like to use `PodTemplateSpec` [5] as this allows the usage of annotations. Anyhow, this proposal is still pending and I assume it has not been implemented yet as this would be a breaking change. I have also opened a feature request upstream [6].

• Were there any recent changes to the environment? or is this the first occurrence
  The customer runs self developed operators on top of their OCP clusters

• on which environment(VMware, Baremetal, AWS, Azure, etc) cluster is deployed?
  At least three clusters have this operator installed and a facing the same problem. The environment is on bare metal.

• Is it IPI or UPI-based installation?
  It is an UPI-based installation.

[1]
https://github.com/openshift/operator-framework-olm
[2]
https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md#run-pod-with-network-annotation
[3]
https://github.com/operator-framework/operator-lifecycle-manager/blob/b3cded1d82f74feb0879ccc68abc1388c6ca4e70/vendor/github.com/operator-framework/api/pkg/operators/v1alpha1/subscription_types.go#L42
[4]
https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/contributors/design-proposals/operator-config.md
[5]
https://github.com/kubernetes/api/blob/b8c40e080bc5e830097df540d4ef804034cb5bdb/core/v1/types.go#L3907
[6]
https://github.com/operator-framework/operator-lifecycle-manager/issues/2684