Federation for OSSM 3

Challenges with current federation:

• As it is part of Maistra.io, we do not benefit from the hardening that being in a large multi-vendor project like Istio provides.
• As it is embedded with Istio, it is difficult to maintain, as the underlying code change frequently during rebases.
• It is still relatively complicated to setup and maintain - particularly certificate/trust management between meshes.

That said, some customers value the federated model of service mesh that allows a mesh admin to import services from a remote mesh and then manage them within the mesh as if they were local services. The two primary use cases for this are: high-availability/failover of services between clusters and location transparency (for services that may be moved between clusters). 

The federated model potentially also supports greater scale than multi-primary/primary-remote, as you could potentially have an unlimited number of clusters involved, as long as each mesh only federates with a few others. A few large enterprise customers have identified this, and more may come as mesh usage expands.

The Maistra implementation of federation may be over built as it assumed a strong separation of information between meshes and clusters which is not a requirement that has been seen with customers. In general, if meshes are federated, they can assume to have some level of shared trust (shared trust?), but some independence when it comes to service discovery (ie import/export). In most cases, federated meshes will also have a common admin user... thus information hiding between meshes is not that important, except as a convenience feature (for example, rewriting service names so as not to have duplication)

Options for OSSM3 to explore:

• Consider... "As an upstream Istio user, if I want to federate service meshes,...such that I can include a select number(not all)  of remote services locally...how would I set this up?" Are there features that could be added to Istio to support this? If so, could we put a community proposal together?
• Refactor federation out into its own controller to be contributed as a separate project to istio-ecosystem (may have to review how it compares with Admiral, another multi-cluster project in istio-ecosystem)
• Can the use of SPIRE for trust offer a simplification? 
• Contribute CRDs upstream to make it easier to configure with upstream resources.

Related upstream issues:

• https://github.com/istio/istio/issues/48998
• https://github.com/istio/istio/issues/40589
• (Some discussion on federation related to Ambient): https://github.com/istio/istio/issues/43937
• https://github.com/istio/istio/issues/31967
• https://github.com/istio/istio/issues/31077
•  

Other similar federation implementations:
https://github.com/vmware/hamlet
Admiral