User Story
As a developer I want Network Endpoint Groups (NEGs) supported on GCP so that problems with Instance Group-backed load balancers are avoided.
Background
OpenShift on GCP currently uses Instance Groups to place nodes behind a load balancer. This works, but has led to issues that have been difficult to trace and debug. Examples include OCPBUGS-26583 and OCPBUGS-5755.
Additionally, GCP only allows instances to participate in one instance group; OpenShift currently assumes instances can participate in multiple instances groups to get different load balancer groupings.
Our fix for OCPBUGS-5755 was discussed upstream, and is unlikely to be accepted. Upstream maintainers have instead suggested using network endpoint groups (NEGs).
Instances are able to be members of multiple NEGs for load balancer grouping on GCP.
Given that OpenShift installation is moving to using Cluster API instead of Terraform (CORS-3196), implementing support for NEGs in the Terraform-based installer is not considered a viable solution at this time. Instead, the Cluster API Provider for GCP (CAPG) will be used to create cloud resources.
Steps
- Determine if CAPG supports creating load balancers with NEGs. Code for this is at https://github.com/kubernetes-sigs/cluster-api-provider-gcp/tree/main/cloud/services/compute/loadbalancers
- If not, provide an NEG-based implementation that OpenShift can consume.
Stakeholders
- OpenShift developers
- OpenShift end users seeking to use multiple load balancers within GCP.
Definition of Done
- CAPI-based OpenShift installations create NEG-based load balancers rather than instance group-based ones.
- OpenShift installations using instance group-backed load balancers can be upgraded to use network endpoint group-backed load balancers
- Docs
- This should, mostly likely, be considered an implementation detail that most customers will not be concerned with.
- Testing
- Unit and integration tests in the upstream project
- Validation that Kubernetes `Service` load balancers created with a CAPI-based OpenShift installer
cloud.google.com/neg
annotation
gcloud compute network-endpoint-groups list
for the GCP Project associated with the OpenShift cluster returns entries.
- is related to
-
CORS-3196 [Tech Preview] Provision GCP with CAPI (no mgmt cluster)
- Closed
-
OCPBUGS-5755 GCP XPN private cluster install attempts to add masters to k8s-ig-xxxx instance groups
- Closed