ROSA Machinepool EgressIP Labels Not Discovered

A ROSA machinepool is created and the label k8s.ovn.org/egress-assignable is added during creation. The newly created nodes are not discovered as egressIP nodes and no egressIP addresses are assigned.

It was discovered that removing the k8s.ovn.org/egress-assignable label from the nodes, by editing the machinepool, and subsquently reapplying the label causes the nodes to be discovered as egressIP capable.

While it is possible to workaround the issue be removing and reapplying the label, this will likely not work with node auto-scaling.

4.11.18

Always

1. Create a machinepool and label for egressIP
$ rosa create machinepool -c brosenbe --name mp-1 --labels k8s.ovn.org/egress-assignable="" --replicas=3
I: Machine pool 'mp-1' created successfully on cluster 'brosenbe'
I: To view all machine pools, run 'rosa list machinepools -c brosenbe'


2. Wait for nodes to be instantiated
$ watch -n 60 oc get nodes -l k8s.ovn.org/egress-assignable

Every 60.0s: oc get nodes -l k8s.ovn.org/egress-assignable         brosenbe.syd.csb: Fri Dec 16 15:20:47 2022
NAME                                              STATUS   ROLES    AGE     VERSION
ip-10-0-136-123.ap-southeast-2.compute.internal   Ready    worker   7m55s   v1.24.6+5658434
ip-10-0-178-34.ap-southeast-2.compute.internal    Ready    worker   7m59s   v1.24.6+5658434
ip-10-0-192-110.ap-southeast-2.compute.internal   Ready    worker   8m      v1.24.6+5658434


3. Create egressip object
$ cat << EOF >egressip.yaml 
apiVersion: k8s.ovn.org/v1
kind: EgressIP
metadata:
  name: egress-group1
spec:
  egressIPs:
  - 10.0.128.152
  - 10.0.160.152
  - 10.0.192.152
  namespaceSelector:
    matchLabels:
      env: dev
EOF


4. Apply egressip object
$ oc apply -f egressip.yaml 
egressip.k8s.ovn.org/egress-group1 created


5. Note that no IP addresses from egressip/egress-group1 have been assigned
$ oc get egressip
NAME            EGRESSIPS         ASSIGNED NODE   ASSIGNED EGRESSIPS
egress-group1   10.0.128.152
                   
$ oc get event -n default | egrep egressip | tail -1
34s         Warning   NoMatchingNodeFound         egressip/egress-group1                                      no assignable nodes for EgressIP: egress-group1, please tag at least one node with label: k8s.ovn.org/egress-assignable

$ ns=openshift-ovn-kubernetes; for pod in $(oc get pods -n $ns -l app=ovnkube-master -o name); do pod=${pod##*/}; echo $pod; oc logs -n $ns $pod -c ovnkube-master | grep 'No assignable nodes found for EgressIP' | tail -1; done
ovnkube-master-bgz84
ovnkube-master-kzgpc
ovnkube-master-pbtn9
E1216 04:21:50.578203       1 egressip.go:1567] No assignable nodes found for EgressIP: egress-group1 and requested IPs: [10.0.128.152 10.0.160.152 10.0.192.152]


6. Remove egressIP labels
$ rosa edit machinepool -c brosenbe mp-1 --replicas 3 --labels ''
I: Updated machine pool 'mp-1' on cluster 'brosenbe'


7. Wait a bit for labels to be removed...
$ watch -n 60 oc get nodes -l k8s.ovn.org/egress-assignable

Every 60.0s: oc get nodes -l k8s.ovn.org/egress-assignable          brosenbe.syd.csb: Fri Dec 16 15:51:57 2022

No resources found


8. Reapply label k8s.ovn.org/egress-assignable 
$ rosa edit machinepool -c brosenbe mp-1 --replicas 3 --labels k8s.ovn.org/egress-assignable=''
I: Updated machine pool 'mp-1' on cluster 'brosenbe'9. Wait a bit for labels to be applied...


9. Wait a while for labels to be applied
$ watch -n 60 oc get nodes -l k8s.ovn.org/egress-assignable

Every 60.0s: oc get nodes -l k8s.ovn.org/egress-assignable          brosenbe.syd.csb: Fri Dec 16 16:00:03 2022
NAME                                              STATUS   ROLES    AGE   VERSION
ip-10-0-136-123.ap-southeast-2.compute.internal   Ready    worker   47m   v1.24.6+5658434
ip-10-0-178-34.ap-southeast-2.compute.internal    Ready    worker   47m   v1.24.6+5658434
ip-10-0-192-110.ap-southeast-2.compute.internal   Ready    worker   47m   v1.24.6+5658434


10. Note that egressIP addresses have now been assigned to nodes
$ oc get egressip egress-group1
NAME            EGRESSIPS      ASSIGNED NODE                                     ASSIGNED EGRESSIPS
egress-group1   10.0.128.152   ip-10-0-167-202.ap-southeast-2.compute.internal   10.0.160.152

$ oc get egressip egress-group1 -o yaml | yq -y '.status'
items:
  - egressIP: 10.0.128.152
    node: ip-10-0-136-123.ap-southeast-2.compute.internal
  - egressIP: 10.0.192.152
    node: ip-10-0-192-110.ap-southeast-2.compute.internal
  - egressIP: 10.0.160.152
    node: ip-10-0-178-34.ap-southeast-2.compute.internal 

EgressIP addresses not applied to nodes with k8s.ovn.org/egress-assignable label

EgressIP addresses are applied to nodes with k8s.ovn.org/egress-assignable label