Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-6034

ROSA Machinepool EgressIP Labels Not Discovered

XMLWordPrintable

      This is a clone of issue OCPBUGS-4969. The following is the description of the original issue:

      Description of problem:

      A ROSA machinepool is created and the label k8s.ovn.org/egress-assignable is added during creation. The newly created nodes are not discovered as egressIP nodes and no egressIP addresses are assigned.
      
      It was discovered that removing the k8s.ovn.org/egress-assignable label from the nodes, by editing the machinepool, and subsquently reapplying the label causes the nodes to be discovered as egressIP capable.
      
      While it is possible to workaround the issue be removing and reapplying the label, this will likely not work with node auto-scaling.

       

      Version-Release number of selected component (if applicable):

      4.11.18

      How reproducible:

      Always

      Steps to Reproduce:

      1. Create a machinepool and label for egressIP
      $ rosa create machinepool -c brosenbe --name mp-1 --labels k8s.ovn.org/egress-assignable="" --replicas=3
      I: Machine pool 'mp-1' created successfully on cluster 'brosenbe'
      I: To view all machine pools, run 'rosa list machinepools -c brosenbe'
      
      
      2. Wait for nodes to be instantiated
      $ watch -n 60 oc get nodes -l k8s.ovn.org/egress-assignable
      
      Every 60.0s: oc get nodes -l k8s.ovn.org/egress-assignable         brosenbe.syd.csb: Fri Dec 16 15:20:47 2022
      NAME                                              STATUS   ROLES    AGE     VERSION
      ip-10-0-136-123.ap-southeast-2.compute.internal   Ready    worker   7m55s   v1.24.6+5658434
      ip-10-0-178-34.ap-southeast-2.compute.internal    Ready    worker   7m59s   v1.24.6+5658434
      ip-10-0-192-110.ap-southeast-2.compute.internal   Ready    worker   8m      v1.24.6+5658434
      
      
      3. Create egressip object
      $ cat << EOF >egressip.yaml 
      apiVersion: k8s.ovn.org/v1
      kind: EgressIP
      metadata:
        name: egress-group1
      spec:
        egressIPs:
        - 10.0.128.152
        - 10.0.160.152
        - 10.0.192.152
        namespaceSelector:
          matchLabels:
            env: dev
      EOF
      
      
      4. Apply egressip object
      $ oc apply -f egressip.yaml 
      egressip.k8s.ovn.org/egress-group1 created
      
      
      5. Note that no IP addresses from egressip/egress-group1 have been assigned
      $ oc get egressip
      NAME            EGRESSIPS         ASSIGNED NODE   ASSIGNED EGRESSIPS
      egress-group1   10.0.128.152
                         
      $ oc get event -n default | egrep egressip | tail -1
      34s         Warning   NoMatchingNodeFound         egressip/egress-group1                                      no assignable nodes for EgressIP: egress-group1, please tag at least one node with label: k8s.ovn.org/egress-assignable
      
      $ ns=openshift-ovn-kubernetes; for pod in $(oc get pods -n $ns -l app=ovnkube-master -o name); do pod=${pod##*/}; echo $pod; oc logs -n $ns $pod -c ovnkube-master | grep 'No assignable nodes found for EgressIP' | tail -1; done
      ovnkube-master-bgz84
      ovnkube-master-kzgpc
      ovnkube-master-pbtn9
      E1216 04:21:50.578203       1 egressip.go:1567] No assignable nodes found for EgressIP: egress-group1 and requested IPs: [10.0.128.152 10.0.160.152 10.0.192.152]
      
      
      6. Remove egressIP labels
      $ rosa edit machinepool -c brosenbe mp-1 --replicas 3 --labels ''
      I: Updated machine pool 'mp-1' on cluster 'brosenbe'
      
      
      7. Wait a bit for labels to be removed...
      $ watch -n 60 oc get nodes -l k8s.ovn.org/egress-assignable
      
      Every 60.0s: oc get nodes -l k8s.ovn.org/egress-assignable          brosenbe.syd.csb: Fri Dec 16 15:51:57 2022
      
      No resources found
      
      
      8. Reapply label k8s.ovn.org/egress-assignable 
      $ rosa edit machinepool -c brosenbe mp-1 --replicas 3 --labels k8s.ovn.org/egress-assignable=''
      I: Updated machine pool 'mp-1' on cluster 'brosenbe'9. Wait a bit for labels to be applied...
      
      
      9. Wait a while for labels to be applied
      $ watch -n 60 oc get nodes -l k8s.ovn.org/egress-assignable
      
      Every 60.0s: oc get nodes -l k8s.ovn.org/egress-assignable          brosenbe.syd.csb: Fri Dec 16 16:00:03 2022
      NAME                                              STATUS   ROLES    AGE   VERSION
      ip-10-0-136-123.ap-southeast-2.compute.internal   Ready    worker   47m   v1.24.6+5658434
      ip-10-0-178-34.ap-southeast-2.compute.internal    Ready    worker   47m   v1.24.6+5658434
      ip-10-0-192-110.ap-southeast-2.compute.internal   Ready    worker   47m   v1.24.6+5658434
      
      
      10. Note that egressIP addresses have now been assigned to nodes
      $ oc get egressip egress-group1
      NAME            EGRESSIPS      ASSIGNED NODE                                     ASSIGNED EGRESSIPS
      egress-group1   10.0.128.152   ip-10-0-167-202.ap-southeast-2.compute.internal   10.0.160.152
      
      $ oc get egressip egress-group1 -o yaml | yq -y '.status'
      items:
        - egressIP: 10.0.128.152
          node: ip-10-0-136-123.ap-southeast-2.compute.internal
        - egressIP: 10.0.192.152
          node: ip-10-0-192-110.ap-southeast-2.compute.internal
        - egressIP: 10.0.160.152
          node: ip-10-0-178-34.ap-southeast-2.compute.internal 

      Actual results:

      EgressIP addresses not applied to nodes with k8s.ovn.org/egress-assignable label

      Expected results:

      EgressIP addresses are applied to nodes with k8s.ovn.org/egress-assignable label

      Additional info:

       

            pdiak@redhat.com Patryk Diak
            openshift-crt-jira-prow OpenShift Prow Bot
            Jean Chen Jean Chen
            Red Hat Employee
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: