Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-2380

Submariner Gateway on OVN: unable to parse node L3 gw annotation

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Critical Critical
    • None
    • None
    • None
    • False
    • None
    • False
    • No

      Description of problem:

      Submariner 0.14.1 in ACM 2.7.0 failed to create Gateway on the AWS node.

      Submariner Addon keeps restarting, and the OC events for the AWS node with the external IP shows:

      ErrorAddingLogicalPort addLogicalPort failed for openshift-multus/network-metrics-daemon-6b7r5: unable to parse node L3 gw annotation: k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-16-60-108.us-west-1.compute.internal"

      Version-Release number of selected component (if applicable):

      AWS with OCP 4.12.0-rc.4with OVN

      ACM 2.7.0

      Submariner 0.14.1

       

      How reproducible:

      ?

      Steps to Reproduce:

      1. Install OCP 4.12 on AWS with OVN
      2. Install ACM 2.7.0
      3. Install Submariner 0.14.1

      Full scenario:

      https://qe-jenkins-csb-skynet.apps.ocp-c1.prod.psi.redhat.com/job/ACM-2.7.0-Submariner-0.14.0-AWS-OSP-Globalnet-OVN/60/Test-Report/

      Actual results:

      oc  get nodes -l node-role.kubernetes.io/worker -o wide

      NAME                                          STATUS   ROLES    AGE   VERSION           INTERNAL-IP     EXTERNAL-IP     OS-IMAGE                                                        KERNEL-VERSION                 CONTAINER-RUNTIME
      ip-10-16-156-56.us-west-1.compute.internal    Ready    worker   78m   v1.25.4+86bd4ff   10.16.156.56    <none>          Red Hat Enterprise Linux CoreOS 412.86.202212081411-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.25.1-5.rhaos4.12.git6005903.el8
      ip-10-16-163-121.us-west-1.compute.internal   Ready    worker   81m   v1.25.4+86bd4ff   10.16.163.121   <none>          Red Hat Enterprise Linux CoreOS 412.86.202212081411-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.25.1-5.rhaos4.12.git6005903.el8
      ip-10-16-243-53.us-west-1.compute.internal    Ready    worker   78m   v1.25.4+86bd4ff   10.16.243.53    <none>          Red Hat Enterprise Linux CoreOS 412.86.202212081411-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.25.1-5.rhaos4.12.git6005903.el8
      ip-10-16-60-108.us-west-1.compute.internal    Ready    worker   21m   v1.25.4+86bd4ff   10.16.60.108    52.53.225.212   Red Hat Enterprise Linux CoreOS 412.86.202212081411-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.25.1-5.rhaos4.12.git6005903.el8

       

      OC Events for this AWS node shows:

       

      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        daemonset-controller                                                                                                                                          Normal    SuccessfulCreate                             Created pod: node-exporter-l5rsf
      <nil>                  <nil>                  <none>   <none>                                                                                                                                                        Normal    Scheduled                                    Successfully assigned openshift-monitoring/node-exporter-l5rsf to ip-10-16-60-108.us-west-1.compute.internal
      2022-12-12T18:21:13Z   2022-12-12T18:21:18Z   3        controlplane                                                                                                                                                  Warning   ErrorAddingLogicalPort                       addLogicalPort failed for openshift-multus/network-metrics-daemon-6b7r5: unable to parse node L3 gw annotation: k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-16-60-108.us-west-1.compute.internal"
      <nil>                  <nil>                  <none>   <none>                                                                                                                                                        Normal    Scheduled                                    Successfully assigned openshift-multus/network-metrics-daemon-6b7r5 to ip-10-16-60-108.us-west-1.compute.internal
      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        daemonset-controller                                                                                                                                          Normal    SuccessfulCreate                             Created pod: multus-wlsm7
      <nil>                  <nil>                  <none>   <none>                                                                                                                                                        Normal    Scheduled                                    Successfully assigned openshift-multus/multus-wlsm7 to ip-10-16-60-108.us-west-1.compute.internal
      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        daemonset-controller                                                                                                                                          Normal    SuccessfulCreate                             Created pod: node-ca-6tjr7
      <nil>                  <nil>                  <none>   <none>                                                                                                                                                        Normal    Scheduled                                    Successfully assigned openshift-image-registry/node-ca-6tjr7 to ip-10-16-60-108.us-west-1.compute.internal
      <nil>                  <nil>                  <none>   <none>                                                                                                                                                        Normal    Scheduled                                    Successfully assigned openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bzxqf to ip-10-16-60-108.us-west-1.compute.internal
      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        daemonset-controller                                                                                                                                          Normal    SuccessfulCreate                             Created pod: node-resolver-gg782
      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        submariner-agent-submarineragentconfigcontroller                                                                                                              Normal    SubmarinerConfigStatusUpdated                Updated status conditions:  []v1.Condition{v1.Condition{Type:"SubmarinerClusterEnvironmentPrepared", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 10, 0, time.Local), Reason:"SubmarinerClusterEnvPrepared", Message:"Submariner cluster environment was prepared"}, v1.Condition{Type:"SubmarinerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 12, 0, time.Local), Reason:"SubmarinerConfigApplied", Message:"SubmarinerConfig was applied"}, v1.Condition{Type:"SubmarinerGatewaysLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 21, 13, 0, time.Local), Reason:"Success", Message:"1 node(s) (\"ip-10-16-60-108.us-west-1.compute.internal\") are labeled as gateways"}}
      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        controlplane                                                                                                                                                  Warning   ErrorAddingLogicalPort                       addLogicalPort failed for openshift-network-diagnostics/network-check-target-qt6c6: timed out waiting for logical switch in logical switch cache "ip-10-16-60-108.us-west-1.compute.internal" subnet: error getting logical switch for node ip-10-16-60-108.us-west-1.compute.internal: switch not in logical switch cache
      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        submariner-agent-submarineragentstatuscontroller                                                                                                              Normal    ManagedClusterAddOnStatusUpdated             Updated status conditions:  []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 9, 0, time.Local), Reason:"RegistrationConfigured", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 10, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 9, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2022-12-12 18:12:09 +0000 UTC to 2022-12-13 17:01:21 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 12, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 21, 13, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"ip-10-16-60-108.us-west-1.compute.internal\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 33, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}}
      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        daemonset-controller                                                                                                                                          Normal    SuccessfulCreate                             Created pod: network-metrics-daemon-6b7r5
      2022-12-12T18:21:00Z   2022-12-12T18:21:13Z   8        kubelet                                                                                                                                                       Normal    NodeHasSufficientMemory                      Node ip-10-16-60-108.us-west-1.compute.internal status is now: NodeHasSufficientMemory
      <nil>                  <nil>                  <none>   <none>                                                                                                                                                        Normal    Scheduled                                    Successfully assigned openshift-network-diagnostics/network-check-target-qt6c6 to ip-10-16-60-108.us-west-1.compute.internal
      <nil>                  <nil>                  <none>   <none>                                                                                                                                                        Normal    Scheduled                                    Successfully assigned openshift-ovn-kubernetes/ovnkube-node-f2d6d to ip-10-16-60-108.us-west-1.compute.internal
      <nil>                  <nil>                  <none>   <none>                                                                                                                                                        Normal    Scheduled                                    Successfully assigned openshift-machine-config-operator/machine-config-daemon-h8d8b to ip-10-16-60-108.us-west-1.compute.internal
      <nil>                  <nil>                  <none>   <none>                                                                                                                                                        Normal    Scheduled                                    Successfully assigned openshift-dns/node-resolver-gg782 to ip-10-16-60-108.us-west-1.compute.internal
      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        daemonset-controller                                                                                                                                          Normal    SuccessfulCreate                             Created pod: multus-additional-cni-plugins-v552s
      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        daemonset-controller                                                                                                                                          Normal    SuccessfulCreate                             Created pod: aws-ebs-csi-driver-node-bzxqf
      <nil>                  <nil>                  <none>   <none>                                                                                                                                                        Normal    Scheduled                                    Successfully assigned openshift-cluster-node-tuning-operator/tuned-zbplw to ip-10-16-60-108.us-west-1.compute.internal
      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        daemonset-controller                                                                                                                                          Normal    SuccessfulCreate                             Created pod: machine-config-daemon-h8d8b
      <nil>                  <nil>                  <none>   <none>                                                                                                                                                        Normal    Scheduled                                    Successfully assigned openshift-multus/multus-additional-cni-plugins-v552s to ip-10-16-60-108.us-west-1.compute.internal
      2022-12-12T18:21:13Z   2022-12-12T18:21:58Z   16       controlplane                                                                                                                                                  Warning   ErrorReconcilingNode                         [k8s.ovn.org/node-chassis-id annotation not found for node ip-10-16-60-108.us-west-1.compute.internal, macAddress annotation not found for node "ip-10-16-60-108.us-west-1.compute.internal" , k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-16-60-108.us-west-1.compute.internal"]
      2022-12-12T18:21:00Z   2022-12-12T18:21:13Z   8        kubelet                                                                                                                                                       Normal    NodeHasNoDiskPressure                        Node ip-10-16-60-108.us-west-1.compute.internal status is now: NodeHasNoDiskPressure
      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        daemonset-controller                                                                                                                                          Normal    SuccessfulCreate                             Created pod: tuned-zbplw
      2022-12-12T18:21:13Z   2022-12-12T18:21:13Z   1        daemonset-controller                                                                                                                                          Normal    SuccessfulCreate                             Created pod: network-check-target-qt6c6
      2022-12-12T18:21:16Z   2022-12-12T18:21:16Z   1        node-controller                                                                                                                                               Normal    RegisteredNode                               Node ip-10-16-60-108.us-west-1.compute.internal event: Registered Node ip-10-16-60-108.us-west-1.compute.internal in Controller
      2022-12-12T18:21:18Z   2022-12-12T18:21:18Z   2        controlplane                                                                                                                                                  Warning   ErrorAddingLogicalPort                       addLogicalPort failed for openshift-network-diagnostics/network-check-target-qt6c6: unable to parse node L3 gw annotation: k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-16-60-108.us-west-1.compute.internal"
      2022-12-12T18:21:18Z   2022-12-12T18:21:50Z   7        kubelet                                                                                                                                                       Warning   FailedMount                                  MountVolume.SetUp failed for volume "kube-api-access-f7wxg" : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]

      Expected results:

      Gateway should be created on the AWS node

      Additional info:

              rh-ee-vthapar Vishal Thapar
              nmanos@redhat.com Noam Manos
              Noam Manos Noam Manos
              ACM QE Team
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: