-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
None
-
None
-
None
-
False
-
None
-
False
-
-
-
No
Description of problem:
Submariner 0.14.1 in ACM 2.7.0 failed to create Gateway on the AWS node.
Submariner Addon keeps restarting, and the OC events for the AWS node with the external IP shows:
ErrorAddingLogicalPort addLogicalPort failed for openshift-multus/network-metrics-daemon-6b7r5: unable to parse node L3 gw annotation: k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-16-60-108.us-west-1.compute.internal"
Version-Release number of selected component (if applicable):
AWS with OCP 4.12.0-rc.4with OVN
ACM 2.7.0
Submariner 0.14.1
How reproducible:
?
Steps to Reproduce:
- Install OCP 4.12 on AWS with OVN
- Install ACM 2.7.0
- Install Submariner 0.14.1
Full scenario:
Actual results:
oc get nodes -l node-role.kubernetes.io/worker -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-16-156-56.us-west-1.compute.internal Ready worker 78m v1.25.4+86bd4ff 10.16.156.56 <none> Red Hat Enterprise Linux CoreOS 412.86.202212081411-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.25.1-5.rhaos4.12.git6005903.el8 ip-10-16-163-121.us-west-1.compute.internal Ready worker 81m v1.25.4+86bd4ff 10.16.163.121 <none> Red Hat Enterprise Linux CoreOS 412.86.202212081411-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.25.1-5.rhaos4.12.git6005903.el8 ip-10-16-243-53.us-west-1.compute.internal Ready worker 78m v1.25.4+86bd4ff 10.16.243.53 <none> Red Hat Enterprise Linux CoreOS 412.86.202212081411-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.25.1-5.rhaos4.12.git6005903.el8 ip-10-16-60-108.us-west-1.compute.internal Ready worker 21m v1.25.4+86bd4ff 10.16.60.108 52.53.225.212 Red Hat Enterprise Linux CoreOS 412.86.202212081411-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.25.1-5.rhaos4.12.git6005903.el8
OC Events for this AWS node shows:
2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 daemonset-controller Normal SuccessfulCreate Created pod: node-exporter-l5rsf <nil> <nil> <none> <none> Normal Scheduled Successfully assigned openshift-monitoring/node-exporter-l5rsf to ip-10-16-60-108.us-west-1.compute.internal 2022-12-12T18:21:13Z 2022-12-12T18:21:18Z 3 controlplane Warning ErrorAddingLogicalPort addLogicalPort failed for openshift-multus/network-metrics-daemon-6b7r5: unable to parse node L3 gw annotation: k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-16-60-108.us-west-1.compute.internal" <nil> <nil> <none> <none> Normal Scheduled Successfully assigned openshift-multus/network-metrics-daemon-6b7r5 to ip-10-16-60-108.us-west-1.compute.internal 2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 daemonset-controller Normal SuccessfulCreate Created pod: multus-wlsm7 <nil> <nil> <none> <none> Normal Scheduled Successfully assigned openshift-multus/multus-wlsm7 to ip-10-16-60-108.us-west-1.compute.internal 2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 daemonset-controller Normal SuccessfulCreate Created pod: node-ca-6tjr7 <nil> <nil> <none> <none> Normal Scheduled Successfully assigned openshift-image-registry/node-ca-6tjr7 to ip-10-16-60-108.us-west-1.compute.internal <nil> <nil> <none> <none> Normal Scheduled Successfully assigned openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bzxqf to ip-10-16-60-108.us-west-1.compute.internal 2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 daemonset-controller Normal SuccessfulCreate Created pod: node-resolver-gg782 2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 submariner-agent-submarineragentconfigcontroller Normal SubmarinerConfigStatusUpdated Updated status conditions: []v1.Condition{v1.Condition{Type:"SubmarinerClusterEnvironmentPrepared", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 10, 0, time.Local), Reason:"SubmarinerClusterEnvPrepared", Message:"Submariner cluster environment was prepared"}, v1.Condition{Type:"SubmarinerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 12, 0, time.Local), Reason:"SubmarinerConfigApplied", Message:"SubmarinerConfig was applied"}, v1.Condition{Type:"SubmarinerGatewaysLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 21, 13, 0, time.Local), Reason:"Success", Message:"1 node(s) (\"ip-10-16-60-108.us-west-1.compute.internal\") are labeled as gateways"}} 2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 controlplane Warning ErrorAddingLogicalPort addLogicalPort failed for openshift-network-diagnostics/network-check-target-qt6c6: timed out waiting for logical switch in logical switch cache "ip-10-16-60-108.us-west-1.compute.internal" subnet: error getting logical switch for node ip-10-16-60-108.us-west-1.compute.internal: switch not in logical switch cache 2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 submariner-agent-submarineragentstatuscontroller Normal ManagedClusterAddOnStatusUpdated Updated status conditions: []v1.Condition{v1.Condition{Type:"RegistrationApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 9, 0, time.Local), Reason:"RegistrationConfigured", Message:"Registration of the addon agent is configured"}, v1.Condition{Type:"ManifestApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 10, 0, time.Local), Reason:"AddonManifestApplied", Message:"manifests of addon are applied successfully"}, v1.Condition{Type:"ClusterCertificateRotated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 9, 0, time.Local), Reason:"ClientCertificateUpdated", Message:"client certificate rotated starting from 2022-12-12 18:12:09 +0000 UTC to 2022-12-13 17:01:21 +0000 UTC"}, v1.Condition{Type:"SubmarinerBrokerConfigApplied", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 12, 0, time.Local), Reason:"BrokerConfigApplied", Message:""}, v1.Condition{Type:"SubmarinerGatewayNodesLabeled", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 21, 13, 0, time.Local), Reason:"SubmarinerGatewayNodesLabeled", Message:"The nodes \"ip-10-16-60-108.us-west-1.compute.internal\" are labeled with \"submariner.io/gateway\""}, v1.Condition{Type:"Available", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.December, 12, 18, 17, 33, 0, time.Local), Reason:"ManagedClusterAddOnLeaseUpdated", Message:"submariner add-on is available."}} 2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 daemonset-controller Normal SuccessfulCreate Created pod: network-metrics-daemon-6b7r5 2022-12-12T18:21:00Z 2022-12-12T18:21:13Z 8 kubelet Normal NodeHasSufficientMemory Node ip-10-16-60-108.us-west-1.compute.internal status is now: NodeHasSufficientMemory <nil> <nil> <none> <none> Normal Scheduled Successfully assigned openshift-network-diagnostics/network-check-target-qt6c6 to ip-10-16-60-108.us-west-1.compute.internal <nil> <nil> <none> <none> Normal Scheduled Successfully assigned openshift-ovn-kubernetes/ovnkube-node-f2d6d to ip-10-16-60-108.us-west-1.compute.internal <nil> <nil> <none> <none> Normal Scheduled Successfully assigned openshift-machine-config-operator/machine-config-daemon-h8d8b to ip-10-16-60-108.us-west-1.compute.internal <nil> <nil> <none> <none> Normal Scheduled Successfully assigned openshift-dns/node-resolver-gg782 to ip-10-16-60-108.us-west-1.compute.internal 2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 daemonset-controller Normal SuccessfulCreate Created pod: multus-additional-cni-plugins-v552s 2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 daemonset-controller Normal SuccessfulCreate Created pod: aws-ebs-csi-driver-node-bzxqf <nil> <nil> <none> <none> Normal Scheduled Successfully assigned openshift-cluster-node-tuning-operator/tuned-zbplw to ip-10-16-60-108.us-west-1.compute.internal 2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 daemonset-controller Normal SuccessfulCreate Created pod: machine-config-daemon-h8d8b <nil> <nil> <none> <none> Normal Scheduled Successfully assigned openshift-multus/multus-additional-cni-plugins-v552s to ip-10-16-60-108.us-west-1.compute.internal 2022-12-12T18:21:13Z 2022-12-12T18:21:58Z 16 controlplane Warning ErrorReconcilingNode [k8s.ovn.org/node-chassis-id annotation not found for node ip-10-16-60-108.us-west-1.compute.internal, macAddress annotation not found for node "ip-10-16-60-108.us-west-1.compute.internal" , k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-16-60-108.us-west-1.compute.internal"] 2022-12-12T18:21:00Z 2022-12-12T18:21:13Z 8 kubelet Normal NodeHasNoDiskPressure Node ip-10-16-60-108.us-west-1.compute.internal status is now: NodeHasNoDiskPressure 2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 daemonset-controller Normal SuccessfulCreate Created pod: tuned-zbplw 2022-12-12T18:21:13Z 2022-12-12T18:21:13Z 1 daemonset-controller Normal SuccessfulCreate Created pod: network-check-target-qt6c6 2022-12-12T18:21:16Z 2022-12-12T18:21:16Z 1 node-controller Normal RegisteredNode Node ip-10-16-60-108.us-west-1.compute.internal event: Registered Node ip-10-16-60-108.us-west-1.compute.internal in Controller 2022-12-12T18:21:18Z 2022-12-12T18:21:18Z 2 controlplane Warning ErrorAddingLogicalPort addLogicalPort failed for openshift-network-diagnostics/network-check-target-qt6c6: unable to parse node L3 gw annotation: k8s.ovn.org/l3-gateway-config annotation not found for node "ip-10-16-60-108.us-west-1.compute.internal" 2022-12-12T18:21:18Z 2022-12-12T18:21:50Z 7 kubelet Warning FailedMount MountVolume.SetUp failed for volume "kube-api-access-f7wxg" : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Expected results:
Gateway should be created on the AWS node
Additional info:
- is caused by
-
ACM-2483 Failure adding managed cluster: Klusterlet unable to list AppliedManifestWorks
- Closed