-
Bug
-
Resolution: Unresolved
-
Normal
-
ACM 2.10.0, ACM 2.11.0
-
None
-
False
-
None
-
False
-
-
-
Submariner Sprint 2024-26, Submariner Sprint 2024-27, Submariner Sprint 2024-28
-
None
Description of problem:
A. I deployed Submariner - nonGN (ACM2.11) on two ROKS clusters
B. Add another gateway pod on one of the clusters (by labeling node)
C. Reboot the node where the active GW is running
Noticed that :
- [1] error is reported in RouteAgent pod running on the node that becomes active GW.
- IPsec connections status is in errored state on this node
Checking the Routing table on the new gateway node , it seems that the routing rules used for host networking (table 150) are missing [2] , restarting the RA pod running on the GW node fixed this issue.
[1]
024-08-13T07:21:44.075Z DBG ../pkg/ipset/ipset.go:208 IPSet Running ipset [del SUBMARINER-LOCALCIDRS 172.30.0.0/16] 2024-08-13T07:21:44.159Z INF ../endpoint_created.go:58 EventController Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90 2024-08-13T07:21:44.159Z INF ../endpoint_created.go:58 EventController Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90 2024-08-13T07:21:44.159Z DBG ../pkg/ipset/ipset.go:208 IPSet Running ipset [add -exist SUBMARINER-LOCALCIDRS 172.25.0.0/16] 2024-08-13T07:21:44.159Z INF ../endpoint_created.go:58 EventController Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90 2024-08-13T07:21:44.159Z INF ..o/ippool_handler.go:118 CalicoIPPool TransitionToGateway 2024-08-13T07:21:44.159Z INF ../endpoint_created.go:58 EventController Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90 2024-08-13T07:21:44.159Z DBG ..oxy/gw_transition.go:44 KubeProxy The current node has become a Gateway 2024-08-13T07:21:44.159Z INF ..oxy/gw_transition.go:48 KubeProxy Creating the vxlan interface: vx-submariner on the gateway node 2024-08-13T07:21:44.160Z DBG ../pkg/vxlan/vxlan.go:192 VxlanAPI Successfully added the bridge fdb entry 10.244.0.89 00:00:00:00:00:00 2024-08-13T07:21:44.160Z DBG ../pkg/vxlan/vxlan.go:192 VxlanAPI Successfully added the bridge fdb entry 10.244.0.90 00:00:00:00:00:00 2024-08-13T07:21:44.160Z DBG ../pkg/ipset/ipset.go:208 IPSet Running ipset [add -exist SUBMARINER-LOCALCIDRS 172.30.0.0/16] 2024-08-13T07:21:44.160Z DBG ../pkg/vxlan/vxlan.go:192 VxlanAPI Successfully added the bridge fdb entry 10.244.0.91 00:00:00:00:00:00 2024-08-13T07:21:44.161Z DBG ../pkg/ipset/ipset.go:208 IPSet Running ipset [add -exist SUBMARINER-LOCALCIDRS 172.30.0.0/16] 2024-08-13T07:21:44.162Z INF ../endpoint_created.go:58 EventController Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90 2024-08-13T07:21:44.269Z WRN ..o/ippool_handler.go:124 CalicoIPPool Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83 2024-08-13T07:21:44.269Z ERR ../endpoint_created.go:44 EventController Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : [error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org), error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.28.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)]" 2024-08-13T07:21:44.362Z INF ..s/kubeproxy/vxlan.go:79 KubeProxy Successfully configured reverse path filter to loose mode on "vx-submariner" 2024-08-13T07:21:44.370Z INF ../endpoint_created.go:58 EventController Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90 2024-08-13T07:21:44.370Z INF ..o/ippool_handler.go:118 CalicoIPPool TransitionToGateway 2024-08-13T07:21:44.548Z WRN ..o/ippool_handler.go:124 CalicoIPPool Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83 2024-08-13T07:21:44.549Z ERR ../endpoint_created.go:44 EventController Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)" 2024-08-13T07:21:44.749Z INF ../endpoint_created.go:58 EventController Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90 2024-08-13T07:21:44.749Z INF ..o/ippool_handler.go:118 CalicoIPPool TransitionToGateway 2024-08-13T07:21:44.920Z WRN ..o/ippool_handler.go:124 CalicoIPPool Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83 2024-08-13T07:21:44.920Z ERR ../endpoint_created.go:44 EventController Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)" 2024-08-13T07:21:45.321Z INF ../endpoint_created.go:58 EventController Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90 2024-08-13T07:21:45.321Z INF ..o/ippool_handler.go:118 CalicoIPPool TransitionToGateway 2024-08-13T07:21:45.493Z WRN ..o/ippool_handler.go:124 CalicoIPPool Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83 2024-08-13T07:21:45.493Z ERR ../endpoint_created.go:44 EventController Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)" 2024-08-13T07:21:46.294Z INF ../endpoint_created.go:58 EventController Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90 2024-08-13T07:21:46.294Z INF ..o/ippool_handler.go:118 CalicoIPPool TransitionToGateway 2024-08-13T07:21:46.470Z WRN ..o/ippool_handler.go:124 CalicoIPPool Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83 2024-08-13T07:21:46.470Z ERR ../endpoint_created.go:44 EventController Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.28.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)" 2024-08-13T07:21:48.071Z INF ../endpoint_created.go:58 EventController Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90 2024-08-13T07:21:48.071Z INF ..o/ippool_handler.go:118 CalicoIPPool TransitionToGateway 2024-08-13T07:21:48.230Z WRN ..o/ippool_handler.go:124 CalicoIPPool Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83 2024-08-13T07:21:48.230Z ERR ../endpoint_created.go:44 EventController Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)"
[2]
sh-4.4# ip r show table 150 Error: ipv4: FIB table does not exist. Dump terminated
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
- ...