Loading...

Type: Bug
Resolution: Done
Priority: Normal
Fix Version/s: ACM 2.11.Z
Affects Version/s: ACM 2.10.0, ACM 2.11.0
Component/s: Multicluster Networking
Labels:
None

Blocked:
False
Blocked Reason:
None
Ready:
False
GitHub Issue:
https://github.com/submariner-io/submariner/issues/3120
Git Pull Request:
https://github.com/submariner-io/submariner/pull/3135
Intelligence Requested:
Market:

Sprint:
Submariner Sprint 2024-27, Submariner Sprint 2024-28, Submariner Sprint 2024-29

Target Version:

ACM 2.12.0

Regression:
None

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

A. I deployed Submariner - nonGN (ACM2.11) on two ROKS clusters

B. Add another gateway pod on one of the clusters (by labeling node)

C. Reboot the node where the active GW is running

Noticed that :

[1] error is reported in RouteAgent pod running on the node that becomes active GW.
IPsec connections status is in errored state on this node

Checking the Routing table on the new gateway node , it seems that the routing rules used for host networking (table 150) are missing [2] , restarting the RA pod running on the GW node fixed this issue.

[1]

024-08-13T07:21:44.075Z DBG ../pkg/ipset/ipset.go:208 IPSet                Running ipset [del SUBMARINER-LOCALCIDRS 172.30.0.0/16]
2024-08-13T07:21:44.159Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.159Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.159Z DBG ../pkg/ipset/ipset.go:208 IPSet                Running ipset [add -exist SUBMARINER-LOCALCIDRS 172.25.0.0/16]
2024-08-13T07:21:44.159Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.159Z INF ..o/ippool_handler.go:118 CalicoIPPool         TransitionToGateway
2024-08-13T07:21:44.159Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.159Z DBG ..oxy/gw_transition.go:44 KubeProxy            The current node has become a Gateway
2024-08-13T07:21:44.159Z INF ..oxy/gw_transition.go:48 KubeProxy            Creating the vxlan interface: vx-submariner on the gateway node
2024-08-13T07:21:44.160Z DBG ../pkg/vxlan/vxlan.go:192 VxlanAPI             Successfully added the bridge fdb entry 10.244.0.89 00:00:00:00:00:00
2024-08-13T07:21:44.160Z DBG ../pkg/vxlan/vxlan.go:192 VxlanAPI             Successfully added the bridge fdb entry 10.244.0.90 00:00:00:00:00:00
2024-08-13T07:21:44.160Z DBG ../pkg/ipset/ipset.go:208 IPSet                Running ipset [add -exist SUBMARINER-LOCALCIDRS 172.30.0.0/16]
2024-08-13T07:21:44.160Z DBG ../pkg/vxlan/vxlan.go:192 VxlanAPI             Successfully added the bridge fdb entry 10.244.0.91 00:00:00:00:00:00
2024-08-13T07:21:44.161Z DBG ../pkg/ipset/ipset.go:208 IPSet                Running ipset [add -exist SUBMARINER-LOCALCIDRS 172.30.0.0/16]
2024-08-13T07:21:44.162Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.269Z WRN ..o/ippool_handler.go:124 CalicoIPPool         Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83
2024-08-13T07:21:44.269Z ERR ../endpoint_created.go:44 EventController      Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : [error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org), error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.28.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)]"
2024-08-13T07:21:44.362Z INF ..s/kubeproxy/vxlan.go:79 KubeProxy            Successfully configured reverse path filter to loose mode on "vx-submariner"
2024-08-13T07:21:44.370Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.370Z INF ..o/ippool_handler.go:118 CalicoIPPool         TransitionToGateway
2024-08-13T07:21:44.548Z WRN ..o/ippool_handler.go:124 CalicoIPPool         Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83
2024-08-13T07:21:44.549Z ERR ../endpoint_created.go:44 EventController      Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)"
2024-08-13T07:21:44.749Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.749Z INF ..o/ippool_handler.go:118 CalicoIPPool         TransitionToGateway
2024-08-13T07:21:44.920Z WRN ..o/ippool_handler.go:124 CalicoIPPool         Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83
2024-08-13T07:21:44.920Z ERR ../endpoint_created.go:44 EventController      Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)"
2024-08-13T07:21:45.321Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:45.321Z INF ..o/ippool_handler.go:118 CalicoIPPool         TransitionToGateway
2024-08-13T07:21:45.493Z WRN ..o/ippool_handler.go:124 CalicoIPPool         Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83
2024-08-13T07:21:45.493Z ERR ../endpoint_created.go:44 EventController      Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)"
2024-08-13T07:21:46.294Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:46.294Z INF ..o/ippool_handler.go:118 CalicoIPPool         TransitionToGateway
2024-08-13T07:21:46.470Z WRN ..o/ippool_handler.go:124 CalicoIPPool         Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83
2024-08-13T07:21:46.470Z ERR ../endpoint_created.go:44 EventController      Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.28.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)"
2024-08-13T07:21:48.071Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:48.071Z INF ..o/ippool_handler.go:118 CalicoIPPool         TransitionToGateway
2024-08-13T07:21:48.230Z WRN ..o/ippool_handler.go:124 CalicoIPPool         Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83
2024-08-13T07:21:48.230Z ERR ../endpoint_created.go:44 EventController      Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)"

[2]

sh-4.4# ip r show table 150  
Error: ipv4: FIB table does not exist.

Dump terminated

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

...

Actual results:

Expected results:

Additional info:

clones

ACM-13259 Submariner on ROKS: pod with HostNetworking connectivity test fails after GW switchover

Closed

Details

Description

Description of problem:

[2]

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates