Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-14145

ACM 2.10 Submariner on ROKS: pod with HostNetworking connectivity test fails after GW switchover

XMLWordPrintable

    • Submariner Sprint 2024-27, Submariner Sprint 2024-28, Submariner Sprint 2024-29
    • None

      Description of problem:

      A. I deployed Submariner - nonGN (ACM2.11) on two ROKS clusters

      B. Add another gateway pod on one of the clusters (by labeling node)

      C. Reboot the node where the active GW is running

       

      Noticed that :

      • [1] error is reported in RouteAgent pod running on the node that becomes active GW.
      • IPsec connections status is in errored state on this node

       

      Checking the Routing table on the new gateway node , it seems that the routing rules used for host networking (table 150) are missing  [2] , restarting the RA pod running on the GW node fixed this issue.

      [1] 

      024-08-13T07:21:44.075Z DBG ../pkg/ipset/ipset.go:208 IPSet                Running ipset [del SUBMARINER-LOCALCIDRS 172.30.0.0/16]
2024-08-13T07:21:44.159Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.159Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.159Z DBG ../pkg/ipset/ipset.go:208 IPSet                Running ipset [add -exist SUBMARINER-LOCALCIDRS 172.25.0.0/16]
2024-08-13T07:21:44.159Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.159Z INF ..o/ippool_handler.go:118 CalicoIPPool         TransitionToGateway
2024-08-13T07:21:44.159Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.159Z DBG ..oxy/gw_transition.go:44 KubeProxy            The current node has become a Gateway
2024-08-13T07:21:44.159Z INF ..oxy/gw_transition.go:48 KubeProxy            Creating the vxlan interface: vx-submariner on the gateway node
2024-08-13T07:21:44.160Z DBG ../pkg/vxlan/vxlan.go:192 VxlanAPI             Successfully added the bridge fdb entry 10.244.0.89 00:00:00:00:00:00
2024-08-13T07:21:44.160Z DBG ../pkg/vxlan/vxlan.go:192 VxlanAPI             Successfully added the bridge fdb entry 10.244.0.90 00:00:00:00:00:00
2024-08-13T07:21:44.160Z DBG ../pkg/ipset/ipset.go:208 IPSet                Running ipset [add -exist SUBMARINER-LOCALCIDRS 172.30.0.0/16]
2024-08-13T07:21:44.160Z DBG ../pkg/vxlan/vxlan.go:192 VxlanAPI             Successfully added the bridge fdb entry 10.244.0.91 00:00:00:00:00:00
2024-08-13T07:21:44.161Z DBG ../pkg/ipset/ipset.go:208 IPSet                Running ipset [add -exist SUBMARINER-LOCALCIDRS 172.30.0.0/16]
2024-08-13T07:21:44.162Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.269Z WRN ..o/ippool_handler.go:124 CalicoIPPool         Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83
2024-08-13T07:21:44.269Z ERR ../endpoint_created.go:44 EventController      Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : [error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org), error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.28.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)]"
2024-08-13T07:21:44.362Z INF ..s/kubeproxy/vxlan.go:79 KubeProxy            Successfully configured reverse path filter to loose mode on "vx-submariner"
2024-08-13T07:21:44.370Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.370Z INF ..o/ippool_handler.go:118 CalicoIPPool         TransitionToGateway
2024-08-13T07:21:44.548Z WRN ..o/ippool_handler.go:124 CalicoIPPool         Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83
2024-08-13T07:21:44.549Z ERR ../endpoint_created.go:44 EventController      Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)"
2024-08-13T07:21:44.749Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:44.749Z INF ..o/ippool_handler.go:118 CalicoIPPool         TransitionToGateway
2024-08-13T07:21:44.920Z WRN ..o/ippool_handler.go:124 CalicoIPPool         Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83
2024-08-13T07:21:44.920Z ERR ../endpoint_created.go:44 EventController      Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)"
2024-08-13T07:21:45.321Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:45.321Z INF ..o/ippool_handler.go:118 CalicoIPPool         TransitionToGateway
2024-08-13T07:21:45.493Z WRN ..o/ippool_handler.go:124 CalicoIPPool         Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83
2024-08-13T07:21:45.493Z ERR ../endpoint_created.go:44 EventController      Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)"
2024-08-13T07:21:46.294Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:46.294Z INF ..o/ippool_handler.go:118 CalicoIPPool         TransitionToGateway
2024-08-13T07:21:46.470Z WRN ..o/ippool_handler.go:124 CalicoIPPool         Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83
2024-08-13T07:21:46.470Z ERR ../endpoint_created.go:44 EventController      Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.28.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)"
2024-08-13T07:21:48.071Z INF ../endpoint_created.go:58 EventController      Transitioned to gateway node "kube-cqpn0q1t0kulhnjjfqdg-shaikhdr415-default-00000372" with endpoint private IP 10.244.0.90
2024-08-13T07:21:48.071Z INF ..o/ippool_handler.go:118 CalicoIPPool         TransitionToGateway
2024-08-13T07:21:48.230Z WRN ..o/ippool_handler.go:124 CalicoIPPool         Failed to create ippool dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83
2024-08-13T07:21:48.230Z ERR ../endpoint_created.go:44 EventController      Error handling created endpoint error="error creating Calico IPPool for endpoint \"dr415-odf1-submariner-cable-dr415-odf1-10-241-64-83\" : error creating Calico IPPool for ClusterID \"dr415-odf1\" subnet \"172.24.0.0/16\" (is Calico API server running?): the server is currently unable to handle the request (post ippools.projectcalico.org)"

       [2]

      sh-4.4# ip r show table 150  
Error: ipv4: FIB table does not exist.

Dump terminated

      Version-Release number of selected component (if applicable):

      How reproducible:

      Steps to Reproduce:

      1.  
      2.  
      3. ...

      Actual results:

      Expected results:

      Additional info:

              yboaron Yossi Boaron
              yboaron Yossi Boaron
              Prachi Yadav Prachi Yadav
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: