-
Feature Request
-
Resolution: Unresolved
-
Major
-
None
-
None
-
None
-
Product / Portfolio Work
-
None
-
False
-
-
None
-
None
-
None
-
-
None
-
None
-
None
-
None
-
None
- Proposed title of this feature request
- Using custom targetVRF on RouteAdvertisements results in dysfunctional FRR configuration
- What is the nature and description of the request/Why does the customer need it?
Details: While trying to setup multiple CUDNs to be advertised using a common VRF (in this example "bgp"), the resulting FRR configuration fails to leak the CUDN routes into the "bgp" VRF.Consider the following setupNodeNetworkConfigurationPolicy to create the VRF and bgp peers:
apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: enp2s0-wk1 spec: desiredState: interfaces: - name: bgp state: up type: vrf vrf: route-table-id: 500 - controller: bgp ipv4: address: - ip: 172.21.73.10 prefix-length: 24 dhcp: false enabled: true name: enp2s0 state: up nodeSelector: kubernetes.io/hostname: wk1.ovnbgpt.openinfra.lab
FrrConfiguration to create the BGP peers using the "bgp" VRF:
apiVersion: frrk8s.metallb.io/v1beta1 kind: FRRConfiguration metadata: labels: use-for-advertisements: "true" name: receive-filtered-wk1 spec: bgp: bfdProfiles: - detectMultiplier: 3 name: bfd-default receiveInterval: 200 transmitInterval: 200 routers: - asn: 65519 neighbors: - address: 172.21.73.1 asn: 65110 bfdProfile: bfd-default disableMP: true dualStackAddressFamily: false holdTime: 9s keepaliveTime: 3s toReceive: allowed: mode: all vrf: bgp nodeSelector: matchExpressions: - key: kubernetes.io/hostname operator: In values: - wk1.ovnbgpt.openinfra.lab
The RouteAdvertisment using the "bgp
" VRF:
apiVersion: k8s.ovn.org/v1 kind: RouteAdvertisements metadata: name: extranet spec: advertisements: - PodNetwork frrConfigurationSelector: matchLabels: use-for-advertisements: 'true' networkSelectors: - clusterUserDefinedNetworkSelector: networkSelector: matchLabels: advertise: 'true' networkSelectionType: ClusterUserDefinedNetworks nodeSelector: {} targetVRF: bgp
Results in the following FRRNodestate generated for the FRR router:
apiVersion: frrk8s.metallb.io/v1beta1 kind: FRRNodeState metadata: creationTimestamp: "2025-10-10T18:55:38Z" generation: 1 name: wk1.ovnbgpt.openinfra.lab resourceVersion: "46138717" uid: 2d0060b9-ca1c-40e3-8ed0-623ae0b394fb spec: {} status: lastConversionResult: success lastReloadResult: | 2025-11-10 15:50:19,893 INFO: Called via "Namespace(input=None, reload=False, test=True, debug=False, log_level='info', stdout=True, pathspace=None, filename='/etc/frr_reloader/frr.conf', overwrite=False, bindir='/usr/bin', confdir='/etc/frr', rundir='/var/run/frr', vty_socket=None, daemon='', test_reset=False)" 2025-11-10 15:50:19,893 INFO: Loading Config object from file /etc/frr_reloader/frr.conf 2025-11-10 15:50:19,937 INFO: Loading Config object from vtysh show running 2025-11-10 15:50:19,981 INFO: "frr version 8.5.3" cannot be removed 2025-11-10 15:50:19,982 INFO: "frr defaults traditional" cannot be removed 2025-11-10 15:50:19,982 INFO: "service integrated-vtysh-config" cannot be removed runningConfig: | Building configuration... Current configuration: ! frr version 8.5.3 frr defaults traditional hostname wk1.ovnbgpt.openinfra.lab log file /etc/frr/frr.log informational log timestamp precision 3 service integrated-vtysh-config ! router bgp 65519 vrf bgp no bgp ebgp-requires-policy no bgp default ipv4-unicast bgp graceful-restart preserve-fw-state no bgp network import-check neighbor 172.21.73.1 remote-as 65110 neighbor 172.21.73.1 bfd neighbor 172.21.73.1 bfd profile bfd-default neighbor 172.21.73.1 timers 3 9 ! address-family ipv4 unicast network 172.21.19.0/24 network 172.21.20.0/24 network 172.21.21.0/24 neighbor 172.21.73.1 activate neighbor 172.21.73.1 route-map 172.21.73.1-bgp-in in neighbor 172.21.73.1 route-map 172.21.73.1-bgp-out out import vrf extranet import vrf extranet2 import vrf extranet3 exit-address-family ! address-family ipv6 unicast import vrf extranet import vrf extranet2 import vrf extranet3 exit-address-family exit ! router bgp 65519 no bgp hard-administrative-reset no bgp graceful-restart notification exit ! router bgp 65519 vrf extranet no bgp ebgp-requires-policy no bgp default ipv4-unicast bgp graceful-restart preserve-fw-state no bgp network import-check ! address-family ipv4 unicast import vrf bgp exit-address-family ! address-family ipv6 unicast import vrf bgp exit-address-family exit ! router bgp 65519 vrf extranet2 no bgp ebgp-requires-policy no bgp default ipv4-unicast bgp graceful-restart preserve-fw-state no bgp network import-check ! address-family ipv4 unicast import vrf bgp exit-address-family ! address-family ipv6 unicast import vrf bgp exit-address-family exit ! router bgp 65519 vrf extranet3 no bgp ebgp-requires-policy no bgp default ipv4-unicast bgp graceful-restart preserve-fw-state no bgp network import-check ! address-family ipv4 unicast import vrf bgp exit-address-family ! address-family ipv6 unicast import vrf bgp exit-address-family exit ! ip prefix-list 172.21.73.1-bgp-pl-ipv4 seq 1 permit 172.21.19.0/24 ip prefix-list 172.21.73.1-bgp-pl-ipv4 seq 2 permit 172.21.20.0/24 ip prefix-list 172.21.73.1-bgp-pl-ipv4 seq 3 permit 172.21.21.0/24 ip prefix-list 172.21.73.1-bgp-inpl-ipv4 seq 1 permit any ! ipv6 prefix-list 172.21.73.1-bgp-inpl-ipv4 seq 2 permit any ipv6 prefix-list 172.21.73.1-bgp-pl-ipv4 seq 4 deny any ! route-map 172.21.73.1-bgp-out permit 1 match ip address prefix-list 172.21.73.1-bgp-pl-ipv4 exit ! route-map 172.21.73.1-bgp-out permit 2 match ipv6 address prefix-list 172.21.73.1-bgp-pl-ipv4 exit ! route-map 172.21.73.1-bgp-in permit 3 match ip address prefix-list 172.21.73.1-bgp-inpl-ipv4 exit ! route-map 172.21.73.1-bgp-in permit 4 match ipv6 address prefix-list 172.21.73.1-bgp-inpl-ipv4 exit ! ip nht resolve-via-default ! ipv6 nht resolve-via-default ! bfd profile bfd-default transmit-interval 200 receive-interval 200 exit ! exit ! end
The route leaking was configured on lines 48-50 but the routes from the CUDN VRF referenced are not leaked into the "bgp" VRF. Potentially thats because the routes of interest are of scope local :
sh-5.1# ip route list vrf extranet scope link 172.21.19.0/24 dev ovn-k8s-mp1 proto kernel src 172.21.19.2 sh-5.1# ip route list vrf extranet2 scope link 172.21.20.0/24 dev ovn-k8s-mp4 proto kernel src 172.21.20.2 sh-5.1# ip route list vrf extranet3 scope link 172.21.21.0/24 dev ovn-k8s-mp25 proto kernel src 172.21.21