When metallb is used on any interface other than br-ex/ovn-k8s-mp0 via L2Advertisement spec.interfaces it is expected, that not only the speakers are just listening on those interfaces, but that the traffic is also correctly forwarded. As the default until 4.13 was to forward all traffic between interfaces, this was working until 4.14. With 4.14. and https://issues.redhat.com/browse/OCPBUGS-16051 we changed the default to not forward anymore between interfaces. This breaks metallb for almost all installations, as it is mostly used on 2ndary interfaces.

Customers coming from 4.13 with an update are not affected as for them a new global parameter is set during upgrade to enable global forwarding (networking.defaultNetwork.ovnKubernetesConfig.gatewayConfig.ipForwarding: Global), whereas the new default is restricting forwarding to Kubernetes related traffic, only.

To mitigate there are two ways:

(1) Change install-config manifests prior to deployment to reflect the new parameter and globally enable forwarding between *all* interfaces.

(2) Add a machine configuration to enable forwarding for required interfaces only, like
---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 81-enable-global-forwarding
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,`echo -e "net.ipv4.conf.br-vms.forwarding = 1\nnet.ipv6.conf.br-vms.forwarding = 1\nnet.ipv4.conf.br-vms.rp_filter = 0\nnet.ipv6.conf.br-vms.rp_filter = 0" | base64 -w0`
           verification: {}
        filesystem: root
        mode: 644
        path: /etc/sysctl.d/enable-global-forwarding.conf
  osImageURL: ""
---

For sure the latter is the better way as it is not exposing the whole system as router!

To resolve this bug there are 2 ways:

(1) metallb should add the required forwarding rules automatically to the nodes, when the spec.interfaces parameter is used.

(2) Add documentation to metallb to make users aware that metallb needs some additional configuration in OpenShift to be usable with 2ndary networks. An example like the above should be added.

For a complete solution I would like the former (1) to be implemented, while the latter (2) should be added to the documentation immediately.

since 4.14

always

    1. install an new 4.14 cluster
    2. install metallb
    3. configure metallb to use a 2ndary interface, e.g. br-vms
    4. create a pod/VM to expose some service
    5. expose the pod/VM via a service using the loadbalancer
    6. try to access the service from outside

see the connection failing

connection to the service is successful