Bootstrap and masters come up. Bootstap loops and times out. Masters never join the cluster

Version-Release number of selected component (if applicable):

4.12.32

    

How reproducible:

After many attempts over the WAN, The cluster succeeded . the same build in the local labs worked every time
    

Steps to Reproduce:

    1. ACM used to start the install
    2. Fails in about 1 hour
    3.
    

Actual results:

Expected results:

install completes
    

Additional info:

> journal

> bootstrap node

> Hostname does not change when switching root

May 16 00:41:03 localhost.localdomain systemd[1]: Switching root.
May 16 00:41:03 localhost.localdomain systemd-journald[964]: Journal stopped
May 16 00:41:03 localhost.localdomain systemd[1]: Stopped udev Wait for Complete Device Initialization.
May 16 00:41:03 localhost.localdomain systemd[1]: systemd-udev-trigger.service: Succeeded.
May 16 00:41:03 localhost.localdomain systemd[1]: Stopped udev Coldplug all Devices.
May 16 00:41:03 localhost.localdomain coreos-teardown-initramfs[2362]: info: taking down network device: eno12399
May 16 00:41:03 localhost.localdomain coreos-teardown-initramfs[2388]: RTNETLINK answers: Operation not supported
May 16 00:41:03 localhost.localdomain coreos-teardown-initramfs[2362]: info: taking down network device: eno12409
May 16 00:41:03 localhost.localdomain coreos-teardown-initramfs[2392]: RTNETLINK answers: Operation not supported
May 16 00:41:03 localhost.localdomain coreos-teardown-initramfs[2362]: info: taking down network device: eno12419
May 16 00:41:03 localhost.localdomain coreos-teardown-initramfs[2396]: RTNETLINK answers: Operation not supported

> podman0 link is activated

localhost.localdomain systemd-udevd[3677]: Could not generate persistent MAC address for cni-podman0: No such file or directory
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.6884] manager: (cni-podman0): new Bridge device (/org/freedesktop/NetworkManager/Devices/14)
May 16 00:42:34 localhost.localdomain systemd-udevd[3682]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
May 16 00:42:34 localhost.localdomain systemd-udevd[3682]: Could not generate persistent MAC address for veth4957fcf1: No such file or directory
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.6955] manager: (veth4957fcf1): new Veth device (/org/freedesktop/NetworkManager/Devices/15)
May 16 00:42:34 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
May 16 00:42:34 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP): veth4957fcf1: link is not ready
May 16 00:42:34 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth4957fcf1: link becomes ready
May 16 00:42:34 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
May 16 00:42:34 localhost.localdomain kernel: cni-podman0: port 1(veth4957fcf1) entered blocking state
May 16 00:42:34 localhost.localdomain kernel: cni-podman0: port 1(veth4957fcf1) entered disabled state
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.6961] device (veth4957fcf1): carrier: link connected
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.6963] device (cni-podman0): carrier: link connected
May 16 00:42:34 localhost.localdomain kernel: device veth4957fcf1 entered promiscuous mode
May 16 00:42:34 localhost.localdomain kernel: cni-podman0: port 1(veth4957fcf1) entered blocking state
May 16 00:42:34 localhost.localdomain kernel: cni-podman0: port 1(veth4957fcf1) entered forwarding state
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.6975] device (cni-podman0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.6979] device (cni-podman0): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.6984] device (cni-podman0): Activation: starting connection 'cni-podman0' (ecf0c365-3f0a-46e4-93b7-d0d991e64df2)
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.6985] device (cni-podman0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.6987] device (cni-podman0): state change: prepare -> config (reason 'none', sys-iface-state: 'external')
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.6988] device (cni-podman0): state change: config -> ip-config (reason 'none', sys-iface-state: 'external')
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.6991] device (cni-podman0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.7092] device (cni-podman0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.7095] device (cni-podman0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
May 16 00:42:34 localhost.localdomain NetworkManager[3093]: <info> [1715820154.7101] device (cni-podman0): Activation: successful, device activated.
... lines omitted ...
May 16 00:42:35 localhost.localdomain kernel: cni-podman0: port 1(veth4957fcf1) entered disabled state
May 16 00:42:35 localhost.localdomain kernel: device veth4957fcf1 left promiscuous mode
May 16 00:42:35 localhost.localdomain kernel: cni-podman0: port 1(veth4957fcf1) entered disabled state

> loops through several veth devices