Loading...

Type: Bug
Resolution: Won't Do
Priority: Normal
Fix Version/s: None
Affects Version/s: 4.12
Component/s: Machine Config Operator / platform-openstack
Labels:
- triaged

Test Coverage:

-
Severity:
Moderate
Regression:
None
Blocked:
False

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Description of problem:

When installing with an additional network in the workers (in this case IPv6), the NM-wait-online service is timing out and the additional interface takes a long time to be connected. From the NetworkManager-dispatcher it looks like the /var/run/NetworkManager/resolv.conf took a long time to become available and likely had an impact in connecting the interface.

[core@ostest-6m2dl-worker-0-zvqsl ~]$ sudo journalctl -u NetworkManager-wait-online.service
-- Logs begin at Tue 2022-09-20 11:49:08 UTC, end at Tue 2022-09-20 12:28:19 UTC. --
Sep 20 11:53:09 localhost.localdomain systemd[1]: Starting Network Manager Wait Online...
Sep 20 11:53:30 localhost.localdomain systemd[1]: Started Network Manager Wait Online.
Sep 20 11:55:30 ostest-6m2dl-worker-0-zvqsl systemd[1]: NetworkManager-wait-online.service: Succeeded.
Sep 20 11:55:30 ostest-6m2dl-worker-0-zvqsl systemd[1]: Stopped Network Manager Wait Online.
Sep 20 11:55:30 ostest-6m2dl-worker-0-zvqsl systemd[1]: NetworkManager-wait-online.service: Consumed 0 CPU time
-- Reboot --
Sep 20 11:56:17 ostest-6m2dl-worker-0-zvqsl systemd[1]: Starting Network Manager Wait Online...
Sep 20 11:57:18 ostest-6m2dl-worker-0-zvqsl systemd[1]: NetworkManager-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Sep 20 11:57:18 ostest-6m2dl-worker-0-zvqsl systemd[1]: NetworkManager-wait-online.service: Failed with result 'exit-code'.
Sep 20 11:57:18 ostest-6m2dl-worker-0-zvqsl systemd[1]: Failed to start Network Manager Wait Online.
Sep 20 11:57:18 ostest-6m2dl-worker-0-zvqsl systemd[1]: NetworkManager-wait-online.service: Consumed 60ms CPU time

[core@ostest-6m2dl-worker-0-zvqsl ~]$ sudo nmcli dev
DEVICE                                             TYPE           STATE         CONNECTION
br-ex                                              ovs-interface  connected     ovs-if-br-ex
ens3                                               ethernet       connected     ovs-if-phys0
br-ex                                              ovs-bridge     connected     br-ex                                                                                                                                                        
br-ex                                              ovs-port       connected     ovs-port-br-ex                                                                                                                                               
ens3                                               ovs-port       connected     ovs-port-phys0
ens4                                               ethernet       disconnected  --

[core@ostest-6m2dl-worker-0-zvqsl ~]$ sudo journalctl -u NetworkManager |grep ens4                                                                                                                                                            
Sep 20 11:53:09 localhost.localdomain NetworkManager[1943]: <info>  [1663674789.8717] manager: (ens4): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)                                                                        
Sep 20 11:53:09 localhost.localdomain NetworkManager[1943]: <info>  [1663674789.8783] settings: (ens4): created default wired connection 'Wired connection 2'                                                                                 
Sep 20 11:53:09 localhost.localdomain NetworkManager[1943]: <info>  [1663674789.8788] device (ens4): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')                                                   
Sep 20 11:53:09 localhost.localdomain NetworkManager[1943]: <info>  [1663674789.8809] device (ens4): carrier: link connected                                                                                                                 
Sep 20 11:53:09 localhost.localdomain NetworkManager[1943]: <info>  [1663674789.8984] device (ens4): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
Sep 20 11:53:09 localhost.localdomain NetworkManager[1943]: <info>  [1663674789.9016] device (ens4): Activation: starting connection 'Wired connection 2' (5f0e4070-ca4b-3b51-be94-47364daf238d)
Sep 20 11:53:09 localhost.localdomain NetworkManager[1943]: <info>  [1663674789.9049] device (ens4): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Sep 20 11:53:09 localhost.localdomain NetworkManager[1943]: <info>  [1663674789.9060] device (ens4): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Sep 20 11:53:09 localhost.localdomain NetworkManager[1943]: <info>  [1663674789.9209] device (ens4): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Sep 20 11:53:09 localhost.localdomain NetworkManager[1943]: <info>  [1663674789.9221] dhcp4 (ens4): activation: beginning transaction (timeout in 45 seconds)
Sep 20 11:53:13 localhost.localdomain NetworkManager[1943]: <info>  [1663674793.9643] device (ens4): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Sep 20 11:53:30 localhost.localdomain NetworkManager[1943]: <info>  [1663674810.2789] policy: set 'Wired connection 2' (ens4) as default for IPv6 routing and DNS
Sep 20 11:53:30 localhost.localdomain NetworkManager[1943]: <info>  [1663674810.3427] device (ens4): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Sep 20 11:53:30 localhost.localdomain NetworkManager[1943]: <info>  [1663674810.3430] device (ens4): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Sep 20 11:53:30 localhost.localdomain NetworkManager[1943]: <info>  [1663674810.3439] device (ens4): Activation: successful, device activated.
Sep 20 11:55:30 ostest-6m2dl-worker-0-zvqsl NetworkManager[1943]: <info>  [1663674930.3947] dhcp4 (ens4): canceled DHCP transaction
Sep 20 11:56:17 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663674977.7182] manager: (ens4): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Sep 20 11:56:17 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663674977.7209] settings: (ens4): created default wired connection 'Wired connection 2'
Sep 20 11:56:17 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663674977.7212] device (ens4): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Sep 20 11:56:17 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663674977.7224] device (ens4): carrier: link connected
Sep 20 11:56:17 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663674977.7332] device (ens4): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
Sep 20 11:56:17 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663674977.7400] device (ens4): Activation: starting connection 'Wired connection 2' (5f0e4070-ca4b-3b51-be94-47364daf238d)
Sep 20 11:56:17 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663674977.7444] device (ens4): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Sep 20 11:56:17 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663674977.7453] device (ens4): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Sep 20 11:56:17 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663674977.7511] device (ens4): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Sep 20 11:56:17 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663674977.7525] dhcp4 (ens4): activation: beginning transaction (timeout in 45 seconds)
Sep 20 11:57:03 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663675023.6760] device (ens4): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed')
Sep 20 11:57:03 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <warn>  [1663675023.6765] device (ens4): Activation: failed for connection 'Wired connection 2'
Sep 20 11:57:03 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663675023.6768] device (ens4): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Sep 20 11:57:03 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663675023.6839] dhcp4 (ens4): canceled DHCP transaction

....

Sep 20 12:36:18 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663677378.6796] device (ens4): Activation: starting connection 'Wired connection 2' (5f0e4070-ca4b-3b51-be94-47364daf238d)   
Sep 20 12:36:18 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663677378.6798] device (ens4): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Sep 20 12:36:18 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663677378.6805] device (ens4): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')                
Sep 20 12:36:18 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663677378.6824] device (ens4): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')                 
Sep 20 12:36:18 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663677378.6850] dhcp4 (ens4): activation: beginning transaction (timeout in 45 seconds)                                   
Sep 20 12:36:21 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663677381.3727] device (ens4): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')               
Sep 20 12:36:21 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663677381.5418] device (ens4): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Sep 20 12:36:21 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663677381.5424] device (ens4): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')            
Sep 20 12:36:21 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663677381.5435] policy: set 'Wired connection 2' (ens4) as default for IPv6 routing and DNS                     
Sep 20 12:36:21 ostest-6m2dl-worker-0-zvqsl NetworkManager[1256]: <info>  [1663677381.5436] device (ens4): Activation: successful, device activated.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1. Install OCP with an additional network
2.
3.

Actual results:

Expected results:

Additional info:

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates