-
Task
-
Resolution: Done
-
Major
-
None
-
None
test_connectivity_after_nncp_change sometimes passes and sometimes fails.
We need to debug it, and find whether the root cause of this flakiness is in the tests (and in that case - fix it), or in the product (and in that case - issue a BZ).
This a snippet from a run where this tests fails:
It starts in the setup section, where ping between VMs.
Then a change is done in the NNCP, and after the NNCP is applied successfully - the test call is executed with another ping attempt - which ends with 100% packet loss:
ocp_resources.node_network_configuration_policy.wait_for_configuration_conditions_unknown_or_progressing.lambda: self.instance.status.conditions) 14:03:48 2023-08-04T11:03:19.235203 ocp_resources.utils INFO Elapsed time: 1.012794017791748 14:03:48 2023-08-04T11:03:19.235465 ocp_resources.utils INFO Waiting for 480 seconds, retry every 1 seconds. (Function: ocp_resources.node_network_configuration_policy.wait_for_status_success.lambda: self.status) 14:03:48 2023-08-04T11:03:27.330180 ocp_resources NodeNetworkConfigurationPolicy INFO NNCP restart-nmstate-infd-vrf-414t2-h7vht-worker-0-m4rnb configured Successfully 14:03:48 2023-08-04T11:03:27.330672 ocp_resources.utils INFO Elapsed time: 8.0898277759552Traceback (most recent call last): 14:03:48 File "/cnv-tests/tests/network/nmstate/test_connectivity_after_nmstate_changes.py", line 320, in test_connectivity_after_nncp_change 14:03:48 for sample in TimeoutSampler( 14:03:48 File "/cnv-tests/.venv/lib/python3.9/site-packages/ocp_resources/utils.py", line 152, in __iter__ 14:03:48 raise TimeoutExpiredError(self._get_exception_log(exp=last_exp)) 14:03:48 ocp_resources.utils.TimeoutExpiredError: Timed Out: 10 14:03:48 Function: utilities.network.is_destination_pingable_from_vm Kwargs: {'src_vm': <ocp_utilities.infra.DynamicClassCreator.__call__.<locals>.BaseResource object at 0x7efc08352520>, 'dst_ip': IPv4Address('10.200.0.2'), 'count': 10} 14:03:48 Last exception: N/A: None 14:04:26 14:04:26 ----------------------------- Captured stderr call ----------------------------- 14:04:26 --------------------------------------------------------- CALL --------------------------------------------------------- 14:04:26 2023-08-04T11:03:27.335294 ocp_resources.utils INFO Waiting for 10 seconds, retry every 1 seconds. (Function: utilities.network.is_destination_pingable_from_vm Kwargs: {'src_vm': <ocp_utilities.infra.DynamicClassCreator.__call__.<locals>.BaseResource object at 0x7efc08352520>, 'dst_ip': IPv4Address('10.200.0.2'), 'count': 10}) 14:04:26 2023-08-04T11:03:27.335535 utilities.virt INFO Username: fedora, password: password, SSH key: /cnv-tests/utilities/cnv-qe-jenkins.key 14:04:26 SSH command: ssh -o 'ProxyCommand=virtctl port-forward --stdio=true vma-1691146891-67266.nmstate-test-connectivity-after-nmstate-changes 22' fedora@vma-1691146891-67266 14:04:26 2023-08-04T11:03:27.418356 Host INFO [vma-1691146891-67266] Executing command ping -q -c 10 10.200.0.2 14:04:26 2023-08-04T11:03:27.519769 paramiko.transport INFO Connected (version 2.0, client OpenSSH_9.0) 14:04:26 2023-08-04T11:03:27.564967 paramiko.transport INFO Authentication (publickey) successful! 14:04:26 2023-08-04T11:03:47.127615 Host ERROR [vma-1691146891-67266] Failed to run command ['ping', '-q', '-c', '10', '10.200.0.2'] ERR: OUT: PING 10.200.0.2 (10.200.0.2) 56(84) bytes of data. 14:04:26 14:04:26 --- 10.200.0.2 ping statistics --- 14:04:26 10 packets transmitted, 0 received, 100% packet loss, time 9220ms 14:04:26 14:04:26 14:04:26 2023-08-04T11:03:47.128379 utilities.network INFO ping returned 10 packets transmitted, 0 received, 100% packet loss, time 9220ms
