Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-40369

[cnv-tests] Debug failure of 'running_vm' function

XMLWordPrintable

    • 3
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • ---
    • ---
    • CNV-net-QE-257, CNV-net-QE-259, CNV-net-QE-260
    • No

      Currently, in many of our tests where it is possible, we start the VM in the fixture where we create the using a call to `vm.start()`, and afterwards we verify it completed its start using a later call to `running_vm()`.
      This methodology saves time because it enables starting more than one VM in parallel, but there are occasions where it leads to a race, and the VM start ends with a failure.
      We need to avoid starting the VMs with `start()` (except for specific places where it is required to do it like that), and instead start our VMs in an orderly manner using `running_vm()`.

      Another solution discussed here:
      https://redhat-internal.slack.com/archives/C01953U55QC/p1723013536771989

      The failure stack is:

      File "/cnv-tests/tests/network/connectivity/utils.py", line 45, in create_running_vm
          running_vm(vm=vm, wait_for_cloud_init=True)
        File "/cnv-tests/utilities/virt.py", line 1792, in running_vm
          wait_for_running_vm(
        File "/cnv-tests/utilities/virt.py", line 1719, in wait_for_running_vm
          wait_for_vm_interfaces(vmi=vm.vmi)
        File "/cnv-tests/utilities/virt.py", line 124, in wait_for_vm_interfaces
          if wait_for_guest_agent(vmi=vmi, timeout=timeout):
        File "/cnv-tests/utilities/virt.py", line 94, in wait_for_guest_agent
          for sample in sampler:
        File "/cnv-tests/.venv/lib/python3.9/site-packages/ocp_resources/utils.py", line 153, in __iter__
          raise TimeoutExpiredError(self._get_exception_log(exp=last_exp))
      ocp_resources.utils.TimeoutExpiredError: Timed Out: 720
      Function: utilities.virt.<locals>.lambda: vmi.instance
      Last exception: N/A: None 

              sbahar Shahaf Bahar
              ysegev@redhat.com Yossi Segev
              Nir Rozen Nir Rozen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Estimated:
                  Original Estimate - 3 days
                  3d
                  Remaining:
                  Remaining Estimate - 3 days
                  3d
                  Logged:
                  Time Spent - Not Specified
                  Not Specified