Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-49607

Concurrent Creation of VMs Through Libvirt Creates Coredump

    • libvirt-10.8.0-1.el10
    • None
    • Moderate
    • rhel-sst-virtualization
    • ssg_virtualization
    • 10
    • 3
    • Dev ack
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • All
    • 10.7.0
    • None

      I am attempting to automate a libvirt reliability/scalability test. In the course of implementing this test, I have discovered a bug.
       
      What Went Wrong
      What was I trying to do?
      Create multiple transient VMs with the same UUID/name simultaneously.
       
      What was the expected outcome?
      One "virsh create" command should succeed, creating a VM. The other(s) should fail, complaining that such a VM already exists.
       ++ 
      What actually happened?
      Virtqemud crashed, resulting in a core dump. Sometimes virtqemud can recover, but it will exist at a new PID. 
       
      The following error message is printed to the console
      error: Disconnected from qemu:///system due to end of file
      error: Failed to destroy domain 'trans-vm'
      error: End of file while reading data: Input/output error
       
      The following is printed to virtqemud's logs

      Jul 17 14:27:27 gigabyte-r181-01.khw.eng.rdu2.dc.redhat.com systemd[1]: virtqemud.service: Main process exited, code=dumped, status=1>
      Jul 17 14:27:27 gigabyte-r181-01.khw.eng.rdu2.dc.redhat.com systemd[1]: virtqemud.service: Failed with result 'core-dump'.
       
      Steps To Reproduce:
      The Hard Way

      1. Define a persistent VM. Do not start it.
      2. Create three vm XML files. Ensure that both of these files have the same name/UUID pair
      3. I have attached examples of the XML files I used in "min_reproduce.tar.xz"
      1. Create three separate scripts that look like this:
      2. end=$((SECONDS+60));
        while [ $SECONDS -lt $end ] ; do
            virsh create ./xml_0.xml 1> /dev/null 2> /dev/null || continue ;
            virsh destroy trans-vm ;
        done ;
      3. NOTE: We create three separate scripts and XML files in order to keep read/write locks from preventing concurrent VM creation
      4. Execute all three scripts simultaneously. EG: script1.sh & script2.sh & script3.sh

       
      The Easy Way
      I have attached a tarball, "min_reproduce.tar.xz," that contains the necessary vm XML files and shell scripts. The contents are designed to be executed on an aarch64 machine. So this method will not work on other architectures 

      1. Untar the file on the correct machine.
      2. Execute "bash main.sh" in the newly created directory 

       
      How Reproducible:
      Always reproducible
       
      Hardware and Versioning
      I (rh-ee-slancast) have confirmed the bug on an aarch64 machine with the following information: * uname -a:  6.10.0-0.rc5.12.el10.aarch64+64k #1 SMP PREEMPT_DYNAMIC Wed Jun 26 06:28:03 UTC 2024 aarch64 GNU/Linux

      • Libvirt Version: libvirt-10.5.0-1.el10.aarch64
      • Qemu-KVM Version: qemu-kvm-9.0.0-4.el10.aarch64

      hanhansolo has confirmed it for x86_64 as well

      • Libvirt Version: libvirt-10.5.0-2.el9.x86_64 

      Related Materials

      Bugs On Related Issue: These bugs prompted the creation of this test case.

      1. https://bugzilla.redhat.com/show_bug.cgi?id=1022924 
      2. https://bugzilla.redhat.com/show_bug.cgi?id=1029632 

      Patch on Above Bugs: https://post-office.corp.redhat.com/archives/rhvirt-patches/2013-November/msg00486.html
       
      Tarball: min_reproduce.tar.xz
      Core Dump: virtqemud.dump
      Coredumpctl Log: virtqemud.dump.log

        1. virtqemud.dump.log
          18 kB
        2. virtqemud.dump
          178.69 MB
        3. min_reproduce.tar.xz
          20 kB

              mprivozn@redhat.com Michal Privoznik
              rh-ee-slancast Stanley Lancaster (Inactive)
              virt-maint virt-maint
              Yan Fu Yan Fu
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

                Created:
                Updated: