Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-126448

Guest crashed while cutoff the nfs traffic about 100s and restore it

Linking RHIVOS CVEs to...Migration: Automation ...SWIFT: POC ConversionSync from "Extern...XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • rhel-9.4.z, rhel-10.2
    • qemu-kvm / Storage
    • None
    • None
    • None
    • 1
    • rhel-virt-storage
    • 5
    • False
    • False
    • Hide

      None

      Show
      None
    • None
    • VirtStorage Sprint15 CY251203
    • None
    • None
    • Unspecified
    • Unspecified
    • Unspecified
    • None

      What were you trying to do that didn't work?

      Guest crashed while cutoff the nfs tranfic about 100s and restore it, both for windows and rhel guests

      What is the impact of this issue to you?

      Please provide the package NVR for which the bug is seen:

      qemu-kvm-8.2.0-11.el9_4.16

      kernel-5.14.0-427.96.1.el9_4.x86_64

      How reproducible is this bug?:

      100%

      Steps to reproduce

      Preparation:

      # nfsstat -m
      /home/timao/bsod/share from 10.72.140.12:/home/timao/nfs_share
       Flags:    rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.72.140.84,local_lock=none,addr=10.72.140.12

       

      Steps:

      1.Start Win10 guest with the qemu-kvm command below

      /usr/libexec/qemu-kvm \
      -name guest=virtual-machines_qa-py-mtl-node8,debug-threads=on \
      -S \
      -blockdev '{"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
      -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
      -blockdev '{"driver":"file","filename":"/root/avocado/data/avocado-vt/avocado-vt-vm1_win10-64-virtio-ovmf_raw_filesystem_VARS.raw","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
      -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
      -machine pc-q35-rhel9.4.0,usb=off,smm=on,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,acpi=on \
      -accel kvm \
      -cpu EPYC-Genoa,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,spec-ctrl=on,stibp=on,flush-l1d=on,arch-capabilities=on,ssbd=on,cmp-legacy=on,virt-ssbd=on,lbrv=on,tsc-scale=on,vmcb-clean=on,pause-filter=on,pfthreshold=on,v-vmsave-vmload=on,vgif=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,gds-no=on,hv-time=on,tsc-frequency=3245122000,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vpindex=on,hv-runtime=on,hv-synic=on,hv-stimer=on,hv-stimer-direct=on,hv-reset=on,hv-frequencies=on,hv-reenlightenment=on,hv-tlbflush=on,hv-ipi=on \
      -m size=67108864k,maxmem=268435456k \
      -overcommit mem-lock=off \
      -smp 24,sockets=1,dies=1,clusters=1,cores=24,threads=1 \
      -object '{"qom-type":"iothread","id":"iothread1"}' \
      -object '{"qom-type":"memory-backend-ram","id":"ram-node0","size":68719476736}' \
      -numa node,nodeid=0,cpus=0-23,memdev=ram-node0 \
      -uuid 326840b3-949d-5617-8ce2-779a46a12a0d \
      -smbios 'type=1,manufacturer=Red Hat,product=OpenShift Virtualization,version=4.18.13,uuid=326840b3-949d-5617-8ce2-779a46a12a0d,sku=4.18.13,family=Red Hat' \
      -no-user-config \
      -nodefaults \
      -rtc base=utc \
      -global kvm-pit.lost_tick_policy=delay \
      -boot strict=on \
      -device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
      -device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
      -device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
      -device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
      -device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
      -device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' \
      -device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \
      -device '{"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"}' \
      -device '{"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"}' \
      -device '{"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"}' \
      -device '{"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"}' \
      -device '{"driver":"virtio-scsi-pci-non-transitional","id":"scsi0","bus":"pci.5","addr":"0x0"}' \
      -device '{"driver":"virtio-serial-pci-non-transitional","id":"virtio-serial0","bus":"pci.6","addr":"0x0"}' \
      -blockdev '{"driver":"file","filename":"/home/timao/bsod/share/win10-64-virtio-ovmf.raw","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false}}' \
      -blockdev '{"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
      -device '{"driver":"virtio-blk-pci-non-transitional","bus":"pci.7","addr":"0x0","drive":"libvirt-2-format","id":"ua-rootdisk","bootindex":1,"write-cache":"on","werror":"report","rerror":"report"}' \
      \
      -blockdev '{"node-name": "drive_winutils", "driver": "file", "read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/iso/windows/winutils.iso", "cache": {"direct": true, "no-flush": false}}' \
      -device '{"driver": "ide-cd", "id": "winutils", "drive": "drive_winutils", "write-cache": "on", "bus": "ide.1", "unit": 0}' \
      -device '{"driver": "virtio-net-pci", "mac": "9a:91:56:6c:8c:e7", "id": "idI1tvEI", "netdev": "idsgNetH", "bus": "pci.8", "addr": "0x0"}' \
      -netdev  '{"id": "idsgNetH", "type": "tap", "vhost": true}' \
      -vnc :0 \
      -device '{"driver":"bochs-display","id":"video0","vgamem":16777216,"bus":"pcie.0","addr":"0x1"}' \
      -global ICH9-LPC.noreboot=off \
      -watchdog-action reset \
      -device '{"driver":"virtio-balloon-pci-non-transitional","id":"balloon0","free-page-reporting":true,"bus":"pci.9","addr":"0x0"}' \
      -object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
      -device '{"driver":"virtio-rng-pci-non-transitional","rng":"objrng0","id":"rng0","bus":"pci.10","addr":"0x0"}' \
      -sandbox on,obsolete=deny,elevateprivileges=deny,obsolete=deny,resourcecontrol=deny \
      -msg timestamp=on \
      -monitor stdio
      

      2.Running diskspd in guest

      # diskspd.exe -d36000 -w50 -r -t16 -o64 -b4k C:\test\io.dat

      3.Stop the nfs tranfic 100s and cont it

      [root@dell-per660-02 bsod]# cat nfstest.sh 
      nfs_disable() {
        local host=$1
        [ "$host" == "" ] && host=$nfs_host
        # DROP AND REJECT both good
        #cmd="iptables -A OUTPUT -d $host -p tcp --dport 2049 -j DROP"
        
        cmd="iptables -A OUTPUT -d $host -p tcp --dport 2049 -j REJECT"
        echo "$cmd"
        $cmd
      }nfs_enable() {
        local host=$1
        [ "$host" == "" ] && host=$nfs_host
        #cmd="iptables -D OUTPUT -d $host -p tcp --dport 2049 -j DROP"
        cmd="iptables -D OUTPUT -d $host -p tcp --dport 2049 -j REJECT"
        echo "$cmd"
        $cmd
      }
      nfs_switch() {
        #usage nfs_switch timeout
        local i n
        [ "$1" == "" ] && n=9 || n=$1
        echo $n
        nfs_disable
        date
        iptables -L OUTPUT
        for i in $(seq $n); do
          echo -n "$i: "$(date "+%H:%M:%S")" ... "
          sleep 10
          echo $(date "+%H:%M:%S")
        done
        #sleep 90
        date
        nfs_enable
        iptables -L OUTPUT
      }nfs_switch "$@"
      [root@dell-per660-02 bsod]# sh nfstest.sh 10
      

       

      Results:

      Within 5 minutes, the guest hit BSOD with *Stop code: CRITICAL PROCESS DIED*

       

       

      And the kernel dumped file is  http://10.73.156.41/pub/section2/kvm/timao/z1qzMemorybsod.dmp 

       

              shajnocz@redhat.com Stefan Hajnoczi
              timao@redhat.com Tingting Mao
              virt-maint virt-maint
              Tingting Mao Tingting Mao
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated: