Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-26049

When max vcpu is greater than or equal to 246, qemu unable to init event notifier

    • qemu-kvm-8.2.0-7.el9_4
    • None
    • Moderate
    • CustomerScenariosInitiative
    • rhel-sst-virtualization
    • ssg_virtualization
    • 26
    • 30
    • 2
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • Approved Exception
    • All
    • None

      What were you trying to do that didn't work?

      I'm testing max vcpu on ampere-mtjade-altramax-01.khw.eng.rdu2.dc.redhat.com
      [root@ampere-mtjade-altramax-01 ~]# lscpu
      Architecture:           aarch64
        CPU op-mode(s):       32-bit, 64-bit
        Byte Order:           Little Endian
      CPU(s):                 256
        On-line CPU(s) list:  0-255
      Vendor ID:              ARM
        BIOS Vendor ID:       Ampere(R)
        Model name:           Neoverse-N1
          BIOS Model name:    Ampere(R) Altra(R) Max Processor

      When max vcpu is greater than or equal to 246
      (qemu) qemu-kvm: virtio_bus_set_host_notifier: unable to init event
      notifier: Too many open files (-24)
      virtio-scsi: Failed to set host notifier (-24)
      qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
      qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier:
      Too many open files (-24)
      virtio-scsi: Failed to set host notifier (-24)
      qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).

      Please provide the package NVR for which bug is seen:

      hostname: ampere-mtjade-altramax-01.khw.eng.rdu2.dc.redhat.com
      RHEL-9.4.0-20240218.15
      edk2-aarch64-20231122-4.el9.noarch
      qemu version: qemu-kvm-8.2.0-5.el9
      guest kernel: 5.14.0-422.el9.aarch64
      host kernel: 5.14.0-422.el9.aarch64

      How reproducible:

      100%

      Steps to reproduce

      Boot guest with 246 vcpu

      /usr/libexec/qemu-kvm  \
      -name 'avocado-vt-vm1'  \
      -sandbox on \
      -blockdev '{"node-name": "file_aavmf_code", "driver": "file", "filename": "/usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.qcow2", "auto-read-only": true, "discard": "unmap"}' \
      -blockdev '{"node-name": "drive_aavmf_code", "driver": "qcow2", "read-only": true, "file": "file_aavmf_code"}' \
      -blockdev '{"node-name": "file_aavmf_vars", "driver": "file", "filename": "/home/kar/workspace/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel940-aarch64-64k-virtio-scsi_qcow2_filesystem_VARS.qcow2", "auto-read-only": true, "discard": "unmap"}' \
      -blockdev '{"node-name": "drive_aavmf_vars", "driver": "qcow2", "read-only": false, "file": "file_aavmf_vars"}' \
      -machine virt,gic-version=host,pflash0=drive_aavmf_code,pflash1=drive_aavmf_vars,memory-backend=mem-machine_mem \
      -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
      -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
      -nodefaults \
      -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
      -device '{"driver": "virtio-gpu-pci", "bus": "pcie-root-port-1", "addr": "0x0"}' \
      -m 8192 \
      -object '{"size": 8589934592, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
      -smp 246  \
      -cpu 'host' \
      -chardev socket,path=/var/tmp/monitor-qmpmonitor1-20240219-011759-O8gOdseU,wait=off,id=qmp_id_qmpmonitor1,server=on  \
      -mon chardev=qmp_id_qmpmonitor1,mode=control \
      -serial unix:'/var/tmp/serial-serial0',server=on,wait=off \
      -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
      -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-2", "addr": "0x0"}' \
      -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
      -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
      -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-3", "addr": "0x0"}' \
      -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/rhel940-aarch64-64k-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
      -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache":

      {"direct": true, "no-flush": false}

      , "file": "file_image1"}' \
      -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
      -device '{"id": "pcie-root-port-4", "port": 4, "driver": "pcie-root-port", "addr": "0x1.0x4", "bus": "pcie.0", "chassis": 5}' \
      -device '{"driver": "virtio-net-pci", "mac": "9a:07:b6:1d:f6:ae", "rombar": 0, "id": "idRWghpz", "netdev": "idVWHq77", "bus": "pcie-root-port-4", "addr": "0x0"}' \
      -netdev  '{"id": "idVWHq77", "type": "tap", "vhost": true}'  \
      -vnc :20  \
      -rtc base=utc,clock=host \
      -enable-kvm \
      -monitor stdio

      Hit issue:

      (qemu) qemu-kvm: virtio_bus_set_host_notifier: unable to init event
      notifier: Too many open files (-24)
      virtio-scsi: Failed to set host notifier (-24)
      qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
      qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier:
      Too many open files (-24)
      virtio-scsi: Failed to set host notifier (-24)
      qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).

       

      After debugging, I found that this is due to the host's default
      DefaultLimitNOFILE=1024:524288.
      [root@ampere-mtjade-altramax-01 ~]# for pid in $(pidof qemu-kvm); do
      prlimit -p $pid | grep NOFILE; ls -1 /proc/$pid/fd/ | wc -l; done
      NOFILE     max number of open files                1024    524288 files
      1024
      [root@ampere-mtjade-altramax-01 ~]# cat /etc/systemd/system.conf |grep
      DefaultLimitNOFILE
      #DefaultLimitNOFILE=1024:524288

      And I saw a patch with a similar issue:
       [PULL 03/17] qemu_init: increase NOFILE soft limit on POSIX

       

              cohuck@redhat.com Cornelia Huck
              zhenyzha@redhat.com Zhenyu Zhang
              virt-maint virt-maint
              Zhenyu Zhang Zhenyu Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

                Created:
                Updated:
                Resolved: