When max vcpu is greater than or equal to 246, qemu unable to init event notifier

What were you trying to do that didn't work?

I'm testing max vcpu on ampere-mtjade-altramax-01.khw.eng.rdu2.dc.redhat.com
[root@ampere-mtjade-altramax-01 ~]# lscpu
Architecture:           aarch64
  CPU op-mode(s):       32-bit, 64-bit
  Byte Order:           Little Endian
CPU(s):                 256
  On-line CPU(s) list:  0-255
Vendor ID:              ARM
  BIOS Vendor ID:       Ampere(R)
  Model name:           Neoverse-N1
    BIOS Model name:    Ampere(R) Altra(R) Max Processor

When max vcpu is greater than or equal to 246
(qemu) qemu-kvm: virtio_bus_set_host_notifier: unable to init event
notifier: Too many open files (-24)
virtio-scsi: Failed to set host notifier (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier:
Too many open files (-24)
virtio-scsi: Failed to set host notifier (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).

Please provide the package NVR for which bug is seen:

hostname: ampere-mtjade-altramax-01.khw.eng.rdu2.dc.redhat.com
RHEL-9.4.0-20240218.15
edk2-aarch64-20231122-4.el9.noarch
qemu version: qemu-kvm-8.2.0-5.el9
guest kernel: 5.14.0-422.el9.aarch64
host kernel: 5.14.0-422.el9.aarch64

How reproducible:

100%

Steps to reproduce

Boot guest with 246 vcpu

/usr/libexec/qemu-kvm  \
-name 'avocado-vt-vm1'  \
-sandbox on \
-blockdev '{"node-name": "file_aavmf_code", "driver": "file", "filename": "/usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.qcow2", "auto-read-only": true, "discard": "unmap"}' \
-blockdev '{"node-name": "drive_aavmf_code", "driver": "qcow2", "read-only": true, "file": "file_aavmf_code"}' \
-blockdev '{"node-name": "file_aavmf_vars", "driver": "file", "filename": "/home/kar/workspace/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel940-aarch64-64k-virtio-scsi_qcow2_filesystem_VARS.qcow2", "auto-read-only": true, "discard": "unmap"}' \
-blockdev '{"node-name": "drive_aavmf_vars", "driver": "qcow2", "read-only": false, "file": "file_aavmf_vars"}' \
-machine virt,gic-version=host,pflash0=drive_aavmf_code,pflash1=drive_aavmf_vars,memory-backend=mem-machine_mem \
-device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
-device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
-nodefaults \
-device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
-device '{"driver": "virtio-gpu-pci", "bus": "pcie-root-port-1", "addr": "0x0"}' \
-m 8192 \
-object '{"size": 8589934592, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
-smp 246  \
-cpu 'host' \
-chardev socket,path=/var/tmp/monitor-qmpmonitor1-20240219-011759-O8gOdseU,wait=off,id=qmp_id_qmpmonitor1,server=on  \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-serial unix:'/var/tmp/serial-serial0',server=on,wait=off \
-device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
-device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-2", "addr": "0x0"}' \
-device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
-device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
-device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-3", "addr": "0x0"}' \
-blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/rhel940-aarch64-64k-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
-blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache":

, "file": "file_image1"}' \
-device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
-device '{"id": "pcie-root-port-4", "port": 4, "driver": "pcie-root-port", "addr": "0x1.0x4", "bus": "pcie.0", "chassis": 5}' \
-device '{"driver": "virtio-net-pci", "mac": "9a:07:b6:1d:f6:ae", "rombar": 0, "id": "idRWghpz", "netdev": "idVWHq77", "bus": "pcie-root-port-4", "addr": "0x0"}' \
-netdev  '{"id": "idVWHq77", "type": "tap", "vhost": true}'  \
-vnc :20  \
-rtc base=utc,clock=host \
-enable-kvm \
-monitor stdio

Hit issue:

(qemu) qemu-kvm: virtio_bus_set_host_notifier: unable to init event
notifier: Too many open files (-24)
virtio-scsi: Failed to set host notifier (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier:
Too many open files (-24)
virtio-scsi: Failed to set host notifier (-24)
qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).

After debugging, I found that this is due to the host's default
DefaultLimitNOFILE=1024:524288.
[root@ampere-mtjade-altramax-01 ~]# for pid in $(pidof qemu-kvm); do
prlimit -p $pid | grep NOFILE; ls -1 /proc/$pid/fd/ | wc -l; done
NOFILE     max number of open files                1024    524288 files
1024
[root@ampere-mtjade-altramax-01 ~]# cat /etc/systemd/system.conf |grep
DefaultLimitNOFILE
#DefaultLimitNOFILE=1024:524288

And I saw a patch with a similar issue:
 [PULL 03/17] qemu_init: increase NOFILE soft limit on POSIX