What were you trying to do that didn't work?
In OpenShift Virtualization, I'm querying the metric 'kubevirt_vmi_vcpu_wait_seconds_total'. All values are 0.
When querying libvirt directly, I can confirm that the returned value is 0:
# virsh -r domstats centos9 | grep wait vcpu.0.wait=0 <-- vcpu.0.halt_wait_ns.sum=147337004989 vcpu.1.wait=0 <-- vcpu.1.halt_wait_ns.sum=163969618614
What is the impact of this issue to you?
Incorrect metrics for VMs.
Please provide the package NVR for which the bug is seen:
libvirt-10.0.0-6.7.el9_4.x86_64
qemu-kvm-8.2.0-11.el9_4.6.x86_64
kernel-5.14.0-427.37.1.el9_4.x86_64
How reproducible is this bug?:
100%
Steps to reproduce
- Using a RHEL 9.4 hypervisor, add the kernel cmdline option 'schedstats=enable'
- Run a VM and generate IO load.
- Check the vCPU wait stats: "virsh -r domstats VM_NAME |grep wait"
Expected results
IO wait correctly reported.
Actual results
IO wait is 0, however the kernel is reporting IO wait in the 'wait_sum' stat:
# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt3)/vmlinuz-5.14.0-427.37.1.el9_4.x86_64 root=UUID=aec1c1e8-3576-4eb2-ab62-f62984e655a2 console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M schedstats=enable
# virsh -r domstats centos9 |grep wait
vcpu.0.wait=0
vcpu.0.halt_wait_ns.sum=147337004989
vcpu.1.wait=0
vcpu.1.halt_wait_ns.sum=163969618614
# pstree -t -p 4506
qemu-kvm(4506)─┬─{CPU 0/KVM}(4518)
├─{CPU 1/KVM}(4519)
├─{IO mon_iothread}(4517)
├─{qemu-kvm}(4513)
├─{vnc_worker}(4521)
├─{worker}(4514)
├─{worker}(4573)
├─{worker}(4575)
├─{worker}(4576)
├─{worker}(4577)
├─{worker}(4578)
├─{worker}(4579)
├─{worker}(4580)
├─{worker}(4581)
├─{worker}(4582)
└─{worker}(4583)
# grep wait /proc/4506/task/{4518,4519}/sched
/proc/4506/task/4518/sched:wait_start : 0.000000
/proc/4506/task/4518/sched:wait_max : 4.657302
/proc/4506/task/4518/sched:wait_sum : 951.535557
/proc/4506/task/4518/sched:wait_count : 162006
/proc/4506/task/4518/sched:iowait_sum : 0.000000
/proc/4506/task/4518/sched:iowait_count : 0
/proc/4506/task/4519/sched:wait_start : 0.000000
/proc/4506/task/4519/sched:wait_max : 3.558880
/proc/4506/task/4519/sched:wait_sum : 865.273765
/proc/4506/task/4519/sched:wait_count : 154549
/proc/4506/task/4519/sched:iowait_sum : 0.000000
/proc/4506/task/4519/sched:iowait_count : 0
- is blocked by
-
RHEL-50574 Rebase libvirt in RHEL-9.6.0
-
- In Progress
-
- is depended on by
-
CNV-49442 kubevirt_vmi_vcpu_wait_seconds_total metric is always 0
-
- Closed
-
- is duplicated by
-
-
- Closed
-
- links to
