-
Bug
-
Resolution: Done-Errata
-
Normal
-
rhel-9.4
-
qemu-kvm-9.0.0-1.el9
-
None
-
Low
-
Upstream
-
rhel-sst-virtualization-storage
-
ssg_virtualization
-
3
-
False
-
-
None
-
None
-
Pass
-
RegressionOnly
-
None
What were you trying to do that didn't work?
The --trace output of qemu-img/qemu-io goes to stap instead, which is not user-friendly.
Please provide the package NVR for which bug is seen:
qemu-kvm-8.2.0-2.el9
kernel-5.14.0-404.el9.x86_64
How reproducible:
100%
Steps to reproduce
$ qemu-io -f raw -c 'w -n -z 0 5g' nbd://localhost:10804 --trace=nbd_send*
wrote 5368709120/5368709120 bytes at offset 0
5 GiB, 1 ops; 00.04 sec (134.806 GiB/sec and 26.9612 ops/sec)
Expected results
There is trace output of nbd_send related events.
Actual results
As above
- is blocked by
-
RHEL-28073 Rebase qemu-kvm to QEMU 9.0.0 for RHEL 9.5
-
- Closed
-
- split to
-
-
- In Progress
-
-
-
- Closed
-
- links to
-
RHSA-2024:131023
qemu-kvm update
[RHEL-21695] [Qemu] Improve the tracing output of qemu tools(qemu-img/qemu-io)
Eric Blake
added a comment - (In reply to timao@redhat.com from comment-25114153)
Thanks eblake_redhat , so mark the issue here as release pending accordingly.
And regarding packaging adjustment issue, do you need me to file a issue to trace it now?
I've gone ahead and split out two bugs, one to qemu (either we change packaging to pull in dependencies, or we go lighter-weight and change qemu-trace-stap to more gracefully handle lack of stap in the environment to leave the dependency out of the picture on a default install), and a second to systemtap (to have them fix their packaging issues that /usr/bin/stap is provided by both systemtap-client and systemtap-devel (only one RPM should provide a binary, not two) but fails unless the header files of systemtap-devel are installed).
Eric Blake
added a comment - (In reply to timao@redhat.com from comment-25114153 )
Thanks eblake_redhat , so mark the issue here as release pending accordingly.
And regarding packaging adjustment issue, do you need me to file a issue to trace it now?
I've gone ahead and split out two bugs, one to qemu (either we change packaging to pull in dependencies, or we go lighter-weight and change qemu-trace-stap to more gracefully handle lack of stap in the environment to leave the dependency out of the picture on a default install), and a second to systemtap (to have them fix their packaging issues that /usr/bin/stap is provided by both systemtap-client and systemtap-devel (only one RPM should provide a binary, not two) but fails unless the header files of systemtap-devel are installed). Thanks eblake_redhat , so mark the issue here as release pending accordingly.
And regarding packaging adjustment issue, do you need me to file a issue to trace it now?
Eric Blake
added a comment - Yes, that output proves we got it working. It's annoying that the dependencies are not present by default, so I'll discuss with rhn-engineering-berrange on whether we should tweak packaging to make this work easier out-of-the-box, but that can be a separate issue; the original issue of not having the .stp files packaged at all has been addressed.
Eric Blake
added a comment - Yes, that output proves we got it working. It's annoying that the dependencies are not present by default, so I'll discuss with rhn-engineering-berrange on whether we should tweak packaging to make this work easier out-of-the-box, but that can be a separate issue; the original issue of not having the .stp files packaged at all has been addressed. Thanks eblake_redhat , it's indeed because that I have not installed systemtap-devel packages, and after I installed it manually, it works now. But could you please help to take a look again, whether the results here is okay?
Tested with:
qemu-kvm-9.0.0-6.el9
kernel-5.14.0-472.el9
Steps：
On one terminal:
# qemu-img create -f raw test.img 5G
# qemu-nbd -f raw test.img -t &
# qemu-io -f raw nbd://localhost
On anther terminal:
# qemu-trace-stap run `which qemu-io` 'nbd_send*'
And execute write/read operations in the first terminal
# qemu-io -f raw nbd://localhost
qemu-io>
qemu-io> read 0 1G
read 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 00.51 sec (1.958 GiB/sec and 1.9581 ops/sec)
qemu-io> write -P1 0 1G
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:09.55 (107.244 MiB/sec and 0.1047 ops/sec)
Result:
Check the output in the second terminal
# qemu-trace-stap run `which qemu-io` 'nbd_send*'
99637@1720679200103899149 nbd_send_request Sending request to server: { .from = 0, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200120421984 nbd_send_request Sending request to server: { .from = 33554432, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200136454278 nbd_send_request Sending request to server: { .from = 67108864, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200152465707 nbd_send_request Sending request to server: { .from = 100663296, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200168460476 nbd_send_request Sending request to server: { .from = 134217728, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200184250136 nbd_send_request Sending request to server: { .from = 167772160, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200200373638 nbd_send_request Sending request to server: { .from = 201326592, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200216401445 nbd_send_request Sending request to server: { .from = 234881024, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200232260775 nbd_send_request Sending request to server: { .from = 268435456, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200248352531 nbd_send_request Sending request to server: { .from = 301989888, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200264534673 nbd_send_request Sending request to server: { .from = 335544320, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200280373343 nbd_send_request Sending request to server: { .from = 369098752, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200296046443 nbd_send_request Sending request to server: { .from = 402653184, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200311738012 nbd_send_request Sending request to server: { .from = 436207616, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200327793736 nbd_send_request Sending request to server: { .from = 469762048, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200343604520 nbd_send_request Sending request to server: { .from = 503316480, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200359272133 nbd_send_request Sending request to server: { .from = 536870912, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200375190862 nbd_send_request Sending request to server: { .from = 570425344, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200391289288 nbd_send_request Sending request to server: { .from = 603979776, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200407377259 nbd_send_request Sending request to server: { .from = 637534208, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200423588148 nbd_send_request Sending request to server: { .from = 671088640, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200439759183 nbd_send_request Sending request to server: { .from = 704643072, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200455587441 nbd_send_request Sending request to server: { .from = 738197504, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200471661768 nbd_send_request Sending request to server: { .from = 771751936, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200487442738 nbd_send_request Sending request to server: { .from = 805306368, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200503275044 nbd_send_request Sending request to server: { .from = 838860800, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200518958802 nbd_send_request Sending request to server: { .from = 872415232, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200535101574 nbd_send_request Sending request to server: { .from = 905969664, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200550928434 nbd_send_request Sending request to server: { .from = 939524096, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200566778098 nbd_send_request Sending request to server: { .from = 973078528, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200582626117 nbd_send_request Sending request to server: { .from = 1006632960, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }
99637@1720679200598487859 nbd_send_request Sending request to server: { .from = 1040187392, .len = 33554432, .cookie = 1, .flags = 0x0, .type = 0 (read) }99637@1720679237982212579 nbd_send_request Sending request to server: { .from = 0, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679238338723479 nbd_send_request Sending request to server: { .from = 33554432, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679238650626133 nbd_send_request Sending request to server: { .from = 67108864, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679238948287950 nbd_send_request Sending request to server: { .from = 100663296, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679239232561872 nbd_send_request Sending request to server: { .from = 134217728, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679239525408846 nbd_send_request Sending request to server: { .from = 167772160, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679239809245922 nbd_send_request Sending request to server: { .from = 201326592, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679240123054637 nbd_send_request Sending request to server: { .from = 234881024, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679240430609955 nbd_send_request Sending request to server: { .from = 268435456, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679240732665859 nbd_send_request Sending request to server: { .from = 301989888, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679241017536455 nbd_send_request Sending request to server: { .from = 335544320, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679241313543221 nbd_send_request Sending request to server: { .from = 369098752, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679241597250079 nbd_send_request Sending request to server: { .from = 402653184, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679241892212890 nbd_send_request Sending request to server: { .from = 436207616, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679242177273953 nbd_send_request Sending request to server: { .from = 469762048, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679242472913297 nbd_send_request Sending request to server: { .from = 503316480, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679242766174556 nbd_send_request Sending request to server: { .from = 536870912, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679243061951925 nbd_send_request Sending request to server: { .from = 570425344, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679243357109872 nbd_send_request Sending request to server: { .from = 603979776, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679243644241629 nbd_send_request Sending request to server: { .from = 637534208, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679243935153175 nbd_send_request Sending request to server: { .from = 671088640, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679244219635448 nbd_send_request Sending request to server: { .from = 704643072, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679244516355481 nbd_send_request Sending request to server: { .from = 738197504, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679244837229470 nbd_send_request Sending request to server: { .from = 771751936, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679245138671174 nbd_send_request Sending request to server: { .from = 805306368, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679245429112613 nbd_send_request Sending request to server: { .from = 838860800, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679245753793062 nbd_send_request Sending request to server: { .from = 872415232, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679246043120497 nbd_send_request Sending request to server: { .from = 905969664, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679246343999995 nbd_send_request Sending request to server: { .from = 939524096, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679246634379487 nbd_send_request Sending request to server: { .from = 973078528, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679246935958128 nbd_send_request Sending request to server: { .from = 1006632960, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720679247229052032 nbd_send_request Sending request to server: { .from = 1040187392, .len = 33554432, .cookie = 1, .flags = 0x1, .type = 1 (write) }
99637@1720680056636979995 nbd_send_request Sending request to server: { .from = 0, .len = 0, .cookie = 1, .flags = 0x0, .type = 3 (flush) }
99637@1720680056637185074 nbd_send_request Sending request to server: { .from = 0, .len = 0, .cookie = 0, .flags = 0x0, .type = 2 (disconnect) }
While the steps in qemu-kvm-8.2.0-11.el9_4.1, the command below failed directly
# qemu-trace-stap run `which qemu-io` 'nbd_send*' semantic error: while resolving probe point: identifier 'qemu' at <input>:1:7 source: probe qemu.io.log.nbd_send* {} ^semantic error: probe point mismatch (similar: kvm): identifier 'io' at :1:12 source: probe qemu.io.log.nbd_send* {} ^Pass 2: analysis failed. [man error::pass2]
Eric Blake
added a comment - (In reply to timao@redhat.com from comment-24939414)
Ah, still hit errors to execute the command as below. Could you please help to take a look what I missed? eblake_redhat or rhn-engineering-berrange
# qemu-trace-stap run `which qemu-io` nbd_send_option_request /tmp/stap8geNf0/stap_13ec6a5b7da583921c69abce8656d6f7_3711_src.c:12:10: fatal error: runtime_defines.h: No such file or directory 12 | #include "runtime_defines.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated.
What version of systemtap-devel is installed on your test machine? It looks like we've got yet another missing dependency for packages that need to be installed to properly use stap tracing. I'm not sure whether the best approach is to split qemu's `qemu-trace-stap` into its own package that pulls in the needed prerequisites, rather than having qemu-tools add dependencies.
Eric Blake
added a comment - (In reply to timao@redhat.com from comment-24939414 )
Ah, still hit errors to execute the command as below. Could you please help to take a look what I missed? eblake_redhat or rhn-engineering-berrange
# qemu-trace-stap run `which qemu-io` nbd_send_option_request
/tmp/stap8geNf0/stap_13ec6a5b7da583921c69abce8656d6f7_3711_src.c:12:10: fatal error: runtime_defines.h: No such file or directory
12 | #include "runtime_defines.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
What version of systemtap-devel is installed on your test machine? It looks like we've got yet another missing dependency for packages that need to be installed to properly use stap tracing. I'm not sure whether the best approach is to split qemu's `qemu-trace-stap` into its own package that pulls in the needed prerequisites, rather than having qemu-tools add dependencies. Ah, still hit errors to execute the command as below. Could you please help to take a look what I missed? eblake_redhat or rhn-engineering-berrange
Thanks.
# qemu-trace-stap run `which qemu-io` nbd_send_option_request /tmp/stap8geNf0/stap_13ec6a5b7da583921c69abce8656d6f7_3711_src.c:12:10: fatal error: runtime_defines.h: No such file or directory 12 | #include "runtime_defines.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. make[1]: *** [scripts/Makefile.build:296: /tmp/stap8geNf0/stap_13ec6a5b7da583921c69abce8656d6f7_3711_src.o] Error 1 make[1]: *** Waiting for unfinished jobs.... /tmp/stap8geNf0/stap_symbols.c:3:10: fatal error: sym.h: No such file or directory 3 | #include <sym.h> | ^~~~~~~ compilation terminated. make[1]: *** [scripts/Makefile.build:296: /tmp/stap8geNf0/stap_symbols.o] Error 1 make: *** [Makefile:1942: /tmp/stap8geNf0] Error 2 WARNING: kbuild exited with status: 2 Pass 4: compilation failed. [man error::pass4] Kernel version 5.14.0 is within tested range 2.6.32 ... 6.9-rc0
In another terminal
# qemu-img create -f raw test.img 5G
# qemu-nbd -f raw test.img -t
# qemu-io -f raw nbd://localhost
qemu-io>
There's no tracepoint called 'nbd_send':
$ grep 'qemu.io.log.nbd_send' /usr/share/systemtap/tapset/qemu-io-log.stp
probe qemu.io.log.nbd_send_option_request = qemu.io.nbd_send_option_request ?
probe qemu.io.log.nbd_send_request = qemu.io.nbd_send_request ?
So either you need to ask for one of the probes
$ qemu-trace-stap run `which qemu-io` nbd_send_option_request
Or use a wildcard
$ qemu-trace-stap run `which qemu-io` 'nbd_send*'
Daniel Berrangé
added a comment - - edited There's no tracepoint called 'nbd_send':
$ grep 'qemu.io.log.nbd_send' /usr/share/systemtap/tapset/qemu-io-log.stp
probe qemu.io.log.nbd_send_option_request = qemu.io.nbd_send_option_request ?
probe qemu.io.log.nbd_send_request = qemu.io.nbd_send_request ?
So either you need to ask for one of the probes
$ qemu-trace-stap run `which qemu-io` nbd_send_option_request
Or use a wildcard
$ qemu-trace-stap run `which qemu-io` 'nbd_send*' Eric Blake
added a comment - rhn-engineering-berrange can you help with the proper test for using stap traces on qemu-io?
Eric Blake
added a comment - rhn-engineering-berrange can you help with the proper test for using stap traces on qemu-io? Yes, eblake_redhat , the error disapeared after installing systemtap-client, but now I hit another semantic error like below. Could you please help to take another look?
Thanks.
Tested with: qemu-kvm-9.0.0-3.el9 kernel-5.14.0-452.el9 # qemu-trace-stap run `which qemu-io` nbd_send semantic error: while resolving probe point: identifier 'qemu' at <input>:1:7 source: probe qemu.io.log.nbd_send {} ^ semantic error: probe point mismatch (similar: i2c_send, mipsnet_send, multifd_send, nbd_trip, dpcd_read): identifier 'nbd_send' at :1:19 source: probe qemu.io.log.nbd_send {} ^ Pass 2: analysis failed. [man error::pass2]
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory (Moderate: qemu-kvm security update), and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2024:9136