Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-80321

Failed to generate trace.dat for guest with error 'reading client -110 (Unknown error -110)'

Linking RHIVOS CVEs to...Migration: Automation ...RHELPRIO AssignedTeam ...SWIFT: POC ConversionSync from "Extern...XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Undefined Undefined
    • rhel-10.1
    • rhel-10.0
    • trace-cmd
    • None
    • trace-cmd-3.3.1-3.el10
    • No
    • Moderate
    • 1
    • rhel-kernel-tracing
    • ssg_core_kernel
    • 12
    • 16
    • 3
    • QE ack, Dev ack
    • False
    • False
    • Hide

      None

      Show
      None
    • No
    • CK-April-2025
    • Unspecified Release Note Type - Unknown
    • Unspecified
    • Unspecified
    • Unspecified
    • x86_64
    • None

      Host:
      kernel-rt-6.12.0-55.el10.x86_64
      libtraceevent-1.8.4-3.el10.x86_64
      libtracefs-1.8.0-6.el10.x86_64
      trace-cmd-3.3.1-2.el10.x86_64
      libbabeltrace-1.5.11-9.el10.x86_64

      reproduced step:
      1. set up a guest with vsock port and start it
      <devices>
      <vsock model='virtio'>
      <cid auto='no' address='3'/>
      </vsock>
      </devices>

      2. run 'trace-cmd agent' in guest

      3. run below command to collect cyclictest test logs.

      trace-cmd record --poll -m 1000 -e sched_switch -e hrtimer_start -e hrtimer_expire_entry -e hrtimer_expire_exit -e irq_handler_entry -e local_timer_entry -e local_timer_exit -e reschedule_entry -e reschedule_exit -e call_function_entry -e call_function_exit -e call_function_single_entry -e call_function_single_exit -e irq_work_entry -e irq_work_exit -e tick_stop -e ipi_send_cpumask -e kvm_exit -e kvm_entry -e ipi_send_cpu -e csd_queue_cpu -e csd_function_entry -e csd_function_exit -f "cpu==3 || cpu==5 || cpu==7 || cpu==9 || cpu==11 || cpu==13 || cpu==15 || cpu==17 || cpu==19 || cpu==43 || cpu==45 || cpu==47 || cpu==49 || cpu==51 || cpu==53 || cpu==55 || cpu==57 || cpu==59" -A 3 -e sched_switch -e hrtimer_start -e hrtimer_expire_entry -e hrtimer_expire_exit -e irq_handler_entry -e local_timer_entry -e local_timer_exit -e reschedule_entry -e reschedule_exit -e call_function_entry -e call_function_exit -e call_function_single_entry -e call_function_single_exit -e irq_work_entry -e irq_work_exit -e tick_stop -e ipi_send_cpumask -e ipi_send_cpu -e csd_queue_cpu -e csd_function_entry -e csd_function_exit -f "cpu==2 || cpu==3 || cpu==4 || cpu==5 || cpu==6 || cpu==7 || cpu==8 || cpu==9 || cpu==10 || cpu==11 || cpu==12 || cpu==13 || cpu==14 || cpu==15 || cpu==16 || cpu==17" ssh root@$VM_IP cyclictest -m -q -p95 --policy=fifo -D 10min -h60 -t 16 -a 2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17 -i 200 -b 50 --mainaffinity 0,1 --tracemark

      Actual results:

      Negotiated kvm time sync protocol with guest 3
      reading client -110 (Unknown error -110)
      INFO: debugfs mountpoint: /sys/kernel/debug/tracing/
       # /dev/cpu_dma_latency set to 0us
       # Histogram
      000002 000036 000406 000047 000017 000027 000027 000011 000014 000023 000045 000023 000034 000035 000002 000002 000012
      000003 000033 000646 000078 000058 000065 000032 000017 000011 000074 000219 000056 000027 000045 000012 000031 000023
      000004 001732 014020 035644 035553 035646 001997 001959 002012 001849 015505 002623 038232 005201 001947 002040 035716
      000005 036093 015318 002600 002714 002614 035967 035610 035913 033509 014622 035198 000094 032667 036052 035462 002611
      000006 000487 000319 000020 000048 000039 000355 000784 000434 002920 000547 000483 000000 000436 000368 000842 000024
      000007 000011 000067 000002 000000 000000 000012 000005 000005 000011 000575 000005 000000 000003 000005 000008 000000
      000008 000001 002563 000000 000001 000000 000000 000000 000000 000000 004210 000000 000000 000000 000000 000001 000000
      000009 000000 003400 000000 000000 000000 000000 000000 000000 000002 001330 000000 000000 000000 000000 000000 000000
      000010 000000 001462 000000 000000 000000 000000 000000 000000 000000 001322 000000 000000 000000 000000 000000 000000
      000011 000000 000181 000000 000000 000000 000000 000000 000000 000000 000009 000000 000000 000000 000000 000000 000000
      000012 000000 000005 000000 000000 000000 000000 000000 000000 000000 000002 000000 000000 000000 000000 000000 000000
      000013 000000 000003 000000 000000 000000 000000 000000 000000 000000 000001 000000 000000 000000 000000 000000 000000
      000014 000000 000000 000000 000000 000000 000000 000000 000000 000000 000001 000000 000000 000000 000000 000000 000000
      000016 000000 000002 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
      000017 000000 000000 000001 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000 000000
       # Min Latencies: 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002
       # Avg Latencies: 00004 00005 00004 00004 00004 00004 00004 00004 00005 00005 00004 00003 00004 00004 00004 00004
       # Max Latencies: 00008 00016 00017 00008 00006 00007 00475 00007 00009 00014 00007 00005 00007 00007 00008 00006
       # Histogram Overflows: 00000 00000 00000 00000 00000 00000 00001 00000 00000 00000 00000 00000 00000 00000 00000 00000
       # Histogram Overflow at cycle number:
       # Thread 0:
       # Thread 1:
       # Thread 2:
       # Thread 3:
       # Thread 4:
       # Thread 5:
       # Thread 6: 38386
       # Thread 7:
       # Thread 8:
       # Thread 9:
       # Thread 10:
       # Thread 11:
       # Thread 12:
       # Thread 13:
       # Thread 14:
       # Thread 15:
      
       # Thread Ids: 02381 02382 02383 02384 02385 02386 02387 02388 02389 02390 02391 02392 02393 02394 02395 02396
       # Break thread: 2387
       # Break value: 475
      cannot create output handle
      

      Excepted results:
      succeed to generate trace.dat and trace-3.dat

      Workaroud:
      git checkout 41f90a280c2992d82f9062d1f2dc98c9c74f7313
      from upstream trace-cmd repo and rebuild it.

              jmarchan@redhat.com Jerome Marchand
              junzhao@redhat.com Junyao Zhao
              Jerome Marchand Jerome Marchand
              Qiao Zhao Qiao Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: