Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-34022

cyclictest 58us spike 1h run on 4.16.0-rc.2

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Undefined Undefined
    • None
    • 4.16
    • Telco Performance
    • None
    • Important
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • 8/2 - It appears this issue was caused by PTP clock sync events during the test. Should probably be closed.

      Description of problem:

      On SNO spoke with telco DU profile applied, cyclictest reported 58us latency spike on a 1h run.  

      Version-Release number of selected component (if applicable):

      4.16.0-rc.2    

      How reproducible:

      observed once so far

      Steps to Reproduce:

          1. Deploy SNO with DU profile
    2. Run cyclictest
    3.

      Actual results:

      ########## container info ###########
/proc/cmdline:
BOOT_IMAGE=(hd0,gpt3)/boot/ostree/rhcos-8c1652e096f3f863b552d56648a5d39b62609912dab0ba2e3b357792dc160faa/vmlinuz-5.14.0-427.16.1.el9_4.x86_64+rt ignition.platform.id=metal ostree=/ostree/boot.1/rhcos/8c1652e096f3f863b552d56648a5d39b62609912dab0ba2e3b357792dc160faa/0 root=UUID=cf4b0580-72e9-4e87-a31e-feadb09774ac rw rootflags=prjquota boot=UUID=f92cc40a-5c3e-4643-9b32-aa461a118b0f systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all psi=1 crashkernel=512M intel_iommu=on iommu=pt skew_tick=1 tsc=reliable rcupdate.rcu_normal_after_boot=1 nohz=on rcu_nocbs=2-31,34-63 tuned.non_isolcpus=00000003,00000003 systemd.cpu_affinity=0,1,32,33 intel_iommu=on iommu=pt isolcpus=managed_irq,2-31,34-63 nohz_full=2-31,34-63 tsc=reliable nosoftlockup nmi_watchdog=0 mce=off skew_tick=1 rcutree.kthread_prio=11 default_hugepagesz=1G hugepagesz=1G hugepages=32 rcupdate.rcu_normal_after_boot=0 vfio_pci.enable_sriov=1 vfio_pci.disable_idle_d3=1 efi=runtime module_blacklist=irdma intel_pstate=active tsc=reliable systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all psi=1
#####################################
**** uid: 0 ****
cyclictest0 5.14.0-427.16.1.el9_4.x86_64+rt
realtime-tests-2.5-3.fc39.x86_64
allowed cpu list: 2-9,34-41
removing cpu34 from the cpu list because it is a sibling of cpu2 which will be the mainaffinity
new cpu list: 3,4,5,6,7,8,9,35,36,37,38,39,40,41
running cmd: cyclictest -q -D 1h -p 95 -t 14 -a 3,4,5,6,7,8,9,35,36,37,38,39,40,41 -h 30 -i 1000 --mainaffinity 2 -m  -b 20 --tracemark --smi
sleep 60 before test
# /dev/cpu_dma_latency set to 0us
# Histogram
000000 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000001 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000002 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000003 000948    000962    000941    000964    000957    000942    000919    000929    000975    000968    000968    000962    000971    000960
000004 308963    308976    309129    309107    309114    309040    309086    309182    309112    309100    309087    309112    309116    309123
000005 000175    000163    000035    000042    000050    000124    000103    000035    000042    000038    000052    000030    000043    000043
000006 000045    000027    000050    000044    000047    000037    000036    000043    000055    000042    000044    000038    000042    000047
000007 000126    000129    000102    000100    000089    000115    000114    000068    000073    000109    000106    000114    000085    000084
000008 000000    000000    000000    000000    000000    000000    000000    000001    000001    000001    000000    000001    000000    000000
000009 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000010 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000011 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000012 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000013 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000014 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000015 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000001    000000    000000    000000
000016 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000017 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000018 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000019 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000020 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000021 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000022 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000023 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000024 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000025 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000001    000000    000000
000026 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000027 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000028 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
000029 000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000    000000
# Total: 000310257 000310257 000310257 000310257 000310257 000310258 000310258 000310258 000310258 000310258 000310258 000310258 000310257 000310257
# Min Latencies: 00003 00003 00003 00003 00003 00003 00003 00003 00003 00003 00003 00003 00003 00003
# Avg Latencies: 00003 00003 00003 00003 00003 00003 00003 00003 00003 00003 00003 00003 00003 00003
# Max Latencies: 00058 00058 00058 00058 00055 00058 00058 00058 00058 00058 00058 00025 00034 00044
# Histogram Overflows: 00001 00001 00001 00001 00001 00001 00001 00001 00001 00001 00001 00000 00001 00001
# Histogram Overflow at cycle number:
# Thread 0: 310257
# Thread 1: 310257
# Thread 2: 310257
# Thread 3: 310257
# Thread 4: 310257
# Thread 5: 310258
# Thread 6: 310258
# Thread 7: 310258
# Thread 8: 310258
# Thread 9: 310258
# Thread 10: 310258
# Thread 11:
# Thread 12: 310257
# Thread 13: 310257
# SMIs: 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000# Thread Ids: 00045 00046 00047 00048 00049 00050 00051 00052 00053 00054 00055 00056 00057 00058
# Break thread: 56
# Break value: 25    

      Expected results:

      No sample is above 20us  

      Additional info:

      kernel trace available at https://s3.upshift.redhat.com/DH-PROD-OCP-EDGE-QE-CI/ocp-far-edge-vran-collect/4972/sno.kni-qe-24.lab.eng.rdu2.redhat.com-cyclictest-kernel-trace.txt.xz   

            rh-ee-vschneid Valentin Schneider
            mcornea@redhat.com Marius Cornea
            Marius Cornea Marius Cornea
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: