On SNO spoke with telco DU profile applied, oslat reported 20us latency spike on a 1h run
    

OCP 4.16.0-0.nightly-2024-02-26-013420
cluster-logging.v5.9.0
local-storage-operator.v4.16.0-202403050942
packageserver
ptp-operator.v4.16.0-202403050942
sriov-fec.v2.8.0
sriov-network-operator.v4.16.0-202403050942

    

Multiple
    

    1. Deploy DU node
    2. Run OSLAT test
    

    One sample at 20us
    

    All samples below 20us
    

############# dumping env ###########
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_PORT=443
RUNTIME_SECONDS=1h
HOSTNAME=oslat0
NSS_SDB_USE_CACHE=no
DISTTAG=f39container
PWD=/root
TRACE_THRESHOLD=
container=oci
INITIAL_DELAY_SEC=30
HOME=/root
KUBERNETES_PORT_443_TCP=tcp://[fd02::1]:443
FGC=f39
tool=oslat
manual=n
PRIO=1
TERM=xterm
SHLVL=1
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=fd02::1
KUBERNETES_SERVICE_HOST=fd02::1
KUBERNETES_PORT=tcp://[fd02::1]:443
KUBERNETES_PORT_443_TCP_PORT=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
delay=60
_=/usr/bin/env
#####################################
 
########## container info ###########
/proc/cmdline:
BOOT_IMAGE=(hd2,gpt3)/ostree/rhcos-c93172ea2fd2a9678115e61863f53f6c3ddd9480a800d82a05487e67f320655d/vmlinuz-5.14.0-424.el9.x86_64+rt ignition.platform.id=metal ostree=/ostree/boot.0/rhcos/c93172ea2fd2a9678115e61863f53f6c3ddd9480a800d82a05487e67f320655d/0 root=UUID=ecba3874-f410-4a52-a9b3-df1685718abb rw rootflags=prjquota boot=UUID=a96d9c7c-8f98-4f89-8014-9a96b6789954 crashkernel=512M intel_iommu=on iommu=pt skew_tick=1 tsc=reliable rcupdate.rcu_normal_after_boot=1 nohz=on rcu_nocbs=2-19,22-39 tuned.non_isolcpus=00300003 systemd.cpu_affinity=0,1,20,21 intel_iommu=on iommu=pt isolcpus=managed_irq,2-19,22-39 nohz_full=2-19,22-39 tsc=reliable nosoftlockup nmi_watchdog=0 mce=off skew_tick=1 rcutree.kthread_prio=11 default_hugepagesz=1G hugepagesz=1G hugepages=32 rcupdate.rcu_normal_after_boot=0 vfio_pci.enable_sriov=1 vfio_pci.disable_idle_d3=1 efi=runtime module_blacklist=irdma intel_pstate=disable tsc=reliable systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller=1
#####################################
oslat0 5.14.0-424.el9.x86_64+rt
realtime-tests-2.5-3.fc39.x86_64
allowed cpu list: 2-9,22-29
removing cpu22 from the cpu list because it is a sibling of cpu2 which will be the cpu-main-thread
new cpu list: 3,4,5,6,7,8,9,23,24,25,26,27,28,29
cmd to run: oslat -D 1h --rtprio 1 --cpu-list 3,4,5,6,7,8,9,23,24,25,26,27,28,29 --cpu-main-thread 2  
sleep 60 before test
oslat V 2.50
Total runtime: 		3600 seconds
Thread priority: 	SCHED_FIFO:1
CPU list: 		3,4,5,6,7,8,9,23,24,25,26,27,28,29
CPU for main thread: 	2
Workload: 		no
Workload mem: 		0 (KiB)
Preheat cores: 		14

Pre-heat for 1 seconds...
Test starts...
Test completed.

        Core:	 3 4 5 6 7 8 9 23 24 25 26 27 28 29
Counter Freq:	 1400 1400 1400 1400 1400 1400 1400 1400 1400 1400 1400 1400 1400 1400 (Mhz)
    001 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    002 (us):	 110617313046 110777571000 110648587455 110135403042 109779651579 109689688143 110396610660 110617005138 110777382581 110648197969 110135155127 109779651109 109689686845 110396887336
    003 (us):	 7 9 4 3 3 3 4 4 6 6 4 3 3 4
    004 (us):	 9 8 7 7 7 6 9 12 11 11 9 7 7 6
    005 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    006 (us):	 0 0 0 0 0 1 0 0 1 0 0 0 1 1
    007 (us):	 2 2 2 2 2 1 2 2 1 2 2 2 1 1
    008 (us):	 1 0 0 0 0 0 0 0 0 0 0 0 0 0
    009 (us):	 0 0 1 0 0 0 0 1 0 0 0 0 0 0
    010 (us):	 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    011 (us):	 1 2 0 0 0 0 1 0 0 0 0 0 0 1
    012 (us):	 0 0 1 0 1 0 1 1 1 0 0 0 0 0
    013 (us):	 0 0 1 0 1 0 0 0 0 0 1 0 0 0
    014 (us):	 0 1 0 0 0 1 0 0 0 0 0 0 1 1
    015 (us):	 1 0 0 1 0 0 1 0 0 1 0 0 0 0
    016 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    017 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    018 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    019 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    020 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    021 (us):	 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    022 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    023 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    024 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    025 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    026 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    027 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    028 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    029 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    030 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    031 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    032 (us):	 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (including overflows)
     Minimum:	 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (us)
     Average:	 2.000 2.000 2.000 2.000 2.000 2.000 2.000 2.000 2.000 2.000 2.000 2.000 2.000 2.000 (us)
     Maximum:	 14 13 12 14 12 13 14 20 11 14 12 6 13 13 (us)
     Max-Min:	 13 12 11 13 11 12 13 19 10 13 11 5 12 12 (us)
    Duration:	 3599.991 3599.991 3599.991 3599.991 3599.991 3599.991 3599.991 3599.991 3599.991 3599.991 3599.991 3599.991 3599.991 3599.991 (sec)