Diff View Generated by Perun v0.23.5

2024-10-04 17:45:29 UTC

Baseline (base)

[-] Profile Specification [-]
origin? HEAD
command? iperf3 --bind 172.16.10.3 --affinity 28,28 --client 172.16.10.4 --time 10 --len 16
exitcode? 0
collector command? kperf --repeat=1
kernel? 6.11.0-0.rc5.22.el10.x86_64
boot info? BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.11.0-0.rc5.22.el10.x86_64 root=UUID=7fe82f8a-325a-425d-a486-753311cbe707 ro enforcing=0 ksdevice=bootif modprobe.blacklist=cdc_ether resume=UUID=91439da8-7911-4831-ae0b-9aa7e38ad463 console=ttyS1,115200 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G selinux=0
host? cidic1.slevarna.tpb.lab.eng.brq.redhat.com
cpu (total)? 112
memory (total)? 247GiB
 
[-] Profile Stats [-]
Maximum Trace Length [#] (value)? 24
Overall Inclusive Samples [#] (value)? 15899662075
[empty] [#] (value)? [missing]
throughput [MiB/s] (value)? 188.0000
 
[-] Profile Metadata [-]
BHI? BHI_DIS_S
Family? sapphirerapids
Message_size? 16
NIC? mlx5_core
SELinux? off
Test? single TCP stream
cache-misses? 151,252
cache-references? 4,807,830
 

Target (tgt)

[-] Profile Specification [-]
origin? HEAD
command? iperf3 --bind 172.16.10.3 --affinity 28,28 --client 172.16.10.4 --time 10 --len 16
exitcode? 0
collector command? kperf --repeat=1
kernel? 6.11.0-0.rc5.22.el10.x86_64
boot info? BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.11.0-0.rc5.22.el10.x86_64 root=UUID=7fe82f8a-325a-425d-a486-753311cbe707 ro enforcing=0 ksdevice=bootif modprobe.blacklist=cdc_ether resume=UUID=91439da8-7911-4831-ae0b-9aa7e38ad463 console=ttyS1,115200 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G selinux=0 spectre_bhi=off
host? cidic1.slevarna.tpb.lab.eng.brq.redhat.com
cpu (total)? 112
memory (total)? 247GiB
 
[-] Profile Stats [-]
Maximum Trace Length [#] (value)? 24
Overall Inclusive Samples [#] (value)? 15439019002
[empty] [#] (value)? [missing]
throughput [MiB/s] (value)? 188.0000
 
[-] Profile Metadata [-]
BHI? off
Family? sapphirerapids
Message_size? 16
NIC? mlx5_core
SELinux? off
Test? single TCP stream
cache-misses? 120,030
cache-references? 4,785,798
 
[-] Help / Tips [-]
Tip 1 Click on a row to explore the results in a form of Sankey Graph (below the table).
Tip 2 Click on [+] to open a detailed report about concrete calling contexts.
Tip 3 Filter the results through select form on the bottom of table (for selected columns).

Unit Change Metric Absolute Difference Relative Difference
Search:
[-] Help / Tips [-]
Tip 1 Look for the topmost widest rectangle.
Tip 2 Click on the node to explore the results in a form of Sankey Graph (below the table).
Tip 3 Colours are picked randomly (but uniformly); they don't signal any scale.
Tip 4 The bigger the rectangle the more often the UID was seen on the stack.
Tip 5 The higher the flame the more nested calls were made in the context.

Baseline Flamegraph Reset ZoomSearchic clock_gettime@@GLIBC_2.17 (163,830,844 samples, 1.03%)tcp_rate_check_app_limited (45,281,988 samples, 0.28%)tcp_sendmsg_locked (4,987,444,375 samples, 31.37%)tcp_sendmsg_locked__check_object_size.part.0 (512,179,332 samples, 3.22%)poll_select_finish (200,870,506 samples, 1.26%)__x64_sys_write (30,240,507 samples, 0.19%)_copy_from_user (167,911,943 samples, 1.06%)__check_object_size.part.0 (58,707,132 samples, 0.37%)tcp_established_options (206,972,408 samples, 1.30%)iperf_send (344,265,546 samples, 2.17%)core_sys_select (1,561,601,662 samples, 9.82%)core_s..iperf_run_client (54,350,343 samples, 0.34%)sk_page_frag_refill (118,540,440 samples, 0.75%)read_tsc (58,497,323 samples, 0.37%)tcp_send_mss (945,972,917 samples, 5.95%)tc..ktime_get (589,418,975 samples, 3.71%)k..do_syscall_64 (2,175,105,049 samples, 13.68%)do_syscal..ktime_get_ts64 (48,759,108 samples, 0.31%)do_syscall_64 (139,620,766 samples, 0.88%)inet_sendmsg (138,497,951 samples, 0.87%)_raw_spin_lock_irqsave (75,362,250 samples, 0.47%)_raw_spin_unlock_bh (58,708,593 samples, 0.37%)__local_bh_enable_ip (154,057,661 samples, 0.97%)security_file_permission (130,734,595 samples, 0.82%)remove_wait_queue (104,016,664 samples, 0.65%)inet_send_prepare (49,613,311 samples, 0.31%)entry_SYSCALL_64_safe_stack (119,021,481 samples, 0.75%)sock_poll (385,801,275 samples, 2.43%)Maximum (Baseline, Target) (15,899,662,075 samples, 100%)ktime_get_ts64 (80,073,513 samples, 0.50%)__select (152,856,190 samples, 0.96%)ipv4_mtu (127,423,692 samples, 0.80%)tcp_sendmsg (5,999,866,161 samples, 37.74%)tcp_sendmsgread_tsc (31,428,779 samples, 0.20%)skb_page_frag_refill (85,266,796 samples, 0.54%)do_pselect.constprop.0 (123,503,306 samples, 0.78%)__virt_addr_valid (140,140,986 samples, 0.88%)write@plt (32,814,568 samples, 0.21%)read_tsc (430,552,445 samples, 2.71%)tcp_release_cb (46,919,609 samples, 0.30%)rw_verify_area (218,948,617 samples, 1.38%)fput (47,164,459 samples, 0.30%)release_sock (365,303,003 samples, 2.30%)select_estimate_accuracy (145,534,767 samples, 0.92%)tcp_write_xmit (1,355,060,187 samples, 8.52%)tcp_..[[vdso]] (148,210,382 samples, 0.93%)tcp_push (72,769,238 samples, 0.46%)sock_write_iter (6,870,366,478 samples, 43.21%)sock_write_iter__audit_syscall_exit (72,287,595 samples, 0.45%)entry_SYSRETQ_unsafe_stack (68,883,224 samples, 0.43%)__x64_sys_pselect6 (1,996,149,812 samples, 12.55%)__x64_sy.._raw_spin_lock_bh (236,429,129 samples, 1.49%)entry_SYSCALL_64_after_hwframe (140,638,658 samples, 0.88%)get_timespec64 (120,615,780 samples, 0.76%)syscall_exit_to_user_mode (648,550,581 samples, 4.08%)s..tcp_poll (35,034,760 samples, 0.22%)sock_poll (37,070,925 samples, 0.23%)syscall_exit_to_user_mode (104,157,971 samples, 0.66%)iperf_time_now (38,710,083 samples, 0.24%)_copy_from_user (96,068,278 samples, 0.60%)do_syscall_64 (9,560,434,201 samples, 60.13%)do_syscall_64syscall_exit_work (35,832,464 samples, 0.23%)__x64_sys_pselect6 (123,503,306 samples, 0.78%)__tcp_push_pending_frames (1,410,735,259 samples, 8.87%)__tcp..ksys_write (8,167,589,443 samples, 51.37%)ksys_write_raw_spin_lock_irqsave (101,058,563 samples, 0.64%)tcp_tso_segs (164,094,672 samples, 1.03%)[unknown] (15,742,532,622 samples, 99.01%)[unknown]entry_SYSCALL_64 (129,037,355 samples, 0.81%)_raw_spin_lock_bh (195,893,503 samples, 1.23%)lock_sock_nested (325,846,300 samples, 2.05%)bpf_lsm_file_permission (28,712,409 samples, 0.18%)tmr_run (29,342,892 samples, 0.18%)security_socket_sendmsg (173,165,112 samples, 1.09%)poll_freewait (180,721,755 samples, 1.14%)_copy_to_user (72,041,296 samples, 0.45%)__select (2,466,917,230 samples, 15.52%)__selecttcp_poll (361,310,190 samples, 2.27%)iperf_check_throttle (145,241,406 samples, 0.91%)syscall_trace_enter (39,842,869 samples, 0.25%)__memset (43,582,049 samples, 0.27%)vfs_write (7,789,339,862 samples, 48.99%)vfs_writeentry_SYSCALL_64_after_hwframe (9,689,657,403 samples, 60.94%)entry_SYSCALL_64_after_hwframeread_tsc (57,720,663 samples, 0.36%)tcp_current_mss (799,526,202 samples, 5.03%)tc.._copy_from_iter (892,690,632 samples, 5.61%)_c..core_sys_select (120,449,107 samples, 0.76%)__fdelt_warn (40,972,287 samples, 0.26%)__GI___libc_write (11,705,504,305 samples, 73.62%)__GI___libc_writeNwrite (228,789,000 samples, 1.44%)__cond_resched (27,984,313 samples, 0.18%)do_select (72,598,483 samples, 0.46%)tcp_wmem_schedule (41,557,822 samples, 0.26%)check_stack_object (27,913,668 samples, 0.18%)entry_SYSCALL_64_after_hwframe (2,191,309,245 samples, 13.78%)entry_SYS..x64_sys_call (65,931,981 samples, 0.41%)syscall_return_via_sysret (172,253,862 samples, 1.08%)syscall_exit_work (249,219,589 samples, 1.57%)syscall_trace_enter (387,377,947 samples, 2.44%)__fdget_pos (209,306,559 samples, 1.32%)put_timespec64 (90,586,020 samples, 0.57%)__audit_syscall_entry (186,063,447 samples, 1.17%)do_select (979,575,922 samples, 6.16%)do_..__pollwait (90,351,997 samples, 0.57%)check_heap_object (388,486,351 samples, 2.44%)do_pselect.constprop.0 (1,989,289,656 samples, 12.51%)do_psele..audit_reset_context (77,006,464 samples, 0.48%)ktime_get_coarse_real_ts64 (54,645,218 samples, 0.34%)tcp_mtu_probe (28,422,222 samples, 0.18%)iperf_tcp_send (350,901,276 samples, 2.21%)skb_do_copy_data_nocache (1,511,680,844 samples, 9.51%)skb_d..add_wait_queue (132,968,683 samples, 0.84%)entry_SYSCALL_64 (1,307,778,208 samples, 8.23%)entr..syscall_exit_to_user_mode_prepare (40,736,788 samples, 0.26%)ktime_get_ts64 (83,376,747 samples, 0.52%)
Target Flamegraph Reset ZoomSearchic ktime_get_ts64 (65,454,425 samples, 0.41%)__check_object_size.part.0 (503,684,266 samples, 3.17%)__select (151,461,441 samples, 0.95%)entry_SYSRETQ_unsafe_stack (47,456,439 samples, 0.30%)tcp_poll (334,692,622 samples, 2.11%)sock_poll (357,497,488 samples, 2.25%)do_pselect.constprop.0 (114,377,455 samples, 0.72%)syscall_exit_to_user_mode (645,048,561 samples, 4.06%)s..tcp_write_xmit (1,327,222,886 samples, 8.35%)tcp_..syscall_exit_to_user_mode (94,378,204 samples, 0.59%)release_sock (294,053,713 samples, 1.85%)tcp_push (79,139,248 samples, 0.50%)entry_SYSCALL_64_after_hwframe (9,483,397,338 samples, 59.65%)entry_SYSCALL_64_after_hwframeskb_page_frag_refill (97,324,526 samples, 0.61%)ipv4_mtu (118,520,915 samples, 0.75%)read_tsc (428,523,514 samples, 2.70%)_raw_spin_lock_irqsave (93,860,011 samples, 0.59%)__check_object_size.part.0 (68,002,130 samples, 0.43%)_raw_spin_lock_bh (166,737,580 samples, 1.05%)is_vmalloc_addr (35,953,544 samples, 0.23%)syscall_trace_enter (36,457,640 samples, 0.23%)entry_SYSCALL_64 (1,291,764,914 samples, 8.12%)entr..tmr_timeout (51,739,234 samples, 0.33%)iperf_tcp_send (232,369,272 samples, 1.46%)lock_sock_nested (313,837,583 samples, 1.97%)core_sys_select (109,197,676 samples, 0.69%)_raw_spin_unlock_bh (50,382,829 samples, 0.32%)read_tsc (76,114,407 samples, 0.48%)syscall_return_via_sysret (117,052,907 samples, 0.74%)__GI___libc_write (11,440,887,906 samples, 71.96%)__GI___libc_write__pollwait (104,850,274 samples, 0.66%)vfs_write (7,633,617,980 samples, 48.01%)vfs_writesecurity_file_permission (151,374,607 samples, 0.95%)poll_freewait (173,521,763 samples, 1.09%)sk_page_frag_refill (120,951,227 samples, 0.76%)do_select (62,338,498 samples, 0.39%)put_timespec64 (97,700,951 samples, 0.61%)inet_sendmsg (145,429,111 samples, 0.91%)_copy_from_user (63,521,570 samples, 0.40%)do_select (931,365,215 samples, 5.86%)do..tcp_send_mss (958,640,707 samples, 6.03%)tc..audit_reset_context (81,075,160 samples, 0.51%)__audit_syscall_exit (57,517,297 samples, 0.36%)tcp_current_mss (844,335,610 samples, 5.31%)tc..Nwrite (227,592,642 samples, 1.43%)read_tsc (49,296,183 samples, 0.31%)__x64_sys_pselect6 (114,377,455 samples, 0.72%)_copy_from_iter (840,718,508 samples, 5.29%)_c..tcp_wmem_schedule (40,299,028 samples, 0.25%)sock_write_iter (6,671,367,737 samples, 41.96%)sock_write_iterdo_pselect.constprop.0 (1,918,974,691 samples, 12.07%)do_psel..security_socket_sendmsg (161,693,212 samples, 1.02%)ktime_get_ts64 (39,134,590 samples, 0.25%)ktime_get (585,223,555 samples, 3.68%)k.._raw_spin_lock_bh (230,019,518 samples, 1.45%)do_syscall_64 (127,609,797 samples, 0.80%)entry_SYSCALL_64_after_hwframe (128,627,906 samples, 0.81%)[[vdso]] (178,855,063 samples, 1.12%)__local_bh_enable_ip (172,834,754 samples, 1.09%)__x64_sys_pselect6 (1,921,129,987 samples, 12.08%)__x64_s..syscall_exit_work (223,307,934 samples, 1.40%)poll_select_finish (199,471,240 samples, 1.25%)entry_SYSCALL_64_safe_stack (101,752,084 samples, 0.64%)_copy_to_user (78,905,490 samples, 0.50%)entry_SYSCALL_64 (134,174,738 samples, 0.84%)syscall_trace_enter (400,742,905 samples, 2.52%)__fdelt_warn (34,207,836 samples, 0.22%)[unknown] (15,281,773,220 samples, 96.11%)[unknown]__audit_syscall_entry (165,512,245 samples, 1.04%)inet_send_prepare (53,099,346 samples, 0.33%)iperf_check_throttle (171,452,783 samples, 1.08%)tcp_established_options (276,768,436 samples, 1.74%)ksys_write (7,940,473,266 samples, 49.94%)ksys_writektime_get_coarse_real_ts64 (59,688,956 samples, 0.38%)do_syscall_64 (2,096,267,584 samples, 13.18%)do_sysca..tcp_mtu_probe (27,656,246 samples, 0.17%)iperf_time_now (39,856,931 samples, 0.25%)__memset (38,468,579 samples, 0.24%)__fdget_pos (156,627,978 samples, 0.99%)skb_do_copy_data_nocache (1,443,654,152 samples, 9.08%)skb_d..Maximum (Baseline, Target) (15,899,662,075 samples, 100%)tcp_sendmsg (5,833,535,154 samples, 36.69%)tcp_sendmsgtcp_sendmsg_locked (4,913,918,635 samples, 30.91%)tcp_sendmsg_lockedktime_get_ts64 (92,437,330 samples, 0.58%)core_sys_select (1,539,878,038 samples, 9.68%)core_..__virt_addr_valid (159,867,193 samples, 1.01%)x64_sys_call (87,961,614 samples, 0.55%)do_syscall_64 (9,339,510,554 samples, 58.74%)do_syscall_64_raw_spin_lock_irqsave (61,952,400 samples, 0.39%)__select (2,391,050,250 samples, 15.04%)__selectcheck_heap_object (406,003,796 samples, 2.55%)entry_SYSCALL_64_after_hwframe (2,127,823,453 samples, 13.38%)entry_SY..get_timespec64 (84,925,857 samples, 0.53%)rw_verify_area (234,532,547 samples, 1.48%)tcp_rate_check_app_limited (52,734,369 samples, 0.33%)iperf_send (297,452,415 samples, 1.87%)fput (60,119,440 samples, 0.38%)add_wait_queue (122,635,240 samples, 0.77%)_copy_from_user (194,235,753 samples, 1.22%)__tcp_push_pending_frames (1,386,311,208 samples, 8.72%)__tcp..tcp_tso_segs (146,327,744 samples, 0.92%)select_estimate_accuracy (146,593,882 samples, 0.92%)syscall_exit_to_user_mode_prepare (46,516,164 samples, 0.29%)remove_wait_queue (85,991,741 samples, 0.54%)clock_gettime@@GLIBC_2.17 (193,117,275 samples, 1.21%)iperf_run_client (37,743,490 samples, 0.24%)tcp_release_cb (32,445,302 samples, 0.20%)sock_poll (28,622,958 samples, 0.18%)