Diff View Generated by Perun v0.23.5

2024-10-04 17:45:54 UTC

Baseline (base)

[-] Profile Specification [-]
origin? HEAD
command? iperf3 --bind 172.16.10.3 --affinity 28,28 --client 172.16.10.4 --time 10 --len 16
exitcode? 0
collector command? kperf --repeat=1
kernel? 6.11.0-0.rc5.22.el10.x86_64
boot info? BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.11.0-0.rc5.22.el10.x86_64 root=UUID=7fe82f8a-325a-425d-a486-753311cbe707 ro enforcing=0 ksdevice=bootif modprobe.blacklist=cdc_ether resume=UUID=91439da8-7911-4831-ae0b-9aa7e38ad463 console=ttyS1,115200 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G
host? cidic1.slevarna.tpb.lab.eng.brq.redhat.com
cpu (total)? 112
memory (total)? 247GiB
 
[-] Profile Stats [-]
Maximum Trace Length [#] (value)? 23
Overall Inclusive Samples [#] (value)? 17901751664
[empty] [#] (value)? [missing]
throughput [MiB/s] (value)? 167.0000
 
[-] Profile Metadata [-]
BHI? BHI_DIS_S
Family? sapphirerapids
Message_size? 16
NIC? mlx5_core
SELinux? Permissive
Test? single TCP stream
cache-misses? 126,794
cache-references? 4,312,751
 

Target (tgt)

[-] Profile Specification [-]
origin? HEAD
command? iperf3 --bind 172.16.10.3 --affinity 28,28 --client 172.16.10.4 --time 10 --len 16
exitcode? 0
collector command? kperf --repeat=1
kernel? 6.11.0-0.rc5.22.el10.x86_64
boot info? BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.11.0-0.rc5.22.el10.x86_64 root=UUID=7fe82f8a-325a-425d-a486-753311cbe707 ro enforcing=0 ksdevice=bootif modprobe.blacklist=cdc_ether resume=UUID=91439da8-7911-4831-ae0b-9aa7e38ad463 console=ttyS1,115200 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G spectre_bhi=off
host? cidic1.slevarna.tpb.lab.eng.brq.redhat.com
cpu (total)? 112
memory (total)? 247GiB
 
[-] Profile Stats [-]
Maximum Trace Length [#] (value)? 25
Overall Inclusive Samples [#] (value)? 16830839734
[empty] [#] (value)? [missing]
throughput [MiB/s] (value)? 177.0000
 
[-] Profile Metadata [-]
BHI? off
Family? sapphirerapids
Message_size? 16
NIC? mlx5_core
SELinux? Permissive
Test? single TCP stream
cache-misses? 827,260
cache-references? 4,474,119
 
[-] Help / Tips [-]
Tip 1 Click on a row to explore the results in a form of Sankey Graph (below the table).
Tip 2 Click on [+] to open a detailed report about concrete calling contexts.
Tip 3 Filter the results through select form on the bottom of table (for selected columns).

Unit Change Metric Absolute Difference Relative Difference
Search:
[-] Help / Tips [-]
Tip 1 Look for the topmost widest rectangle.
Tip 2 Click on the node to explore the results in a form of Sankey Graph (below the table).
Tip 3 Colours are picked randomly (but uniformly); they don't signal any scale.
Tip 4 The bigger the rectangle the more often the UID was seen on the stack.
Tip 5 The higher the flame the more nested calls were made in the context.

Baseline Flamegraph Reset ZoomSearchic check_stack_object (43,500,014 samples, 0.24%)_raw_spin_unlock_bh (45,187,416 samples, 0.25%)entry_SYSCALL_64_safe_stack (96,079,989 samples, 0.54%)avc_has_perm (569,264,148 samples, 3.18%)__virt_addr_valid (162,371,463 samples, 0.91%)__select (2,219,301,095 samples, 12.40%)__selectremove_wait_queue (100,099,140 samples, 0.56%)__audit_syscall_entry (157,316,969 samples, 0.88%)inode_security (120,575,031 samples, 0.67%)entry_SYSCALL_64 (101,772,772 samples, 0.57%)core_sys_select (1,442,295,427 samples, 8.06%)core..tcp_wmem_schedule (46,168,815 samples, 0.26%)clock_gettime@@GLIBC_2.17 (145,735,138 samples, 0.81%)avc_lookup (117,357,038 samples, 0.66%)_raw_spin_lock_irqsave (96,672,333 samples, 0.54%)syscall_exit_to_user_mode (543,525,689 samples, 3.04%)syscall_return_via_sysret (154,060,706 samples, 0.86%)__memset (41,201,250 samples, 0.23%)select_estimate_accuracy (112,505,490 samples, 0.63%)tcp_release_cb (51,600,834 samples, 0.29%)_copy_to_user (53,714,772 samples, 0.30%)x64_sys_call (98,687,257 samples, 0.55%)audit_reset_context (49,281,526 samples, 0.28%)tcp_established_options (164,773,523 samples, 0.92%)ktime_get_ts64 (66,231,581 samples, 0.37%)tcp_send_mss (817,534,925 samples, 4.57%)t..__rcu_read_unlock (54,200,289 samples, 0.30%)read_tsc (39,028,369 samples, 0.22%)avc_lookup (87,583,862 samples, 0.49%)syscall_exit_work (187,286,376 samples, 1.05%)tcp_poll (325,745,128 samples, 1.82%)put_timespec64 (76,850,611 samples, 0.43%)__check_object_size.part.0 (610,540,402 samples, 3.41%)sock_poll (347,039,416 samples, 1.94%)syscall_exit_to_user_mode (97,088,398 samples, 0.54%)__audit_syscall_exit (63,548,795 samples, 0.35%)avc_has_perm_noaudit (373,955,849 samples, 2.09%)inet_sendmsg (186,711,888 samples, 1.04%)security_file_permission (1,409,771,153 samples, 7.88%)secu..entry_SYSRETQ_unsafe_stack (38,976,808 samples, 0.22%)security_socket_sendmsg (1,715,484,074 samples, 9.58%)secur..selinux_socket_sendmsg (1,487,766,391 samples, 8.31%)seli..[[vdso]] (126,092,473 samples, 0.70%)iperf_time_now (35,585,933 samples, 0.20%)__cond_resched (31,981,711 samples, 0.18%)fput (41,721,505 samples, 0.23%)tcp_current_mss (689,757,630 samples, 3.85%)t..ktime_get_ts64 (37,873,435 samples, 0.21%)is_vmalloc_addr (42,225,682 samples, 0.24%)sock_write_iter (8,434,444,347 samples, 47.12%)sock_write_iterrw_verify_area (1,499,805,759 samples, 8.38%)rw_v..core_sys_select (98,377,773 samples, 0.55%)Nwrite (188,699,936 samples, 1.05%)skb_page_frag_refill (88,229,095 samples, 0.49%)syscall_trace_enter (40,692,261 samples, 0.23%)tmr_timeout (34,958,800 samples, 0.20%)Maximum (Baseline, Target) (17,901,751,664 samples, 100%)iperf_run_client (41,296,645 samples, 0.23%)ktime_get_ts64 (70,420,824 samples, 0.39%)iperf_send (331,658,282 samples, 1.85%)iperf_tcp_send (249,101,714 samples, 1.39%)_copy_from_user (87,596,396 samples, 0.49%)do_syscall_64 (1,986,731,293 samples, 11.10%)do_sysc..tcp_push (77,015,366 samples, 0.43%)__GI___libc_write (14,209,530,540 samples, 79.38%)__GI___libc_writevfs_write (10,741,241,549 samples, 60.00%)vfs_writetcp_write_xmit (1,176,558,774 samples, 6.57%)tcp..ipv4_mtu (122,990,045 samples, 0.69%)tcp_tso_segs (146,737,574 samples, 0.82%)skb_do_copy_data_nocache (1,425,528,042 samples, 7.96%)skb_..entry_SYSCALL_64 (1,176,259,810 samples, 6.57%)ent..__select (133,950,536 samples, 0.75%)_raw_spin_lock_irqsave (78,848,012 samples, 0.44%)ktime_get_coarse_real_ts64 (67,285,905 samples, 0.38%)selinux_file_permission (359,690,116 samples, 2.01%)do_syscall_64 (12,319,191,482 samples, 68.82%)do_syscall_64add_wait_queue (120,240,039 samples, 0.67%)_copy_from_user (170,522,592 samples, 0.95%)poll_freewait (171,434,286 samples, 0.96%)__x64_sys_pselect6 (103,164,783 samples, 0.58%)do_pselect.constprop.0 (1,813,716,009 samples, 10.13%)do_pse.._raw_spin_lock_bh (294,981,675 samples, 1.65%)tcp_sendmsg_locked (4,583,609,222 samples, 25.60%)tcp_sendmsg_lockediperf_check_throttle (141,088,006 samples, 0.79%)check_heap_object (438,200,632 samples, 2.45%)poll_select_finish (178,049,616 samples, 0.99%)syscall_trace_enter (340,784,387 samples, 1.90%)inet_send_prepare (40,338,451 samples, 0.23%)tcp_rate_check_app_limited (36,341,294 samples, 0.20%)__local_bh_enable_ip (190,001,362 samples, 1.06%)do_pselect.constprop.0 (103,164,783 samples, 0.58%)[unknown] (17,764,336,263 samples, 99.23%)[unknown]entry_SYSCALL_64_after_hwframe (2,002,336,455 samples, 11.19%)entry_S..lock_sock_nested (388,772,478 samples, 2.17%)__rcu_read_unlock (79,594,701 samples, 0.44%)__check_object_size.part.0 (50,506,799 samples, 0.28%)file_has_perm (811,214,121 samples, 4.53%)f..__x64_sys_write (34,586,660 samples, 0.19%)do_syscall_64 (111,334,003 samples, 0.62%)_raw_spin_lock_bh (164,486,243 samples, 0.92%)__tcp_push_pending_frames (1,222,505,002 samples, 6.83%)__t..entry_SYSCALL_64_after_hwframe (12,433,232,438 samples, 69.45%)entry_SYSCALL_64_after_hwframedo_select (55,056,262 samples, 0.31%)inode_has_perm (37,684,015 samples, 0.21%)sock_poll (31,389,968 samples, 0.18%)avc_has_perm_noaudit (410,212,722 samples, 2.29%)get_timespec64 (110,435,214 samples, 0.62%)__pollwait (82,420,927 samples, 0.46%)entry_SYSCALL_64_after_hwframe (114,969,647 samples, 0.64%)read_tsc (50,110,297 samples, 0.28%)do_select (851,591,661 samples, 4.76%)d..__fdget_pos (168,762,463 samples, 0.94%)read_tsc (365,384,590 samples, 2.04%)ktime_get (506,687,014 samples, 2.83%)ksys_write (11,066,663,324 samples, 61.82%)ksys_write__x64_sys_pselect6 (1,821,772,509 samples, 10.18%)__x64_..sk_page_frag_refill (128,893,189 samples, 0.72%)_copy_from_iter (693,063,513 samples, 3.87%)_..release_sock (322,532,754 samples, 1.80%)avc_has_perm (561,410,869 samples, 3.14%)tcp_sendmsg (5,602,830,340 samples, 31.30%)tcp_sendmsg
Target Flamegraph Reset ZoomSearchic iperf_send (291,171,743 samples, 1.63%)is_vmalloc_addr (39,655,423 samples, 0.22%)__rcu_read_unlock (45,320,992 samples, 0.25%)__memset (48,173,545 samples, 0.27%)sock_poll (323,324,795 samples, 1.81%)remove_wait_queue (102,932,141 samples, 0.57%)audit_reset_context (64,485,777 samples, 0.36%)file_has_perm (688,909,694 samples, 3.85%)f..skb_page_frag_refill (80,901,107 samples, 0.45%)iperf_check_throttle (124,893,123 samples, 0.70%)ktime_get (548,848,590 samples, 3.07%)ktime_get_ts64 (74,787,260 samples, 0.42%)vfs_write (9,333,593,384 samples, 52.14%)vfs_write__check_object_size.part.0 (499,169,968 samples, 2.79%)rw_verify_area (1,268,930,884 samples, 7.09%)rw_..do_select (58,058,315 samples, 0.32%)inode_has_perm (35,517,895 samples, 0.20%)check_heap_object (374,038,644 samples, 2.09%)syscall_exit_work (43,174,976 samples, 0.24%)_raw_spin_lock_irqsave (97,813,604 samples, 0.55%)read_tsc (41,463,613 samples, 0.23%)__x64_sys_pselect6 (1,875,355,970 samples, 10.48%)__x64_..iperf_run_client (39,935,934 samples, 0.22%)fput (48,283,728 samples, 0.27%)entry_SYSCALL_64_after_hwframe (130,124,907 samples, 0.73%)__audit_syscall_entry (225,278,075 samples, 1.26%)[unknown] (16,679,525,839 samples, 93.17%)[unknown]_raw_spin_unlock_bh (58,935,840 samples, 0.33%)Maximum (Baseline, Target) (17,901,751,664 samples, 100%)_raw_spin_lock_irqsave (77,418,918 samples, 0.43%)sk_page_frag_refill (115,904,000 samples, 0.65%)__x64_sys_write (34,775,409 samples, 0.19%)tcp_write_xmit (1,259,412,836 samples, 7.04%)tcp..tcp_wmem_schedule (43,661,568 samples, 0.24%)syscall_trace_enter (50,393,882 samples, 0.28%)entry_SYSCALL_64 (1,214,971,609 samples, 6.79%)ent..tcp_established_options (134,650,084 samples, 0.75%)write@plt (36,645,557 samples, 0.20%)read_tsc (417,967,235 samples, 2.33%)read_tsc (53,994,659 samples, 0.30%)syscall_exit_work (239,187,782 samples, 1.34%)inet_sendmsg (130,252,394 samples, 0.73%)do_select (884,645,250 samples, 4.94%)do..avc_has_perm_noaudit (322,562,687 samples, 1.80%)syscall_exit_to_user_mode (621,233,760 samples, 3.47%)do_syscall_64 (11,057,979,390 samples, 61.77%)do_syscall_64do_syscall_64 (130,124,907 samples, 0.73%)tcp_rate_check_app_limited (42,936,262 samples, 0.24%)sock_write_iter (7,497,598,153 samples, 41.88%)sock_write_itersyscall_exit_to_user_mode (112,986,166 samples, 0.63%)__local_bh_enable_ip (225,286,136 samples, 1.26%)selinux_file_permission (331,650,939 samples, 1.85%)__virt_addr_valid (135,114,056 samples, 0.75%)clock_gettime@@GLIBC_2.17 (154,571,547 samples, 0.86%)_copy_from_iter (777,645,572 samples, 4.34%)_..syscall_trace_enter (421,009,834 samples, 2.35%)avc_lookup (93,307,985 samples, 0.52%)security_socket_sendmsg (1,290,803,555 samples, 7.21%)sec.._copy_to_user (74,854,861 samples, 0.42%)select_estimate_accuracy (123,773,725 samples, 0.69%)core_sys_select (117,735,725 samples, 0.66%)put_timespec64 (103,406,358 samples, 0.58%)avc_lookup (112,345,092 samples, 0.63%)core_sys_select (1,426,139,458 samples, 7.97%)core.._raw_spin_lock_bh (328,016,311 samples, 1.83%)__audit_syscall_exit (71,868,399 samples, 0.40%)tcp_tso_segs (155,158,132 samples, 0.87%)avc_has_perm_noaudit (354,786,631 samples, 1.98%)__select (2,317,828,979 samples, 12.95%)__selectinet_send_prepare (45,740,460 samples, 0.26%)ksys_write (9,661,686,425 samples, 53.97%)ksys_write_copy_from_user (90,305,502 samples, 0.50%)tcp_release_cb (41,687,333 samples, 0.23%)tcp_push (84,334,614 samples, 0.47%)poll_freewait (176,836,244 samples, 0.99%)__pollwait (84,735,353 samples, 0.47%)ktime_get_coarse_real_ts64 (51,575,911 samples, 0.29%)__GI___libc_write (13,029,437,346 samples, 72.78%)__GI___libc_writepoll_select_finish (219,431,114 samples, 1.23%)iperf_tcp_send (236,997,758 samples, 1.32%)avc_has_perm (438,822,989 samples, 2.45%)_copy_from_user (162,651,131 samples, 0.91%)Nwrite (220,932,098 samples, 1.23%)entry_SYSCALL_64_after_hwframe (2,090,238,776 samples, 11.68%)entry_S..get_timespec64 (114,541,020 samples, 0.64%)do_pselect.constprop.0 (118,753,531 samples, 0.66%)iperf_time_now (34,632,016 samples, 0.19%)lock_sock_nested (431,894,912 samples, 2.41%)tcp_sendmsg (5,670,645,020 samples, 31.68%)tcp_sendmsgavc_has_perm (495,648,733 samples, 2.77%)tcp_sendmsg_locked (4,539,243,183 samples, 25.36%)tcp_sendmsg_lockedktime_get_ts64 (90,285,418 samples, 0.50%)entry_SYSCALL_64_safe_stack (124,439,030 samples, 0.70%)_raw_spin_lock_bh (190,004,108 samples, 1.06%)add_wait_queue (116,595,015 samples, 0.65%)do_pselect.constprop.0 (1,870,594,785 samples, 10.45%)do_pse..do_syscall_64 (2,075,131,875 samples, 11.59%)do_sysc..tmr_timeout (31,896,584 samples, 0.18%)syscall_return_via_sysret (152,492,270 samples, 0.85%)__fdget_pos (188,607,510 samples, 1.05%)entry_SYSCALL_64 (104,957,902 samples, 0.59%)read_tsc (62,189,263 samples, 0.35%)__check_object_size.part.0 (54,477,847 samples, 0.30%)__rcu_read_lock (31,867,283 samples, 0.18%)syscall_exit_to_user_mode_prepare (37,652,171 samples, 0.21%)release_sock (322,466,053 samples, 1.80%)[[vdso]] (137,965,083 samples, 0.77%)__fdelt_warn (34,722,755 samples, 0.19%)ktime_get_ts64 (61,134,912 samples, 0.34%)x64_sys_call (73,057,133 samples, 0.41%)entry_SYSCALL_64_after_hwframe (11,205,259,336 samples, 62.59%)entry_SYSCALL_64_after_hwframe__rcu_read_unlock (42,316,642 samples, 0.24%)ipv4_mtu (86,805,353 samples, 0.48%)skb_do_copy_data_nocache (1,364,373,701 samples, 7.62%)skb_..tcp_current_mss (625,649,672 samples, 3.49%)security_file_permission (1,185,662,830 samples, 6.62%)sec..tcp_send_mss (758,561,254 samples, 4.24%)t..__tcp_push_pending_frames (1,313,427,841 samples, 7.34%)__tc..tcp_poll (302,143,750 samples, 1.69%)__select (146,480,911 samples, 0.82%)__x64_sys_pselect6 (118,753,531 samples, 0.66%)inode_security (126,245,907 samples, 0.71%)selinux_socket_sendmsg (1,108,451,304 samples, 6.19%)sel..