-
Bug
-
Resolution: Can't Do
-
Major
-
None
-
4.15
-
Critical
-
No
-
False
-
-
Description of problem:
One of the goals of performance&scale testing IPsec (See presentation) is to show core-over-core scaling. That is to show how IPsec throughput would scale with more CPUs (reserved CPUs). A test case however shows that IPsec egress does NOT scale. More investigation points to the problem possibly related to the IPsec peer endpoint. In this case, the peer is a RHEL 9.3. When running IPsec, its sofirq uses mostly one CPU. This is not the case when running the same test w/o IPsec when a lot more CPUs are multi-processing the test. In other words, the RHEL is the bottleneck b/c its softIRQ pinned on one CPU when processing IPsec traffic.
Version-Release number of selected component (if applicable) expects to scale
R4.15 IPsec
How reproducible:
100%
Steps to Reproduce:
1. Config IPsec on a R4.15 SNO and an external RHEL according to the procedure https://github.com/jakobmoellerdev/north-south-ipsec-openshift-poc/tree/main/4.15
2. Start UPERF server on the RHEL
3. Create a UPERF POD on the SNO, and run UPERF client.
----------- Egress testcase with IPsec --------------------
On RHEL, start UPERF Server:
[root@d16-h09-000-r650 xml-files]# uperf -s -B 100.50.1.8 -P 41000
on Pod, start UPERF Client:
sh-5.1# remotehost=100.50.1.8 port=41001 duration=60 wsize=512 rsize=4096 nthreads=16 protocol=tcp uperf -v -R -m stream.xml -P 41000
On another RHEL terminal, watch CPUs:
[root@d16-h09-000-r650 ~]# mpstat -P ALL 2 |tail -n+3|awk '$NF<95 {print $0}
..
01:36:54 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
01:36:56 AM 37 1.00 0.00 4.48 0.00 0.00 0.00 0.00 0.00 0.00 94.53
01:36:56 AM 93 0.00 0.00 0.00 0.00 0.46 84.02 0.00 0.00 0.00 15.53 <== high soft on CPU 93
01:36:56 AM 95 1.00 0.00 4.50 0.00 0.50 0.00 0.00 0.00 0.00 94.00
01:36:56 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
01:36:58 AM 93 0.00 0.00 0.00 0.00 0.00 90.00 0.00 0.00 0.00 10.00 <== high soft
01:36:58 AM 95 1.00 0.00 4.50 0.00 0.00 0.00 0.00 0.00 0.00 94.50
01:36:58 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
01:37:00 AM 5 1.00 0.00 4.98 0.00 0.00 0.00 0.00 0.00 0.00 94.03
01:37:00 AM 93 0.00 0.00 0.00 0.00 0.00 91.83 0.00 0.00 0.00 8.17 <--high soft
01:37:00 AM 95 0.50 0.00 5.47 0.00 0.00 0.00 0.00 0.00 0.00 94.03
01:37:00 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
01:37:02 AM 45 1.00 0.00 4.50 0.00 0.00 0.00 0.00 0.00 0.00 94.50
01:37:02 AM 93 0.00 0.00 0.00 0.00 0.48 91.79 0.00 0.00 0.00 7.73 <-- high soft
...
-------- Egress Test case Without IPsec
On RHEL, start UPERF Server:
[root@d16-h09-000-r650 xml-files]# uperf -s -P 41000
On Pod, start UPERF Client:
sh-5.1# remotehost=100.50.1.8 port=41001 duration=60 wsize=512 rsize=4096 nthreads=16 protocol=tcp uperf -v -R -m stream.xml -P 41000
On another RHEL terminal, watch CPUs.
[root@d16-h09-000-r650 ~]# mpstat -P ALL 2 |tail -n+3|awk '$NF<95 {print $0}
...
01:45:51 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
01:45:53 AM 0 1.01 0.00 4.52 0.00 0.00 2.01 0.00 0.00 0.00 92.46
01:45:53 AM 2 1.50 0.00 9.00 0.00 0.00 0.00 0.00 0.00 0.00 89.50
01:45:53 AM 4 1.50 0.00 7.00 0.00 0.00 0.00 0.00 0.00 0.00 91.50
01:45:53 AM 6 1.00 0.00 5.50 0.00 0.00 0.00 0.00 0.00 0.00 93.50
01:45:53 AM 9 0.50 0.00 2.50 0.00 0.50 6.00 0.00 0.00 0.00 90.50
01:45:53 AM 13 3.98 0.00 23.38 0.00 0.00 0.50 0.00 0.00 0.00 72.14
01:45:53 AM 21 0.00 0.00 0.00 0.00 0.00 5.97 0.00 0.00 0.00 94.03
01:45:53 AM 23 4.52 0.00 23.62 0.00 0.00 0.00 0.00 0.00 0.00 71.86
01:45:53 AM 25 4.52 0.00 26.63 0.00 0.00 0.00 0.00 0.00 0.00 68.84
01:45:53 AM 29 0.00 0.00 0.50 0.00 0.50 6.97 0.00 0.00 0.00 92.04
01:45:53 AM 31 3.52 0.00 21.61 0.00 0.00 0.00 0.00 0.00 0.00 74.87
01:45:53 AM 33 0.00 0.00 0.00 0.00 0.00 9.55 0.00 0.00 0.00 90.45
01:45:53 AM 37 6.03 0.00 24.62 0.00 0.50 0.00 0.00 0.00 0.00 68.84
01:45:53 AM 45 1.52 0.00 6.06 0.00 0.00 0.00 0.00 0.00 0.00 92.42
01:45:53 AM 49 0.00 0.00 0.50 0.00 0.50 8.04 0.00 0.00 0.00 90.95
01:45:53 AM 51 2.51 0.00 14.07 0.00 0.50 0.50 0.00 0.00 0.00 82.41
01:45:53 AM 58 2.99 0.00 17.41 0.00 0.00 0.00 0.00 0.00 0.00 79.60
01:45:53 AM 63 1.49 0.00 3.98 0.00 0.00 0.50 0.00 0.00 0.00 94.03
01:45:53 AM 81 0.00 0.00 0.00 0.00 0.00 7.50 0.00 0.00 0.00 92.50
01:45:53 AM 83 0.00 0.00 0.50 0.00 0.00 8.00 0.00 0.00 0.00 91.50
01:45:53 AM 85 0.00 0.00 0.00 0.00 0.00 7.50 0.00 0.00 0.00 92.50
01:45:53 AM 89 5.50 0.00 27.00 0.00 0.50 0.00 0.00 0.00 0.00 67.00
01:45:53 AM 93 0.00 0.00 0.00 0.00 0.00 8.54 0.00 0.00 0.00 91.46
01:45:53 AM 97 4.57 0.00 22.84 0.00 0.00 0.00 0.00 0.00 0.00 72.59
01:45:53 AM 99 0.00 0.00 0.50 0.00 0.50 6.47 0.00 0.00 0.00 92.54
01:45:53 AM 101 4.06 0.00 22.84 0.00 0.00 0.51 0.00 0.00 0.00 72.59
01:45:53 AM 103 0.00 0.00 0.00 0.00 0.00 6.97 0.00 0.00 0.00 93.03
01:45:53 AM 105 4.52 0.00 21.61 0.00 0.00 0.00 0.00 0.00 0.00 73.87
01:45:53 AM 109 0.00 0.00 0.50 0.00 0.00 8.00 0.00 0.00 0.00 91.50
01:45:53 AM 111 3.48 0.00 18.41 0.00 0.50 0.00 0.00 0.00 0.00 77.61
--------
Actual results:
Expected results:
Additional info:
For tips on manual testing IPsec N/S, see https://github.com/HughNhan/IPsecTest For IPsec config, see https://github.com/jakobmoellerdev/north-south-ipsec-openshift-poc/tree/main/4.15