Loading...

XML

Word

Printable

Type: Bug
Resolution: Won't Do
Priority: Major
Fix Version/s: None
Affects Version/s: 4.14
Component/s: Networking / router
Labels:

Severity:
Important
Regression:
No
Story Points:
5
Sprint:
Sprint 242, Sprint 243
sprint_count:
2
Release Blocker:
Rejected
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Release Note Text:

Hide
cause:
In OCP 4.14 HAProxy version is upgraded from 2.6.13.

consequence:
An increase in P99 latency for re-encrypt traffic is observed when the volume of ingress traffic puts the HAProxy component of the OpenShift IngressController under considerable load. This latency increase does not affect overall throughput, which remains consistent.

fix:
The bug did not fix the issue, but there are some tuning Recommendations to mitigate the issue:
The default OpenShift IngressController is configured with 4 HAProxy threads. If you experience elevated P99 latencies during high ingress traffic conditions, specifically with re-encrypt traffic, it's recommended to increase the number of HAProxy threads to reduce latency.

result:
An increase in P99 latency for re-encrypt traffic is observed when the volume of ingress traffic puts the HAProxy component of the OpenShift IngressController under considerable load. Increase the number of HAProxy threads can reduce the latency.

Show
cause: In OCP 4.14 HAProxy version is upgraded from 2.6.13. consequence: An increase in P99 latency for re-encrypt traffic is observed when the volume of ingress traffic puts the HAProxy component of the OpenShift IngressController under considerable load. This latency increase does not affect overall throughput, which remains consistent. fix: The bug did not fix the issue, but there are some tuning Recommendations to mitigate the issue: The default OpenShift IngressController is configured with 4 HAProxy threads. If you experience elevated P99 latencies during high ingress traffic conditions, specifically with re-encrypt traffic, it's recommended to increase the number of HAProxy threads to reduce latency. result: An increase in P99 latency for re-encrypt traffic is observed when the volume of ingress traffic puts the HAProxy component of the OpenShift IngressController under considerable load. Increase the number of HAProxy threads can reduce the latency.
Release Note Type:
Known Issue
Release Note Status:
In Progress

SFDC Cases Counter:
SFDC Cases Links:

Description of problem:

ingress-perf test shows haproxy 2.6.13 has a higher reencrypt latency than haproxy 2.2.24, deviation is about 100%. 
haproxy bump from 2.2.24 to 2.6.13 during OCP 4.14 release.
edge http and passthrough do not have this issue.
Rrevert OCP4.14 haproxy from 2.6.13 to 2.2.24 does not have this issue.
Revert OCP4.14 haproxy 2.6.13 with a change https://github.com/haproxy/haproxy/issues/1914, does not have this issue.

Version-Release number of selected component (if applicable):

4.14 after haproxy 2.6.1

How reproducible:

Everytime

Steps to Reproduce:

1. Install a cluster. AWS OVN, 3 master nodes with type "m5.8xlarge", 24 worker nodes with type m5.2xlarge, 3 infra nodes with type r5.2xlarge.
2. Run ingress-perf test https://github.com/cloud-bulldozer/ingress-perf, with configuration https://github.com/cloud-bulldozer/e2e-benchmarking/blob/master/workloads/ingress-perf/config/standard.yml. Set requestTimeout to >=5s. In this test ingress pods are moved to infra nodes.
3. Compare the RPS and Latency the results.

Actual results:

ingress-perf test shows haproxy 2.6.13 has a higher reencrypt latency than haproxy 2.2.24.

Andrew McDermott provided a image to revert haproxy 2.6.13 to haproxy 2.2.24. After using this image the issue is not seen. So this is a regression since haproxy 2.6.13.

Andrew McDermott provided a image quay.io/amcdermo/openshift-router-qiujieli-ocp414-haproxy26-sans-4c48edba4f45bb78f41af7d79d3c176710fe6a90 which reverts the patch identified in https://github.com/haproxy/haproxy/issues/1914. After using this image, the issue is not seen.

Test results are as below:

0902: 4.14.0-0.nightly-2023-09-02-132842
08-28: 4.14.0-0.nightly-2023-08-28-154013
revert1914: which reverts the patch identified in https://github.com/haproxy/haproxy/issues/1914
ec4: doesn't have ovn-ic
ec2: default haproxy 2.6.13
ec1: default haproxy 2.2.24

	0902	% to 4.13.9	0902 (revert1914)	% to 4.13.9	0828	% to 4.13.9	0528 (revert 2.2.24)	% to 4.13.9	ec4	% to 4.13.9	ec4 (revert1914)	% to 4.13.9	ec2 2.6.13	% to 4.13.9	ec2 (revert 2.2.24)	% to 4.13.9	ec1 2.2.24	% to 4.13.9	4.13.9
edge
RPS (K req/s)	75.33	6.49%	69.44	-1.84%	74.8	5.74%	73.72	4.21%	72.21	2.08%	65.49	-7.42%	75.06	6.11%	72.07	1.88%	72.67	2.73%	70.74
avg_latency(ms)	50.46	-0.98%	55.63	9.16%	56.95	11.75%	53.22	4.43%	53.16	4.32%	58.16	14.13%	49.02	-3.81%	53.68	5.34%	52.73	3.47%	50.96
http
RPS (K req/s)	105.78	-9.78%	106.17	-9.45%	103.83	-11.45%	105.33	-10.17%	115.75	-1.28%	103.45	-11.77%	113.51	-3.19%	117.88	0.54%	122.46	4.44%	117.25
avg_latency(ms)	42.47	32.02%	41.98	30.49%	45.39	41.09%	36.32	12.90%	31.36	-2.52%	35.42	10.10%	32.77	1.87%	41.73	29.72%	31.55	-1.93%	32.17
passthrough
RPS (K req/s)	191.4	-4.78%	183.82	-8.55%	184.74	-8.09%	188.75	-6.10%	202.8	0.89%	198.61	-1.19%	197.61	-1.69%	210.72	4.83%	212.61	5.77%	201.01
avg_latency(ms)	19.35	7.14%	19.61	8.58%	20.3	12.40%	21.01	16.33%	17.98	-0.44%	19.21	6.37%	18.9	4.65%	17.36	-3.88%	17.62	-2.44%	18.06
reencrypt
RPS (K req/s)	73.05	0.74%	65.97	-9.02%	72.19	-0.44%	70.73	-2.45%	67.72	-6.61%	65.3	-9.94%	71.44	-1.48%	68.69	-5.27%	70.06	-3.38%	72.51
avg_latency(ms)	104.29	107.79%	58.88	17.31%	97.52	94.30%	58.24	16.04%	96.52	92.31%	66.05	31.60%	108.11	115.40%	52.67	4.94%	53.69	6.97%	50.19

Expected results:

haproxy 2.6.13 should not have performance regression compared to haproxy 2.2.24

Additional info:

Assignee:: ANDREW MCDERMOTT

Reporter:: Qiujie Li

QA Contact:: Qiujie Li

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Created:: 2023/09/13 10:16 AM

Updated:: 2023/12/28 4:22 AM

Resolved:: 2023/10/05 2:45 PM

Details

Description

Attachments

Activity

People

Dates