Loading...

Type: Bug
Resolution: Won't Do
Priority: Normal
Fix Version/s: rhel-9.7
Affects Version/s: rhel-9.2.0.z, rhel-9.4.z, rhel-9.6.z
Component/s: irqbalance
Labels:
- Telco
- Telco:RAN
- rn-yml

Regression:
No
Severity:
Critical
Keywords:

ZStream
sprint_count:
1

AssignedTeam:
rhel-kernel-debug
Sub-System Group:

ssg_core_kernel

Story Points:
0
Blocked:
False
Ready:
False
Product Documentation Required:
Yes
Sprint:
CK Parent Issues In Progress
Release Blocker:
Approved Blocker

Preliminary Testing:
None
Test Coverage:
None

Release Note Type:
Bug Fix
Release Note Text:

Hide
.`irqbalance` service buffer overflow on aarch64 systems

Previously, the `irqbalance` service could crash due to a buffer overflow when running on specific aarch64 machines. As a consequence, latency-sensitive workloads might have experienced performance degradation because interrupts were not appropriately distributed across CPUs. With this update, the buffer overflow issue in the `irqbalance` service has been fixed.

As a result, the `irqbalance` service runs reliably, and interrupts are distributed as expected, improving performance for latency-sensitive workloads.

Show
.`irqbalance` service buffer overflow on aarch64 systems Previously, the `irqbalance` service could crash due to a buffer overflow when running on specific aarch64 machines. As a consequence, latency-sensitive workloads might have experienced performance degradation because interrupts were not appropriately distributed across CPUs. With this update, the buffer overflow issue in the `irqbalance` service has been fixed. As a result, the `irqbalance` service runs reliably, and interrupts are distributed as expected, improving performance for latency-sensitive workloads.
Release Note Status:
Done
ProdDocsReview-CCS:
Required
ProdDocsReview-Dev:
Unspecified
ProdDocsReview-QE:
Required

Experience:
Architecture:

aarch64
RH Private Keywords:

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Planning:
None

What were you trying to do that didn't work?

Deploy a Single Node OpenShift with Telco DU configuration on a aarch64 machine and check for any potential failed services.

What is the impact of this issue to you?

At this point it's only the failed systemd service. Subsequent testing may reveal other functional impact.

Please provide the package NVR for which the bug is seen:

5.14.0-570.12.1.el9_6.aarch64+64k

irqbalance-1.9.4-2.el9.aarch64

Deployment was done with 4.19.0-ec.5-multi release payload

How reproducible is this bug?:

100%

Steps to reproduce

Deploy a Single Node OpenShift with Telco DU configuration on aarch64 machine
SSH to the machine and check `systemctl status irqbalance.service`

Expected results

Service is not in failed state.

Actual results

core dumps available at https://drive.google.com/drive/folders/11pnWXULI05Mwf6nQX4jusN4pWE-92gCt

Apr 29 15:15:07 cnfdg46.ptp.eng.rdu2.dc.redhat.com systemd[1]: Started irqbalance daemon.
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQBALANCE_BANNED_CPUS is discarded, Use IRQBALANCE_BANNED_CPULIST instead
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 3578151    3565414    3550822       1003        959       1431       1085        902        933       1087        900       1018        900        912       1118   >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: *** buffer overflow detected ***: terminated
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com irqbalance[27920]: IRQ 0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         >
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com systemd-coredump[27958]: [🡕] Process 27920 (irqbalance) of user 0 dumped core.
                                                                            
                                                                            Stack trace of thread 27920:
                                                                            #0  0x0000ffff8c952568 __pthread_kill_implementation (libc.so.6 + 0x82568)
                                                                            #1  0x0000ffff8c90aa7c raise (libc.so.6 + 0x3aa7c)
                                                                            #2  0x0000ffff8c8f6fb4 abort (libc.so.6 + 0x26fb4)
                                                                            #3  0x0000ffff8c9467d8 __libc_message (libc.so.6 + 0x767d8)
                                                                            #4  0x0000ffff8c9ca7b8 __fortify_fail (libc.so.6 + 0xfa7b8)
                                                                            #5  0x0000ffff8c9c8f04 __chk_fail (libc.so.6 + 0xf8f04)
                                                                            #6  0x0000ffff8c9c873c __strcat_chk (libc.so.6 + 0xf873c)
                                                                            #7  0x0000aaaacaa16d18 check_platform_device (irqbalance + 0x6d18)
                                                                            #8  0x0000aaaacaa17238 init_irq_class_and_type (irqbalance + 0x7238)
                                                                            #9  0x0000aaaacaa19ec8 build_object_tree (irqbalance + 0x9ec8)
                                                                            #10 0x0000aaaacaa12a18 main (irqbalance + 0x2a18)
                                                                            #11 0x0000ffff8c8f7280 __libc_start_call_main (libc.so.6 + 0x27280)
                                                                            #12 0x0000ffff8c8f7358 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27358)
                                                                            #13 0x0000aaaacaa13430 _start (irqbalance + 0x3430)
                                                                            ELF object binary architecture: AARCH64
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com systemd[1]: irqbalance.service: Main process exited, code=dumped, status=6/ABRT
Apr 29 15:15:08 cnfdg46.ptp.eng.rdu2.dc.redhat.com systemd[1]: irqbalance.service: Failed with result 'core-dump'.

Details

Description

What were you trying to do that didn't work?

What is the impact of this issue to you?

Please provide the package NVR for which the bug is seen:

How reproducible is this bug?:

Steps to reproduce

Expected results

Actual results

Attachments

Easy Agile Planning Poker

Activity

People

Dates