Loading...

Details

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: rhel-9.5
Affects Version/s: rhel-9.2.0
Component/s: kernel-rt / Core-Kernel
Labels:

Fixed in Build:
kernel-5.14.0-430.el9
Severity:
Normal

Pool Team:

sst_kernel_rts
Sub-System Group:

ssg_core_kernel

Dev Target Milestone:
2
Internal Target Milestone:
6
Story Points:
5
Status Summary:

Hide

10/30: Yellow. No fix identified at this point, unable to forecast.

10/02: Still investigating. R ed for Oct 5th

Show
10/30: Yellow. No fix identified at this point, unable to forecast. 10/02: Still investigating. R ed for Oct 5th
Blocked:
False
Blocked Reason:

Hide

None

Show
None

Preliminary Testing:
Pass
Testable Builds:

Hide
The following Merge Request has pipeline job artifacts available:

Title: genirq: Wake interrupt threads immediately when changing affinity
MR: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3771
MR Pipeline: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/pipelines/1182626034

The Repo URLs are *not* accessible from a web browser! They only function as a dnf or yum baseurl.

Artifacts expire six weeks after creation. If artifacts are needed after that time please rerun the pipeline by visiting the MR's 'Pipelines' tab and clicking the 'Run pipeline' button.

----------------------------------------

Downstream Pipeline Name: RHEL_COMPAT (c9s_rhel9_compat_merge_request)
Downstream Pipeline: https://gitlab.com/redhat/red-hat-ci-tools/kernel/cki-internal-pipelines/cki-internal-contributors/-/pipelines/1182626062
Repo URL: https://s3.upshift.redhat.com/DH-PROD-CKI/internal/1182626062/$basearch/5.14.0-424.3771_1182626034.el9.$basearch
Debug Repo URL: https://s3.upshift.redhat.com/DH-PROD-CKI/internal/1182626062/$basearch-debug/5.14.0-424.3771_1182626034.el9.$basearch

5.14.0-424.3771_1182626034.el9.s390x:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1182626062/publish_s390x/6207946664/artifacts/repo/5.14.0-424.3771_1182626034.el9.s390x/

5.14.0-424.3771_1182626034.el9.aarch64:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1182626062/publish_aarch64/6207946651/artifacts/repo/5.14.0-424.3771_1182626034.el9.aarch64/

5.14.0-424.3771_1182626034.el9.ppc64le:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1182626062/publish_ppc64le/6207946640/artifacts/repo/5.14.0-424.3771_1182626034.el9.ppc64le/

5.14.0-424.3771_1182626034.el9.x86_64-debug:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1182626062/publish_x86_64_debug/6207946630/artifacts/repo/5.14.0-424.3771_1182626034.el9.x86_64/

5.14.0-424.3771_1182626034.el9.x86_64:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1182626062/publish_x86_64/6207946622/artifacts/repo/5.14.0-424.3771_1182626034.el9.x86_64/

----------------------------------------

Downstream Pipeline Name: REALTIME (c9s_rt_merge_request)
Downstream Pipeline: https://gitlab.com/redhat/red-hat-ci-tools/kernel/cki-internal-pipelines/cki-trusted-contributors/-/pipelines/1182626065

5.14.0-424.3771_1182626034.el9.x86_64-debug:
Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1182626065/publish_x86_64_debug/6207946658/artifacts/repo/5.14.0-424.3771_1182626034.el9.x86_64/
Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1182626065/publish_x86_64_debug/6207946658/artifacts/repo/5.14.0-424.3771_1182626034.el9.x86_64/

5.14.0-424.3771_1182626034.el9.x86_64:
Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1182626065/publish_x86_64/6207946647/artifacts/repo/5.14.0-424.3771_1182626034.el9.x86_64/
Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1182626065/publish_x86_64/6207946647/artifacts/repo/5.14.0-424.3771_1182626034.el9.x86_64/

----------------------------------------

Downstream Pipeline Name: 64K (c9s_64k_merge_request)
Downstream Pipeline: https://gitlab.com/redhat/red-hat-ci-tools/kernel/cki-internal-pipelines/cki-trusted-contributors/-/pipelines/1182626080

5.14.0-424.3771_1182626034.el9.aarch64:
Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1182626080/publish_aarch64/6207946800/artifacts/repo/5.14.0-424.3771_1182626034.el9.aarch64/
Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1182626080/publish_aarch64/6207946800/artifacts/repo/5.14.0-424.3771_1182626034.el9.aarch64/

----------------------------------------

Updated 2024-02-20 01:31 UTC by buglinker:
- KWF FAQ: https://red.ht/kernel_workflow_doc
- Slack #team-kernel-workflow: https://redhat-internal.slack.com/archives/C04LRUPMJQ5
- Source: https://gitlab.com/cki-project/kernel-workflow/-/blob/main/webhook/buglinker.py
- Documentation: https://gitlab.com/cki-project/kernel-workflow/-/blob/main/docs/README.buglinker.md
- Report an issue: https://gitlab.com/cki-project/kernel-workflow/-/issues/new?issue%5Btitle%5D=buglinker%20webhook%20issue

Show
The following Merge Request has pipeline job artifacts available: Title: genirq: Wake interrupt threads immediately when changing affinity MR: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3771 MR Pipeline: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/pipelines/1182626034 The Repo URLs are *not* accessible from a web browser! They only function as a dnf or yum baseurl. Artifacts expire six weeks after creation. If artifacts are needed after that time please rerun the pipeline by visiting the MR's 'Pipelines' tab and clicking the 'Run pipeline' button. ---------------------------------------- Downstream Pipeline Name: RHEL_COMPAT (c9s_rhel9_compat_merge_request) Downstream Pipeline: https://gitlab.com/redhat/red-hat-ci-tools/kernel/cki-internal-pipelines/cki-internal-contributors/-/pipelines/1182626062 Repo URL: https://s3.upshift.redhat.com/DH-PROD-CKI/internal/1182626062/$basearch/5.14.0-424.3771_1182626034.el9.$basearch Debug Repo URL: https://s3.upshift.redhat.com/DH-PROD-CKI/internal/1182626062/$basearch-debug/5.14.0-424.3771_1182626034.el9.$basearch 5.14.0-424.3771_1182626034.el9.s390x: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1182626062/publish_s390x/6207946664/artifacts/repo/5.14.0-424.3771_1182626034.el9.s390x/ 5.14.0-424.3771_1182626034.el9.aarch64: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1182626062/publish_aarch64/6207946651/artifacts/repo/5.14.0-424.3771_1182626034.el9.aarch64/ 5.14.0-424.3771_1182626034.el9.ppc64le: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1182626062/publish_ppc64le/6207946640/artifacts/repo/5.14.0-424.3771_1182626034.el9.ppc64le/ 5.14.0-424.3771_1182626034.el9.x86_64-debug: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1182626062/publish_x86_64_debug/6207946630/artifacts/repo/5.14.0-424.3771_1182626034.el9.x86_64/ 5.14.0-424.3771_1182626034.el9.x86_64: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1182626062/publish_x86_64/6207946622/artifacts/repo/5.14.0-424.3771_1182626034.el9.x86_64/ ---------------------------------------- Downstream Pipeline Name: REALTIME (c9s_rt_merge_request) Downstream Pipeline: https://gitlab.com/redhat/red-hat-ci-tools/kernel/cki-internal-pipelines/cki-trusted-contributors/-/pipelines/1182626065 5.14.0-424.3771_1182626034.el9.x86_64-debug: Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1182626065/publish_x86_64_debug/6207946658/artifacts/repo/5.14.0-424.3771_1182626034.el9.x86_64/ Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1182626065/publish_x86_64_debug/6207946658/artifacts/repo/5.14.0-424.3771_1182626034.el9.x86_64/ 5.14.0-424.3771_1182626034.el9.x86_64: Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1182626065/publish_x86_64/6207946647/artifacts/repo/5.14.0-424.3771_1182626034.el9.x86_64/ Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1182626065/publish_x86_64/6207946647/artifacts/repo/5.14.0-424.3771_1182626034.el9.x86_64/ ---------------------------------------- Downstream Pipeline Name: 64K (c9s_64k_merge_request) Downstream Pipeline: https://gitlab.com/redhat/red-hat-ci-tools/kernel/cki-internal-pipelines/cki-trusted-contributors/-/pipelines/1182626080 5.14.0-424.3771_1182626034.el9.aarch64: Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1182626080/publish_aarch64/6207946800/artifacts/repo/5.14.0-424.3771_1182626034.el9.aarch64/ Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1182626080/publish_aarch64/6207946800/artifacts/repo/5.14.0-424.3771_1182626034.el9.aarch64/ ---------------------------------------- Updated 2024-02-20 01:31 UTC by buglinker: - KWF FAQ: https://red.ht/kernel_workflow_doc - Slack #team-kernel-workflow: https://redhat-internal.slack.com/archives/C04LRUPMJQ5 - Source: https://gitlab.com/cki-project/kernel-workflow/-/blob/main/webhook/buglinker.py - Documentation: https://gitlab.com/cki-project/kernel-workflow/-/blob/main/docs/README.buglinker.md - Report an issue: https://gitlab.com/cki-project/kernel-workflow/-/issues/new?issue%5Btitle%5D=buglinker%20webhook%20issue
Errata Link:
https://errata.engineering.redhat.com/advisory/128795

Release Note Type:
If docs needed, set a value

Architecture:

x86_64
Bugzilla Bug:
RHBZ: 2203292

SFDC Cases Counter:
SFDC Cases Links:

Description

Description of problem:

Om preempt-rt, after updating the SMP affinity of a network interrupt, the corresponding IRQ thread does not get updated immediately. This will be done when the next interrupt received and the thread is migrated to the proper core.

<idle>-0 [064] d...2.. 504437.282162: sched_switch: prev_comm=swapper/64 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=oslat next_pid=2563104 next_prio=98
<...>-2563104 [064] dn.h... 504439.963379: reschedule_entry: vector=253
<...>-2563104 [064] dn.h... 504439.963380: reschedule_exit: vector=253
<...>-2563104 [064] d...2.. 504439.963382: sched_switch: prev_comm=oslat prev_pid=2563104 prev_prio=98 prev_state=R ==> next_comm=irq/316-ice-eno next_pid=5596 next_prio=49
<...>-5596 [064] dn..4.. 504439.963388: sched_wakeup: comm=migration/64 pid=607 prio=0 target_cpu=064
<...>-5596 [064] d...2.. 504439.963389: sched_switch: prev_comm=irq/316-ice-eno prev_pid=5596 prev_prio=49 prev_state=R+ ==> next_comm=migration/64 next_pid=607 next_prio=0
<...>-607 [064] d...2.. 504439.963395: sched_switch: prev_comm=migration/64 prev_pid=607 prev_prio=0 prev_state=S ==> next_comm=oslat next_pid=2563104 next_prio=98
<...>-2563104 [064] ....... 504439.963410: tracing_mark_write: oslat: Trace threshold (20 us) triggered with 20 us!

This is problematic if the interrupt was originally affined to an isolated core as the migration will cause a latency spike. In a real deployment this is less of an issue as this as the situation heals itself after 1 interrupt. It is problematic for benchmark testing with oslat etc. as this can cause spikes that potentially exceed the threshold.

Ideally the thread would get reaffined when the SMP affinity for the IRQ is updated

Version-Release number of selected component (if applicable):
9.2 but this will exist in earlier versions as well

How reproducible:
100% if an interrupt thread was originally pinned to an isolated core and an interrupt is received.

Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

Attachments

Issue Links

causes

OCPBUGS-32031 oslat 45us spike 1h run on 4.14.20

ASSIGNED

external trackers

Red Hat Issue Tracker RHELPLAN-157001

links to

Merge Request: genirq: Wake interrupt threads immediately when changing affinity

RHSA-2024:128795 kernel update

Interrupt thread not affined after interrupt reaffined

Details

Description

Attachments

Issue Links

Activity

People

Dates