[OCPBUGS-49845] [oauth-apiserver] Etcd client can unsafely retry timeouts on mutating requests

Type: Bug
Resolution: Unresolved
Priority: Undefined
Fix Version/s: None
Affects Version/s: 4.18.0, 4.19.0
Component/s: oauth-apiserver
Labels:
- rits-work

Regression:
Yes
Release Blocker:
Rejected
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Target Version:

4.19.0
Target Backport Versions:

4.18.0

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Description of problem:

Our carry patch intended to retry retriable requests that fail due to leader change will retry any etcd error with code "Unavailable": https://github.com/openshift/kubernetes/blob/4b2db1ec33faa3ffc305e5ffa7376908cc955370/staging/src/k8s.io/apiserver/pkg/storage/etcd3/etcd3retry/retry_etcdclient.go#L135-L145, but this includes reasons like "timeout" and does not distinguish between writes and reads. So a "timeout" error on a writing request might be retried even though a "timeout" observed by a client does not indicate that the effect of the write has not been persisted.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

    1.
    2.
    3.

Actual results:

Expected results:

Additional info:

clones

OCPBUGS-48694 Etcd client can unsafely retry timeouts on mutating requests

Verified

is cloned by

OCPBUGS-49847 [release-4.18] [oauth-apiserver] Etcd client can unsafely retry timeouts on mutating requests

Closed

is depended on by

OCPBUGS-49847 [release-4.18] [oauth-apiserver] Etcd client can unsafely retry timeouts on mutating requests

Closed

links to

openshift/oauth-apiserver#128: OCPBUGS-49845: Prevent storage layer retries for methods with side effects.

RHEA-2024:11038 OpenShift Container Platform 4.19.z bug fix update

Ke Wang added a comment - 2025/02/10 4:32 AM - edited

Checked logs of CI job periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-serial, we found the following logs,

2025-02-09 13:36:45 I0209 13:36:45.047661       1 retry_etcdclient.go:203] etcd retry - counter: 1, lastErrLabel: LeaderChanged lastError: etcdserver: leader changed, error: <nil>
Fields
app	openshift-oauth-apiserver
container	oauth-apiserver
detected_level	error
host	ci-op-9hz3qq32-6593d-wbdpc-master-1
invoker	openshift-internal-ci/periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-serial/1888565704416825344
namespace	openshift-oauth-apiserver
pod	apiserver-7874d654d8-qjnh2
type	pod

When oauth-apiserver clients retried, shows us clear reason for errors, not only etcd error with code "Unavailable". worked as expected, moving the bug VERIFIED.

Ke Wang added a comment - 2025/02/10 4:32 AM - edited Checked logs of CI job periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-serial , we found the following logs, 2025-02-09 13:36:45 I0209 13:36:45.047661 1 retry_etcdclient.go:203] etcd retry - counter: 1, lastErrLabel: LeaderChanged lastError: etcdserver: leader changed, error: <nil> Fields app openshift-oauth-apiserver container oauth-apiserver detected_level error host ci-op-9hz3qq32-6593d-wbdpc-master-1 invoker openshift-internal-ci/periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-serial/1888565704416825344 namespace openshift-oauth-apiserver pod apiserver-7874d654d8-qjnh2 type pod When oauth-apiserver clients retried, shows us clear reason for errors, not only etcd error with code "Unavailable". worked as expected, moving the bug VERIFIED.

Ke Wang added a comment - 2025/02/06 4:59 AM

The associated PR has been included starting from 4.18.0-0.nightly-2025-02-06-003812

Ke Wang added a comment - 2025/02/06 4:59 AM The associated PR has been included starting from 4.18.0-0.nightly-2025-02-06-003812

OpenShift Jira Bot added a comment - 2025/02/05 7:32 PM

Hi bluddy,

Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

OpenShift Jira Bot added a comment - 2025/02/05 7:32 PM Hi bluddy , Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

Collapse comment: Ke Wang added a comment - 2025/02/10 4:32 AM, Edited by Ke Wang - 2025/02/10 4:32 AM

Expand comment: Ke Wang added a comment - 2025/02/10 4:32 AM, Edited by Ke Wang - 2025/02/10 4:32 AM

Collapse comment: Ke Wang added a comment - 2025/02/06 4:59 AM

Expand comment: Ke Wang added a comment - 2025/02/06 4:59 AM

Collapse comment: OpenShift Jira Bot added a comment - 2025/02/05 7:32 PM

Expand comment: OpenShift Jira Bot added a comment - 2025/02/05 7:32 PM

People

Dates