[OTA-362] CI: fail update suite if any ClusterOperator go Available=False

Type: Story
Resolution: Done
Priority: Major
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- groomed

Work Type:
BU Product Work
Story Points:
2
Blocked:
False
Ready:
False
Epic Link:
Ensure availability of all HA components during upgrades
Feature Link:
OCPSTRAT-835 - Improve upgrades - Reduce False Positives status from operators
Release Note Text:
Undefined
Market:

Sprint:
OTA 243, OTA 244, OTA 245

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

These are alarming conditions which may frighten customers, and we don't want to see them in our own, controlled, repeatable update CI. This example job had logs like:

Feb 18 21:11:25.799 E clusteroperator/openshift-apiserver changed Degraded to True: APIServerDeployment_UnavailablePod: APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()

And the job failed, but none of the failures were "something made openshift-apiserver mad enough to go Degraded".

blocks

OTA-980 Is the Failing=True status condition is a good indicator for admins?

To Do

is blocked by

OTA-701 Communicate available and degraded condition definition to teams

Closed

is depended on by

TRT-1576 CI: fail update suite if any ClusterOperator go Available=False outside of updates

Closed

is related to

OCPBUGS-20056 Single short-lived operand blip shouldn't cause authentication operator Available=False

OCPBUGS-23746 openshift-apiserver ClusterOperator should not blip Available=False on brief missing HTTP content-type

OCPBUGS-20062 kube-storage-version-migrator goes Available=False with reason=KubeStorageVersionMigrator_Deploying during updates

Verified

OCPBUGS-9108 openshift-tests-upgrade.[bz-Machine Config Operator] clusteroperator/machine-config should not change condition/Available

Closed

OCPBUGS-20061 control-plane-machine-set goes Available=False with UnavailableReplicas during updates

Closed

OCPBUGS-23744 operator-lifecycle-manager-packageserver ClusterOperator should not blip Available=False on 4.14 to 4.15 updates

Closed

OCPBUGS-24041 Console blips Available=False with RouteHealth_FailedGet and such

Closed

OCPBUGS-24228 machine-config ClusterOperator should not blip Available=False on brief missing HTTP content-type

Closed

OCPBUGS-32089 Authentication blips Available=False with WellKnown_NotReady

Closed

OCPBUGS-36462 control-plane-machine-set goes Available=False with UnavailableReplicas during etcd scale testing

Closed

TRT-1235 Work with Service Delivery On Reducing Problematic Alerts

Closed

TRT-1575 CI: fail update suite if any ClusterOperator go Degraded=True

Closed

OCPBUGS-825 Available=False with no reason

Closed

OCPBUGS-22364 ControllerCertificate struct validation failed during upgrade from 4.14 to 4.15

Closed

OCPBUGS-23745 monitoring ClusterOperator should not blip Available=False on quick etcd leader changes

Closed

OCPBUGS-35892 monitoring ClusterOperator should not blip Available=Unknown on client rate limiter

Closed

links to

openshift/origin#27231: OTA-362: pkg/monitortests/clusterversionoperator: Fatal unless Available=False in allow-list

origin#25920

(14 is related to, 2 links to)

Scott Dodson added a comment - 2024/02/14 7:47 PM

I've set all related bugs to priority Major and left comments indicating we'd like to have these addressed by 4.16.

Scott Dodson added a comment - 2024/02/14 7:47 PM I've set all related bugs to priority Major and left comments indicating we'd like to have these addressed by 4.16.

W. Trevor King added a comment - 2023/12/03 9:15 PM

origin#27231 landed

W. Trevor King added a comment - 2023/12/03 9:15 PM origin#27231 landed

Lalatendu Mohanty added a comment - 2023/11/20 10:01 PM - edited

rhn-engineering-dgoodwinThis is high priority for us. We want to make progress on this before 4.15 release. This is technical debt we should have addressed long back. I am happy to give it you if you want to take over this and we would help anyway we can. Please sync with trking about how we can pass it to you.

Lalatendu Mohanty added a comment - 2023/11/20 10:01 PM - edited rhn-engineering-dgoodwin This is high priority for us. We want to make progress on this before 4.15 release. This is technical debt we should have addressed long back. I am happy to give it you if you want to take over this and we would help anyway we can. Please sync with trking about how we can pass it to you.

Devan Goodwin added a comment - 2023/11/20 5:48 PM

lmohanty@redhat.com trking just wondering how this effort sits in priority, specific to https://github.com/openshift/origin/pull/27231. I'm working on correlating alerts SD struggles with against CI alerts that fire and this one is a top offender, the change looked like a really big win. If you'd like TRT to take over that PR and wrap it up we'd be happy to just say the word. If it's something you all would like to complete that's fine I'm just looking to see where things are at and if we can help.

Devan Goodwin added a comment - 2023/11/20 5:48 PM lmohanty@redhat.com trking just wondering how this effort sits in priority, specific to https://github.com/openshift/origin/pull/27231 . I'm working on correlating alerts SD struggles with against CI alerts that fire and this one is a top offender, the change looked like a really big win. If you'd like TRT to take over that PR and wrap it up we'd be happy to just say the word. If it's something you all would like to complete that's fine I'm just looking to see where things are at and if we can help.

Lalatendu Mohanty added a comment - 2022/06/27 6:44 PM - edited

Created card for operators going degraded during upgrade https://issues.redhat.com/browse/OTA-699

Changed the title of this Jira card to only cover available condition. Also we need a new card to cover the work to communicate available and degraded condition to teams.

Lalatendu Mohanty added a comment - 2022/06/27 6:44 PM - edited Created card for operators going degraded during upgrade https://issues.redhat.com/browse/OTA-699 Changed the title of this Jira card to only cover available condition. Also we need a new card to cover the work to communicate available and degraded condition to teams.

Lalatendu Mohanty added a comment - 2022/06/07 7:14 PM

Slack thread to understand what others (staff engineers, TRT) think about this card https://coreos.slack.com/archives/CEGKQ43CP/p1654625537501139

Lalatendu Mohanty added a comment - 2022/06/07 7:14 PM Slack thread to understand what others (staff engineers, TRT) think about this card https://coreos.slack.com/archives/CEGKQ43CP/p1654625537501139

Lalatendu Mohanty added a comment - 2022/06/07 5:17 PM

We should do it during beginning of 4.12 release cycle, so that teams can accommodate the extra work coming out of this.

Lalatendu Mohanty added a comment - 2022/06/07 5:17 PM We should do it during beginning of 4.12 release cycle, so that teams can accommodate the extra work coming out of this.

Lalatendu Mohanty added a comment - 2022/05/16 5:11 PM

Lets take this to pillar arch call and see what other teams has to say about this.

Lalatendu Mohanty added a comment - 2022/05/16 5:11 PM Lets take this to pillar arch call and see what other teams has to say about this.

W. Trevor King added a comment - 2021/02/23 4:00 PM - edited

Clayton floated origin#25920 for updates.

W. Trevor King added a comment - 2021/02/23 4:00 PM - edited Clayton floated origin#25920 for updates.

W. Trevor King added a comment - 2021/02/23 3:32 PM

Related, but not for update jobs: origin#25918.

W. Trevor King added a comment - 2021/02/23 3:32 PM Related, but not for update jobs: origin#25918 .

Assignee:: W. Trevor King

Reporter:: W. Trevor King

Votes:: 0 Vote for this issue

Watchers:: 9 Start watching this issue

Created:: 2021/02/18 5:21 PM

Updated:: 2024/09/05 12:16 AM

Resolved:: 2023/12/03 9:15 PM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

Collapse comment: Scott Dodson added a comment - 2024/02/14 7:47 PM

Expand comment: Scott Dodson added a comment - 2024/02/14 7:47 PM

Collapse comment: W. Trevor King added a comment - 2023/12/03 9:15 PM

Expand comment: W. Trevor King added a comment - 2023/12/03 9:15 PM

Collapse comment: Lalatendu Mohanty added a comment - 2023/11/20 10:01 PM, Edited by Lalatendu Mohanty - 2023/11/20 10:06 PM

Expand comment: Lalatendu Mohanty added a comment - 2023/11/20 10:01 PM, Edited by Lalatendu Mohanty - 2023/11/20 10:06 PM

Collapse comment: Devan Goodwin added a comment - 2023/11/20 5:48 PM

Expand comment: Devan Goodwin added a comment - 2023/11/20 5:48 PM

Collapse comment: Lalatendu Mohanty added a comment - 2022/06/27 6:44 PM, Edited by Lalatendu Mohanty - 2022/06/27 6:47 PM

Expand comment: Lalatendu Mohanty added a comment - 2022/06/27 6:44 PM, Edited by Lalatendu Mohanty - 2022/06/27 6:47 PM

Collapse comment: Lalatendu Mohanty added a comment - 2022/06/07 7:14 PM

Expand comment: Lalatendu Mohanty added a comment - 2022/06/07 7:14 PM

Collapse comment: Lalatendu Mohanty added a comment - 2022/06/07 5:17 PM

Expand comment: Lalatendu Mohanty added a comment - 2022/06/07 5:17 PM

Collapse comment: Lalatendu Mohanty added a comment - 2022/05/16 5:11 PM

Expand comment: Lalatendu Mohanty added a comment - 2022/05/16 5:11 PM

Collapse comment: W. Trevor King added a comment - 2021/02/23 4:00 PM, Edited by W. Trevor King - 2021/02/23 4:00 PM

Expand comment: W. Trevor King added a comment - 2021/02/23 4:00 PM, Edited by W. Trevor King - 2021/02/23 4:00 PM

Collapse comment: W. Trevor King added a comment - 2021/02/23 3:32 PM

Expand comment: W. Trevor King added a comment - 2021/02/23 3:32 PM

People

Dates