-
Epic
-
Resolution: Unresolved
-
Normal
-
None
-
None
-
Ensure all HA components are not degraded by design during upgrades
-
BU Product Work
-
False
-
False
-
In Progress
-
OCPSTRAT-1064 - Improve upgrades - phase 3 - Control plane & worker node independence
-
Impediment
-
-
33% To Do, 33% In Progress, 33% Done
-
Undefined
Epic Goal
- Eliminate the gap between measured availability and Available=true
Why is this important?
- Today it's not uncommon, even for CI jobs, to have multiple operators which blip through either Degraded=True or Available=False conditions
- We should assume that if our CI jobs do this then when operating in customer environments with higher levels of chaos things will be even worse
- We have had multiple customers express that they've pursued rolling back upgrades because the cluster is telling them that portions of the cluster are Degraded or Unavailable when they're actually not
- Since our product is self-hosted, we can reasonably expect that the instability that we experience on our platform workloads (kube-apiserver, console, authentication, service availability), will also impact customer workloads that run exactly the same way: we're just better at detecting it.
Scenarios
- In all of the following, assume standard 3 master 0 worker or 3 master 2+ worker topologies
- Add/update CI jobs which ensure 100% Degraded=False and Available=True for the duration of upgrade
- Add/update CI jobs which measure availability of all components which are not explicitly defined as non-HA (ex: metal's DHCP server is singleton)
- Address all identified issues
Acceptance Criteria
- openshift/enhancements CONVENTIONS outlines these requirements
- CI - Release blocking jobs include these new/updated tests
- Release Technical Enablement - N/A if we do this we should need no docs
- No outstanding identified issues
Dependencies (internal and external)
- ...
Previous Work (Optional):
- Clayton, David, and Trevor identified many issues early in 4.8 development but were unable to ensure all teams addressed them that list is in this query, teams will be asked to address everything on this list as a 4.9 blocker+ bug and we will re-evaluate status closer to 4.9 code freeze to see which may be deferred to 4.10
https://bugzilla.redhat.com/buglist.cgi?columnlist=product%2Ccomponent%2Cassigned_to%2Cbug_severity%2Ctarget_release%2Cbug_status%2Cresolution%2Cshort_desc%2Cchangeddate&f1=longdesc&f2=cf_environment&j_top=OR&list_id=12012976&o1=casesubstring&o2=casesubstring&query_based_on=ClusterOperator%20conditions&query_format=advanced&v1=should%20not%20change%20condition%2F&v2=should%20not%20change%20condition%2F
Open questions::
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- DEV - Tests in place
- DEV - No outstanding failing tests
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
- account is impacted by
-
OCPBUGS-38666 clusteroperator/dns blips Degraded=True during upgrade test
-
- New
-
-
-
- New
-
-
-
- New
-
-
-
- New
-
-
-
- New
-
-
-
- New
-
-
-
- New
-
-
-
- New
-
-
-
- New
-
-
-
- ASSIGNED
-
-
-
- ASSIGNED
-
-
-
- POST
-
-
-
- MODIFIED
-
-
-
- Verified
-
-
-
- Closed
-
- clones
-
-
- Closed
-
- is depended on by
-
OCPSTRAT-836 Improve OpenShift upgrade progress feedback for cluster administrators
-
- New
-
- is related to
-
-
- New
-
-
-
- New
-
-
-
- New
-
-
-
- New
-
-
-
- New
-
-
-
- New
-
-
-
- ASSIGNED
-
-
-
- ASSIGNED
-
-
OTA-980 Is the Failing=True status condition is a good indicator for admins?
-
- To Do
-
- links to
1.
|
QE Tracker |
|
New | 
|
Unassigned |
2.
|
TE Tracker |
|
New |
|
Unassigned |