-
Epic
-
Resolution: Done
-
Undefined
-
None
-
None
-
None
-
None
-
2024-10-22 Cincinnati / Update Service outage
-
False
-
None
-
False
-
Not Selected
-
To Do
Epic Goal
Retrospective to squeeze improvements out of the incident.
Why is this important?
Improving the future to limit our exposure to previously-identified issues. We have existing tickets like OTA-681 tracking improvements. This ticket isn't intended to hold work itself, it's tracking the incident and linking out to other tickets (in any project) that would have helped with this incident.
Scenarios
The Cincinnati pods backing the OpenShift Update Service had a few hours of struggle and drama. We would prefer to address the tooling to make that kind of incident less likely in the future. Making it easy to manage related work makes it more likely that that work is identified, prioritized, and delivered.
Dependencies
No actual work under this ticket; it's just a place for priority-setters to see links to tickets in other places that might be worth prioritizing.
Acceptance Criteria
We'll close once we get through the retro, since folks are unlikely to be brainstorming about new work to link after that point.
Drawbacks or Risk
Maybe we're comfortable with our current service level objectives and don't think it's worth trying to brainstorm or prioritize further improvements.
Done - Checklist
- Retro meeting complete.
- At least OTA team lead and project manager have reviewed linked tickets to assess their priority.
- depends on
-
OTA-681 Production/public instance of OSUS should be able to scale without causing issues in a multi-tenant environment- phase2
- To Do
-
OTA-1379 Cincinnati backoff for registry 502s
- Review
-
OCPBUGS-44018 Cluster-version operator should cache update advice through OSUS outages
- Verified
-
OTA-1378 Consolidated logging for Sigstore signatures and other ignored tags
- Closed
-
OTA-1375 Mitigate Cincinnati's slow Quay scrapes
- Closed
-
OTA-1382 Surge into new Deployment configs for internal Cincinnati
- Closed
-
OTA-1387 Raise internal Cincinnati memory limits
- Closed