Based on the initial Feature Presentation here.  

Epic Goal

The goal of this epic is to drive the development of a centralized and automated solution that will be co-developed, owned, and operated by the ACM DevOps team which would accomplish the release iteration process with minimal manual intervention. This automation would do the following:

• Create pipeline/backplane-pipeline repository release-X.Y branches and initial artifacts
• Create release repository release-X.Y branches and initial artifacts/version bumps
• Create the release-X.Y branches on the multiclusterhub-operator and multiclusterhub-repo repositories and update version references
• Create a new release-X.Y branch for the new Y-stream on each component repository in the manifest
• This step would alert the CICD team on repositories with a pre-existing branch matching the release-X.Y branch to be created. These conflicts should be avoided, but if they occur, they will need manual intervention. This step should loudly alert the CICD team to reconcile these colliding branch names.
• Programatically update OSCI/Prow configurations to update versions and version-related artifacts in a "best effort" manner when met with irregular patterns
• Update release-ff for all components to fast-forward to the newly-created release-X.Y branch.
• Notify of any projects skipped or external to the ACM organization that will required manual action

Why is this important?

TLDR: To deliver and iterate with greater confidence in the development stream with less opportunity for human error, meanwhile reducing time to deliver new streams from days to hours while also improving parallel development opportunity.

Centralized and largely automated release iteration has numerous benefits:
1. An automated and centralized release iteration process allows us to cut a new development stream with greater confidence. It is less prone to human error than manual intervention across 70+ projects across numerous repositories, prow configurations, and jobs.
2. An automated and centralized release iteration process reduces the time, coordinational effort, and churn necessary to create a new stream. This automation would allow us to near-atomically cut a new release once we have signoff from the program, as opposed to the current process which involves scheduled work across all development squads, some external contributors, CICD, and Installer in a coordinated manner. This automated process would reduce the timeline from days or weeks to hours.
3. A centralized release iteration process allows us to increase parallel development and program flexibility by potentially opening new streams for development via automation at any time.

Scenarios

We currently plan to address Scenario #1, with #2 and #3 listed as future extensions.

Automated release iteration tooling could be utilized in a few scenarios:
1. To create a new Y-stream release (for ex. ACM 2.8) at the end of a given Y-stream's development cycle (on feature freeze)
2. To automatically kick-off a new z-stream release (for ex. 2.6.2) once the previous z-stream ships
3. To pre-emptively create a new Y-stream release (for ex. ACM 2.8) during the previous Y-stream's development cycle (for parallel development)

Acceptance Criteria

For a given new Y-strea:

• pipeline/backplane-pipeline artifacts and branches can be automatically produced and seeded
• release repo artifacts and branch can be automatically produced and configured
• branches on the multiclusterhub-operator and multiclusterhub-repo repositories are automatically created with updated version references
• branch for the new Y-stream on each component repository in the manifest are automatically created
• OSCI/Prow configurations to update versions and version-related artifacts are automaticaly created in a "best effort" manner when met with irregular patterns
• release-ff is automatically re-configured for all components to target the new y-stream
• Notifications are sent for any projects skipped or manual actions to be taken

Dependencies (internal and external)

• Development squad reveiew and signoff of results
• Installer squad collaboraiton on iteration automation

Previous Work (Optional):

• See: CPaaS work by Brian Smith (Brian-bot) to automate z-stream bumps

Open questions::

• What can be completed in ACM 2.7 in preparation for ACM 2.8.
• Can we sustainably develop and maintain this automation release-over-release

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>