[1041818781] Upstream Reporter: Benjamin Gilbert
Upstream issue status: Closed
Upstream description:

First, verify that you meet all the [prerequisites](https://github.com/coreos/fedora-coreos-streams/blob/main/RELEASE.md#prerequisites)

Name this issue `stable: new release on YYYY-MM-DD` with today's date. Once the pipeline spits out the new version ID, you can append it to the title e.g. `(31.20191117.3.0)`.

1. Pre-release

1. 1. Promote testing changes to stable

• [x] Add the `ok-to-promote` label to the issue
• [x] Review the promotion PR opened by the bot against the `stable` branch on https://github.com/coreos/fedora-coreos-config
• [x] Once CI has passed, merge it

1. 1. Build

• [x] Start a [pipeline build](https://jenkins-fedora-coreos.apps.ocp.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/build?delay=0sec) (select `stable`, leave all other defaults). This will automatically run multi-arch builds.
• Post links to the jobs as a comment to this issue
• [x] x86_64
• [x] aarch64 ([multi-arch pipeline](https://jenkins-fedora-coreos.apps.ocp.ci.centos.org/job/multi-arch-pipeline/))
• Wait for the jobs to finish and succeed
• [x] x86_64
• [x] aarch64

1. 1. Sanity-check the build

Using the [the build browser for the `stable` stream](https://builds.coreos.fedoraproject.org/browser?stream=stable):

• Verify that the parent commit and version match the previous `stable` release (in the future, we'll want to integrate this check in the release job)
• [x] x86_64
• [x] aarch64
• Check [kola AWS runs](https://jenkins-fedora-coreos.apps.ocp.ci.centos.org/job/kola-aws/) to make sure they didn't fail
• [x] x86_64
• [x] aarch64
• [x] Check [kola GCP run](https://jenkins-fedora-coreos.apps.ocp.ci.centos.org/job/kola-gcp/) to make sure it didn't fail
• [x] Check [kola OpenStack run](https://jenkins-fedora-coreos.apps.ocp.ci.centos.org/job/kola-openstack/) to make sure it didn't fail

1. ?? Release ??

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs `rpm-ostree
upgrade` will have the new update.

1. 1. Run the release job

• [x] Run the [release job](https://jenkins-fedora-coreos.apps.ocp.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline-release/build?delay=0sec), filling in for parameters `stable` and the new version ID
• [x] Post a link to the job as a comment to this issue
• [x] Wait for job to finish

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

1. 1. Refresh metadata (stream and updates)

• [x] Wait for all releases that will be released simultaneously to reach this step in the process
• [x] Go to the [rollout workflow](https://github.com/coreos/fedora-coreos-streams/actions/workflows/rollout.yml), click "Run workflow", and fill out the form

<details>
<summary>Manual alternative</summary>

• Make sure your `fedora-coreos-stream-generator` binary is up-to-date.

From a checkout of this repo:

• Update stream metadata, by running:

```
fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/stable/releases.json -output-file=streams/stable.json -pretty-print
```

• Add a rollout. For example, for a 48-hour rollout starting at 10 AM ET the same day, run:

```
./rollout.py add stable <version> "10 am ET today" 48
```

• Commit the changes and open a PR against the repo
  </details>

• [x] Verify that the PR contains the expected OS versions
• [x] Post a link to the resulting PR as a comment to this issue
• [x] Review and approve the PR, then wait for someone else to approve it also
• [x] Once approved, merge it and verify that the [`sync-stream-metadata` job](https://jenkins-fedora-coreos.apps.ocp.ci.centos.org/job/sync-stream-metadata/) syncs the contents to S3
• [x] Verify the new version shows up on [the download page](https://getfedora.org/en/coreos/download?stream=stable)
• Verify the incoming edges are showing up in the update graph.
• [x] [x86_64](https://builds.coreos.fedoraproject.org/graph?stream=stable&basearch=x86_64)
• [x] [aarch64](https://builds.coreos.fedoraproject.org/graph?stream=stable&basearch=aarch64)

<details>
<summary>Update graph manual check</summary>

```
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=stable&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=aarch64&stream=stable&rollout_wariness=0'
```

</details>

NOTE: In the future, most of these steps will be automated.

1. 1. Housekeeping

• [x] If one doesn't already exist, [open an issue](https://github.com/coreos/fedora-coreos-streams/issues/new?labels=kind/release,jira&template=stable.md) in this repo with the approximate date in the title of the next release in this stream.
• [x] Issues opened via the previous link will automatically create a linked Jira card. Assign the GitHub issue and Jira card to the next person in the [rotation](https://hackmd.io/WCA8XqAoRvafnja01JG_YA).