-
Epic
-
Resolution: Unresolved
-
Undefined
-
None
-
None
-
None
-
None
-
Expand scope of openshift ClusterImagePolicy to cover the entire OCP release
-
False
-
None
-
False
-
Not Selected
-
To Do
Epic Goal
Expand openshift ClusterImagePolicy coverage to quay.io/openshift-release-dev.
Why is this important?
In 4.17, OTA-1304's cluster-update-keys#60 added an openshift ClusterImagePolicy for tech-preview clusters. That policy enforced Red Hat signatures for the quay.io/openshift-release-dev/ocp-release scope, which holds OCP release images. It doesn't hold the entire release payload, for example:
$ oc adm release info --pullspecs quay.io/openshift-release-dev/ocp-release:4.17.0-x86_64 | grep -A4 'Images:' Images: NAME PULL SPEC agent-installer-api-server quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1e5188d59af0c2cf53b7c1e679886d5e2993cc43b38c32db51efaa90d364182 agent-installer-csr-approver quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d380f37da2d940292bff39dea2f6fa21a8247913b3c9d2dfb6c1650ae26d49e2 agent-installer-node-agent quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b94108ce9f71bf2725a3e6dc416ddad26266e047f92f10ad6d40b360e624aab
Expanding the scope to all of quay.io/openshift-release-dev would ensure that those images were also signed by appropriate signers, and make it easier to audit clusters for supply-chain compliance, without relying on OpenShift-specific logic like release-image references. It would also limit exposure for folks who run release images using mirror-registry synonyms like registry.build10.ci.openshift.org/ci-op-ts2msk7r/release, because the referenced images will still use the canonical quay.io/openshift-release-dev/ocp-v4.0-art-dev.
Scenarios
Attempting to run an unsigned images from quay.io/openshift-release-dev would fail due to the lack-of-acceptable-signature. For example, 4.12.0 predates Sigstore signing, so it has no signatures:
$ oc image info quay.io/openshift-release-dev/ocp-release:4.12.0-x86_64 | grep Digest Digest: sha256:4c5a7e26d707780be6466ddc9591865beb2e3baa5556432d23e8d57966a2dd18 $ oc image info quay.io/openshift-release-dev/ocp-release:sha256-4c5a7e26d707780be6466ddc9591865beb2e3baa5556432d23e8d57966a2dd18.sig error: image "quay.io/openshift-release-dev/ocp-release:sha256-4c5a7e26d707780be6466ddc9591865beb2e3baa5556432d23e8d57966a2dd18.sig" not found: manifest unknown: manifest unknown
And its etcd images has no signature either:
$ oc adm release info --image-for etcd quay.io/openshift-release-dev/ocp-release:4.12.0-x86_64 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24d9b9d9d7fadacbc505c849a1e4b390b2f0fcd452ad851b7cce21e8cfec2020 $ oc image info -o json quay.io/openshift-release-dev/ocp-v4.0-art-dev:sha256-24d9b9d9d7fadacbc505c849a1e4b390b2f0fcd452ad851b7cce21e8cfec2020.sig error: image "quay.io/openshift-release-dev/ocp-v4.0-art-dev:sha256-24d9b9d9d7fadacbc505c849a1e4b390b2f0fcd452ad851b7cce21e8cfec2020.sig" not found: manifest unknown: manifest unknown
The scope expansion in this Epic would reject attempts to pull that 4.12.0 etcd image.
There's also some old nightly cruft in these repositories, from back when we used to push those to Quay:
$ oc adm release info quay.io/openshift-release-dev/ocp-release@sha256:d8e79fd98a07a44aed76a635deea40381d8fab393de0eb1dd7ef1b5ee7483a5f | head -n5 Name: 4.12.0-0.nightly-multi-2022-08-24-183128 Digest: sha256:d8e79fd98a07a44aed76a635deea40381d8fab393de0eb1dd7ef1b5ee7483a5f Created: 2022-08-24T18:34:04Z OS/Arch: multi (linux/amd64) Manifests: 598 $ oc adm release info --image-for etcd quay.io/openshift-release-dev/ocp-release@sha256:d8e79fd98a07a44aed76a635deea40381d8fab393de0eb1dd7ef1b5ee7483a5f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e6426b55848e10cc10b5d56ce2a75ab5d509f26a895ad6c32abf011ea2a7bdd3 $ oc image info quay.io/openshift-release-dev/ocp-v4.0-art-dev:sha256-e6426b55848e10cc10b5d56ce2a75ab5d509f26a895ad6c32abf011ea2a7bdd3.sig error: image "quay.io/openshift-release-dev/ocp-v4.0-art-dev:sha256-e6426b55848e10cc10b5d56ce2a75ab5d509f26a895ad6c32abf011ea2a7bdd3.sig" not found: manifest unknown: manifest unknown
The scope expansion in this Epic would reject attempts to pull that nightly etcd image.
Dependencies
This Epic would increase the number of Sigstore signatures that need to be mirrored to run a tech-preview OCP release from 1 (just the release image) to low-hundreds:
$ oc adm release info -o json quay.io/openshift-release-dev/ocp-release:4.17.0-x86_64 | jq -r '.references.spec.tags[].from.name' | sort | uniq | wc -l 189
So it depends on OCPSTRAT-1417 to deliver oc-mirror support for OCP-release Sigstore signatures, to avoid burdening customers with the need to mirror hundreds of Sigstore signatures via their own workaround tooling.
Contributing Teams
This Epic can be delivered by updating cluster-update-keys' 0000_90_openshift-cluster-image-policy.yaml. Unfortunately that repository belongs to no team (which is why I'm originally creating this Epic in OTA) and has a stale OWNERS mentioning folks who are not longer OpenShift-focused. Part of delivering this Epic will be figuring out whether it's the approver-set who would be driving policy pivots, or if they're just expected to review and approve/reject policy pivots floated by volunteers.
Acceptance Criteria
Running a pod with 4.12.0's etcd image will fail on pulling the unsigned image:
$ cat test.yaml
apiVersion: v1
kind: Pod
metadata:
name: test
spec:
containers:
- name: main
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24d9b9d9d7fadacbc505c849a1e4b390b2f0fcd452ad851b7cce21e8cfec2020
command: ['etcdctl', '--version']
$ oc apply -f test.yaml
pod/test created
$ oc get pod test
FIXME
Drawbacks or Risk
Increasing the number of signatures required to run an OCP release image increases the risk of recovery difficulty if the registry (or registries) hosting Sigstore signatures go down. But the registries serve the images being pulled, so the odds that they break for signatures but not for the signed images seem low.
Done - Checklist
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
- CI Testing - Existing tech-preview CI running from the release controller should cover this. In fact, checking this run from 4.18.0-ec.3:
$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.18-e2e-aws-ovn-techpreview/1852055780233056257/artifacts/e2e-aws-ovn-techpreview/gather-extra/artifacts/pods.json | jq -r '.items[].spec.containers[].image' | sed 's/\(@sha256:\).*/\1/' | sort | uniq -c
1 ghcr.io/opsgenie/kubernetes-event-exporter:v0.11
6 quay.io/observatorium/token-refresher
6 quay.io/openshift-logging/promtail:v2.9.8
482 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:
1 registry.build10.ci.openshift.org/ci-op-ts2msk7r/release@sha256:
6 registry.redhat.io/openshift4/ose-oauth-proxy:latest
So the registry.build10.ci.openshift.org/ci-op-ts2msk7r/release release is not getting checked, but the referenced quay.io/openshift-release-dev/ocp-v4.0-art-dev would start getting checked as of this Epic.
- Documentation - There's no docs exposure, because we currently don't doc the openshift ClusterImagePolicy at all (that's tracked in OSDOCS-10264).
- QE - Maybe QE wants to add the 4.12-etcd-pod-fails-to-pull test as an automated sad-path test?
- Technical Enablement - Slides are complete (if requested by PLM)
- Other
- is blocked by
-
OCPSTRAT-1417 oc-mirror automatically detects and mirror SigStore-style attachments
-
- New
-