-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
4.16.0, 4.17.0
-
+
-
Critical
-
Yes
-
0.5
-
OTA 255
-
1
-
Approved
-
False
-
-
Standalone clusters (not hosted/HyperShift) with the default feature set (not TechPreviewNoUpgrade) running 4.17.0-ec.0 could not find and download release signatures. This bugfix restores their ability to download release signatures.
-
Bug Fix
-
In Progress
-
-
-
Description of problem:
Attempts to update a cluster to a release payload with a signature published by Red Hat fails with CVO failing to verity the signature, signalled by the ReleaseAccepted=False condition:
Retrieving payload failed version="4.16.0-rc.4" image="quay.io/openshift-release-dev/ocp-release@sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74" failure=The update cannot be verified: unable to verify sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 against keyrings: verifier-public-key-redhat
CVO shows evidence of not being able to find the proper signature in its stores:
$ grep verifier-public-key-redhat cvo.log | head I0610 07:38:16.208595 1 event.go:364] Event(v1.ObjectReference{Kind:"ClusterVersion", Namespace:"openshift-cluster-version", Name:"version", UID:"", APIVersion:"config.openshift.io/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RetrievePayloadFailed' Retrieving payload failed version="4.16.0-rc.4" image="quay.io/openshift-release-dev/ocp-release@sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74" failure=The update cannot be verified: unable to verify sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 against keyrings: verifier-public-key-redhat // [2024-06-10T07:38:16Z: prefix sha256-5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 in config map signatures-managed: no more signatures to check, 2024-06-10T07:38:16Z: ClusterVersion spec.signatureStores is an empty array. Unset signatureStores entirely if you want to enable the default signature stores] ...
Version-Release number of selected component (if applicable):
4.16.0-rc.3
4.16.0-rc.4
4.17.0-ec.0
How reproducible:
Seems always. All CI build farm clusters showed this behavior when trying to update from 4.16.0-rc.3
Steps to Reproduce:
1. Launch update to a version with a signature published by RH
Actual results:
ReleaseAccepted=False and update is stuck
Expected results:
ReleaseAccepted=True and update proceeds
Additional info:
Suspected culprit is https://github.com/openshift/cluster-version-operator/pull/1030/ so the fix may be a revert or an attempt to fix-forward, but revert seems safer at this point.
Evidence:
- #1030 was added in 4.16.0-rc.3
- #1030 code is supposed to work with an updated ClusterVersion CRD field .spec.signatureStores which right now is in TechPreview, so it is not enabled by default
- CVO log hints that it is trying to process the field but fails [1], and that somehow the signature store may be considered as an explicitly, intentionally empty array instead of default set of signature locations
- easily reproducible on any release that contains #1030
- testing cluster with a #1030 revert did not reproduce the bug, successfuly started updating to rc4
[1]
...ClusterVersion spec.signatureStores is an empty array. Unset signatureStores entirely if you want to enable the default signature store... W0610 07:58:59.095970 1 warnings.go:70] unknown field "spec.signatureStores"
- blocks
-
OCPBUGS-35249 CVO fails to validate signatures: signatureStores is an empty array
-
- Closed
-
- clones
-
OCPBUGS-35233 CVO fails to validate signatures: signatureStores is an empty array
-
- Closed
-
- is blocked by
-
OTA-1297 Impact CVO fails to validate signatures: signatureStores is an empty array
-
- Closed
-
- is cloned by
-
OCPBUGS-35249 CVO fails to validate signatures: signatureStores is an empty array
-
- Closed
-
- links to
-
RHEA-2024:3718 OpenShift Container Platform 4.17.z bug fix update