-
Spike
-
Resolution: Done
-
Critical
-
None
-
None
-
None
-
False
-
None
-
False
-
-
-
OTA 254
Impact statement for the OCPBUGS-35236 bug series:
Which 4.y.z to 4.y'.z' updates increase vulnerability?
- Updates into 4.16.0-rc.3, 4.16.0-rc.4, and 4.17.0-ec.0, which have
OCPBUGS-24700's cvo#1030.
Which types of clusters?
All standalone clusters with the default feature set. Hosted/HyperShift and MicroShift clusters are not exposed.
$ oc get -o json featuregate cluster | jq .spec.featureSet "TechPreviewNoUpgrade"
That TechPreviewNoUpgrade cluster is not exposed. But if the output is an empty string, the standalone cluster would be exposed after updating into an impacted release. In PromQL:
0 * group by (invoker) (cluster_installer{_id="",invoker="hypershift"})
or on (_id)
group by (name) (cluster_feature_set{_id="",name=""})
or on (_id)
0 * group by (name) (cluster_feature_set{_id=""})
will return 0 for hosted/HyperShift clusters (OTA-907), 1 for exposed non-HyperShift clusters and 0 for unexposed non-HyperShift clusters
What is the impact? Is it serious enough to warrant removing update recommendations?
Requests to update out of an impacted release (e.g. from 4.16.0-rc.3 to 4.16.0) will be rejected with ReleaseAccepted=False and message like:
Retrieving payload failed version="4.16.0-rc.4" image="quay.io/openshift-release-dev/ocp-release@sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74" failure=The update cannot be verified: unable to verify sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 against keyrings: verifier-public-key-redhat
Cluster-version operator logs and Events will mention ClusterVersion spec.signatureStores is an empty array:
$ oc -n openshift-cluster-version get events | grep RetrievePayloadFailed 39m Warning RetrievePayloadFailed clusterversion/version Retrieving payload failed version="4.16.0-rc.4" image="quay.io/openshift-release-dev/ocp-release@sha256:6c236c400d3bad9b2b54d8a3b247c508f6f13511d37666de1eecca8e43bce0f6" failure=The update cannot be verified: unable to verify sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 against keyrings: verifier-public-key-redhat // [2024-06-10T07:38:16Z: prefix sha256-5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 in config map signatures-managed: no more signatures to check, 2024-06-10T07:38:16Z: ClusterVersion spec.signatureStores is an empty array. Unset signatureStores entirely if you want to enable the default signature stores]
and CVO logs:
$ oc -n openshift-cluster-version logs -l k8s-app=cluster-version-operator --tail -1 | grep RetrievePayloadFailed
I0610 07:38:16.208595 1 event.go:364] Event(v1.ObjectReference{Kind:"ClusterVersion", Namespace:"openshift-cluster-version", Name:"version", UID:"", APIVersion:"config.openshift.io/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RetrievePayloadFailed' Retrieving payload failed version="4.16.0-rc.4" image="quay.io/openshift-release-dev/ocp-release@sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74" failure=The update cannot be verified: unable to verify sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 against keyrings: verifier-public-key-redhat // [2024-06-10T07:38:16Z: prefix sha256-5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 in config map signatures-managed: no more signatures to check, 2024-06-10T07:38:16Z: ClusterVersion spec.signatureStores is an empty array. Unset signatureStores entirely if you want to enable the default signature stores]
How involved is remediation?
Enabling TechPreviewNoUpgrade is a one-way change, but doing that and unsetting signatureStores would be one way out.
Alternatively, clear the current update request with oc adm upgrade --clear, wait for a fixed update target, and set up a ConfigMap with the signature of the target release. rc.4 doesn't have the fix yet, but on an rc.3 cluster, the signature ConfigMap setup looks like:
$ VERSION=4.16.0-rc.4
$ HASH=5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74
$ SIGNATURE="$(curl "https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=${HASH}/signature-1" | base64 -w0)"
$ cat <<EOF >signature.yaml
> apiVersion: v1
> kind: ConfigMap
> metadata:
> name: openshift-release-signature
> namespace: openshift-config-managed
> annotations:
> kubernetes.io/description: Local copy of the OpenShift ${VERSION} release signature to work around https://issues.redhat.com/browse/OTA-1297 .
> labels:
> release.openshift.io/verification-signatures: ""
> binaryData:
> sha256-${HASH}-1: ${SIGNATURE}
> EOF
$ oc apply -f signature.yaml
after which you can request the update as usual, and signature verification should succeed.
Is this a regression?
Yes, OCPBUGS-24700 regressed signature handling for clusters that do not enable the TechPreviewNoUpgrade signatureStores property.
- blocks
-
OCPBUGS-35233 CVO fails to validate signatures: signatureStores is an empty array
-
- Closed
-
-
-
- Closed
-
- is related to
-
-
- Closed
-
- relates to
-
-
- Closed
-
- links to