Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-35236

CVO fails to validate signatures: signatureStores is an empty array

XMLWordPrintable

    • +
    • Critical
    • Yes
    • 0.5
    • OTA 255
    • 1
    • Approved
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, standalone clusters running 4.17.0-ec.0 with the default feature set could not find and download release signatures. With this release, the ability to download release signatures is restored and the issue is resolved. (link:https://issues.redhat.com/browse/OCPBUGS-35236[*OCPBUGS-35236*])
      Show
      * Previously, standalone clusters running 4.17.0-ec.0 with the default feature set could not find and download release signatures. With this release, the ability to download release signatures is restored and the issue is resolved. (link: https://issues.redhat.com/browse/OCPBUGS-35236 [* OCPBUGS-35236 *])
    • Release Note Not Required
    • In Progress

      Description of problem:

      Attempts to update a cluster to a release payload with a signature published by Red Hat fails with CVO failing to verity the signature, signalled by the ReleaseAccepted=False condition:

      Retrieving payload failed version="4.16.0-rc.4" image="quay.io/openshift-release-dev/ocp-release@sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74" failure=The update cannot be verified: unable to verify sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 against keyrings: verifier-public-key-redhat

      CVO shows evidence of not being able to find the proper signature in its stores:

      $ grep verifier-public-key-redhat cvo.log | head
I0610 07:38:16.208595       1 event.go:364] Event(v1.ObjectReference{Kind:"ClusterVersion", Namespace:"openshift-cluster-version", Name:"version", UID:"", APIVersion:"config.openshift.io/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RetrievePayloadFailed' Retrieving payload failed version="4.16.0-rc.4" image="quay.io/openshift-release-dev/ocp-release@sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74" failure=The update cannot be verified: unable to verify sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 against keyrings: verifier-public-key-redhat // [2024-06-10T07:38:16Z: prefix sha256-5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 in config map signatures-managed: no more signatures to check, 2024-06-10T07:38:16Z: ClusterVersion spec.signatureStores is an empty array. Unset signatureStores entirely if you want to enable the default signature stores]
...

      Version-Release number of selected component (if applicable):

      4.16.0-rc.3
      4.16.0-rc.4
      4.17.0-ec.0

      How reproducible:

      Seems always. All CI build farm clusters showed this behavior when trying to update from 4.16.0-rc.3

      Steps to Reproduce:

      1. Launch update to a version with a signature published by RH

      Actual results:

      ReleaseAccepted=False and update is stuck

      Expected results:

      ReleaseAccepted=True and update proceeds

      Additional info:

      Suspected culprit is https://github.com/openshift/cluster-version-operator/pull/1030/ so the fix may be a revert or an attempt to fix-forward, but revert seems safer at this point.

      Evidence:

      • #1030 was added in 4.16.0-rc.3
      • #1030 code is supposed to work with an updated ClusterVersion CRD field .spec.signatureStores which right now is in TechPreview, so it is not enabled by default
      • CVO log hints that it is trying to process the field but fails [1], and that somehow the signature store may be considered as an explicitly, intentionally empty array instead of default set of signature locations
      • easily reproducible on any release that contains #1030
      • testing cluster with a #1030 revert did not reproduce the bug, successfuly started updating to rc4

      [1]

      ...ClusterVersion spec.signatureStores is an empty array. Unset signatureStores entirely if you want to enable the default signature store...
W0610 07:58:59.095970       1 warnings.go:70] unknown field "spec.signatureStores"

              Unassigned Unassigned
              afri@afri.cz Petr Muller
              Jia Liu Jia Liu
              Olivia Brown Olivia Brown
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: