Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-35249

CVO fails to validate signatures: signatureStores is an empty array

XMLWordPrintable

    • Critical
    • Yes
    • 2
    • OTA 255
    • 1
    • Approved
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the CVO was unable to find and download release signatures for standalone clusters that were running 4.16.0-rc.3 and 4.16.0-rc.4 with a default feature set. With this release, the issue has been resolved. (link:https://issues.redhat.com/browse/OCPBUGS-35249[*OCPBUGS-35249*])
      Show
      * Previously, the CVO was unable to find and download release signatures for standalone clusters that were running 4.16.0-rc.3 and 4.16.0-rc.4 with a default feature set. With this release, the issue has been resolved. (link: https://issues.redhat.com/browse/OCPBUGS-35249 [* OCPBUGS-35249 *])
    • Release Note Not Required
    • Done

      This is a clone of issue OCPBUGS-35236. The following is the description of the original issue:
      —

      Description of problem:

      Attempts to update a cluster to a release payload with a signature published by Red Hat fails with CVO failing to verity the signature, signalled by the ReleaseAccepted=False condition:

      Retrieving payload failed version="4.16.0-rc.4" image="quay.io/openshift-release-dev/ocp-release@sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74" failure=The update cannot be verified: unable to verify sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 against keyrings: verifier-public-key-redhat

      CVO shows evidence of not being able to find the proper signature in its stores:

      $ grep verifier-public-key-redhat cvo.log | head
I0610 07:38:16.208595       1 event.go:364] Event(v1.ObjectReference{Kind:"ClusterVersion", Namespace:"openshift-cluster-version", Name:"version", UID:"", APIVersion:"config.openshift.io/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'RetrievePayloadFailed' Retrieving payload failed version="4.16.0-rc.4" image="quay.io/openshift-release-dev/ocp-release@sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74" failure=The update cannot be verified: unable to verify sha256:5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 against keyrings: verifier-public-key-redhat // [2024-06-10T07:38:16Z: prefix sha256-5e76f8c2cdc81fa40abb809ee5e2d56cb84f409aab773aa9b9c7e8ed8811bf74 in config map signatures-managed: no more signatures to check, 2024-06-10T07:38:16Z: ClusterVersion spec.signatureStores is an empty array. Unset signatureStores entirely if you want to enable the default signature stores]
...

      Version-Release number of selected component (if applicable):

      4.16.0-rc.3
      4.16.0-rc.4
      4.17.0-ec.0

      How reproducible:

      Seems always. All CI build farm clusters showed this behavior when trying to update from 4.16.0-rc.3

      Steps to Reproduce:

      1. Launch update to a version with a signature published by RH

      Actual results:

      ReleaseAccepted=False and update is stuck

      Expected results:

      ReleaseAccepted=True and update proceeds

      Additional info:

      Suspected culprit is https://github.com/openshift/cluster-version-operator/pull/1030/ so the fix may be a revert or an attempt to fix-forward, but revert seems safer at this point.

      Evidence:

      • #1030 was added in 4.16.0-rc.3
      • #1030 code is supposed to work with an updated ClusterVersion CRD field .spec.signatureStores which right now is in TechPreview, so it is not enabled by default
      • CVO log hints that it is trying to process the field but fails [1], and that somehow the signature store may be considered as an explicitly, intentionally empty array instead of default set of signature locations
      • easily reproducible on any release that contains #1030
      • testing cluster with a #1030 revert did not reproduce the bug, successfuly started updating to rc4

      [1]

      ...ClusterVersion spec.signatureStores is an empty array. Unset signatureStores entirely if you want to enable the default signature store...
W0610 07:58:59.095970       1 warnings.go:70] unknown field "spec.signatureStores"

            trking W. Trevor King
            openshift-crt-jira-prow OpenShift Prow Bot
            Jia Liu Jia Liu
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: