Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-36344

use-sigstore-attachments should cover relevant mirrors too

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Undefined Undefined
    • None
    • 4.17
    • Node / CRI-O
    • None
    • Moderate
    • None
    • OCPNODE Sprint 256 (Blue)
    • 1
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the Container Runtime Config controller did not detect whether a mirror configuration was in use before adding the scope from a `ClusterImagePolicy` CR to the `/etc/containers/registries.d/sigstore-registries.yaml` file. As a consequence, image verification failed with a `Not looking for sigstore attachments` message. With this fix, images are pulled from the mirror registry as expected. (link:https://issues.redhat.com/browse/OCPBUGS-36344[*OCPBUGS-36344*])
      Show
      * Previously, the Container Runtime Config controller did not detect whether a mirror configuration was in use before adding the scope from a `ClusterImagePolicy` CR to the `/etc/containers/registries.d/sigstore-registries.yaml` file. As a consequence, image verification failed with a `Not looking for sigstore attachments` message. With this fix, images are pulled from the mirror registry as expected. (link: https://issues.redhat.com/browse/OCPBUGS-36344 [* OCPBUGS-36344 *])
    • Bug Fix
    • Done

      Description of problem

      As rhn-engineering-mitr reported upstream, when a ClusterImagePolicy is set on a scope to accept sigstore signatures, the underlying registry needs to be configured with use-sigstore-attachments: true. The current code:

       func generateSigstoreRegistriesdConfig(clusterScopePolicies map[string]signature.PolicyRequirements) ([]byte, error) { 
      

      does do that for the configured scope; but the use-sigstore-attachments option applies not to the "logical name", but to each underlying mirror individually.

      I.e. the option needs to be on every mirror of the scope. Without that, if the image is found on one of such mirrors, the c/image code will not be looking for signatures on the mirror, and policy enforcement is likely to fail.

      Version-Release number of selected component

      Seen in 4.17.0-0.nightly-2024-06-25-162526, but likely all releases which implement ClusterImagePolicy so far, because this is unlikely to be a regression.

      How reproducible

      Every time.

      Steps to Reproduce

      Apply the ClusterImagePolicy suggested in OTA-1294's enhancements#1633:

      $ cat <<EOF >policy.yaml
      apiVersion: config.openshift.io/v1alpha1
      kind: ClusterImagePolicy
      metadata:
        name: openshift
        annotations:
          kubernetes.io/description: Require Red Hat signatures for quay.io/openshift-release-dev/ocp-release container images.
          exclude.release.openshift.io/internal-openshift-hosted: "true"
          include.release.openshift.io/self-managed-high-availability: "true"
          release.openshift.io/feature-set: TechPreviewNoUpgrade
      spec:
        scopes:
        - quay.io/openshift-release-dev/ocp-release
        policy:
          rootOfTrust:
            policyType: PublicKey
            publicKey:
              keyData: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQ0lqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDQ2dLQ0FnRUEzQzJlVGdJQUo3aGxveDdDSCtIcE1qdDEvbW5lYXcyejlHdE9NUmlSaEgya09ZalRadGVLSEtnWUJHcGViajRBcUpWYnVRaWJYZTZKYVFHQUFER0VOZXozTldsVXpCby9FUUEwaXJDRnN6dlhVbTE2cWFZMG8zOUZpbWpsVVovaG1VNVljSHhxMzR2OTh4bGtRbUVxekowR0VJMzNtWTFMbWFEM3ZhYmd3WWcwb3lzSTk1Z1V1Tk81TmdZUHA4WDREaFNoSmtyVEl5dDJLTEhYWW5BMExzOEJlbG9PWVJlTnJhZmxKRHNzaE5VRFh4MDJhQVZSd2RjMXhJUDArRTlZaTY1ZE4zKzlReVhEOUZ6K3MrTDNjZzh3bDdZd3ZZb1Z2NDhndklmTHlJbjJUaHY2Uzk2R0V6bXBoazRjWDBIeitnUkdocWpyajU4U2hSZzlteitrcnVhR0VuVGcyS3BWR0gzd3I4Z09UdUFZMmtqMnY1YWhnZWt4V1pFN05vazNiNTBKNEpnYXlpSnVSL2R0cmFQMWVMMjlFMG52akdsMXptUXlGNlZnNGdIVXYwaktrcnJ2QUQ4c1dNY2NBS00zbXNXU01uRVpOTnljTTRITlNobGNReG5xU1lFSXR6MGZjajdYamtKbnAxME51Z2lVWlNLeVNXOHc0R3hTaFNraGRGbzByRDlkVElRZkJoeS91ZHRQWUkrK2VoK243QTV2UVV4Wk5BTmZqOUhRbC81Z3lFbFV6TTJOekJ2RHpHellSNVdVZEVEaDlJQ1I4ZlFpMVIxNUtZU0h2Tlc3RW5ucDdZT2d5dmtoSkdwRU5PQkF3c1pLMUhhMkJZYXZMMk05NDJzSkhxOUQ1eEsrZyszQU81eXp6V2NqaUFDMWU4RURPcUVpY01Ud05LOENBd0VBQVE9PQotLS0tLUVORCBQVUJMSUMgS0VZLS0tLS0K
      EOF
      $ oc apply -f policy.yaml
      

      Set up an ImageContentSourcePolicy such as the ones Cluster Bot jobs have by default:

      cat <<EOF >mirror.yaml
      apiVersion: operator.openshift.io/v1alpha1
      kind: ImageContentSourcePolicy
      metadata:
        name: pull-through-mirror
      spec:
        repositoryDigestMirrors:
        - mirrors:
          - quayio-pull-through-cache-us-west-2-ci.apps.ci.l2s4.p1.openshiftapps.com
          source: quay.io
      EOF
      $ oc apply -f mirror.yaml
      

      Set CRI-O debug logs, following these docs:

      $ cat <<EOF >custom-loglevel.yaml
      apiVersion: machineconfiguration.openshift.io/v1
      kind: ContainerRuntimeConfig
      metadata:
        name: custom-loglevel
      spec:
        machineConfigPoolSelector:
          matchLabels:
            pools.operator.machineconfiguration.openshift.io/master: ''
        containerRuntimeConfig:
          logLevel: debug
      EOF
      $ oc create -f custom-loglevel.yaml
      

      Wait for that to roll out, as described in docs:

      $ oc get machineconfigpool master
      

      Launch a Sigstore-signed quay.io/openshift-release-dev/ocp-release image, by asking the cluster to update to 4.16.1:

      $ oc adm upgrade --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:c17d4489c1b283ee71c76dda559e66a546e16b208a57eb156ef38fb30098903a
      

      Check the debug CRI-O logs:

      $ oc adm node-logs --role=master -u crio | grep -i1 sigstore | tail -n5
      

      Actual results

      Not looking for sigstore attachments: disabled by configuration entries like:

      $ oc adm node-logs --role=master -u crio' | grep -i1 sigstore | tail -n5
      --
      Jun 28 19:06:34.317335 ip-10-0-43-59 crio[2154]: time="2024-06-28 19:06:34.317169116Z" level=debug msg=" Using transport \"docker\" specific policy section quay.io/openshift-release-dev/ocp-release" file="signature/policy_eval.go:150"
      Jun 28 19:06:34.317335 ip-10-0-43-59 crio[2154]: time="2024-06-28 19:06:34.317207897Z" level=debug msg="Reading /var/lib/containers/sigstore/openshift-release-dev/ocp-release@sha256=c17d4489c1b283ee71c76dda559e66a546e16b208a57eb156ef38fb30098903a/signature-1" file="docker/docker_image_src.go:479"
      Jun 28 19:06:34.317335 ip-10-0-43-59 crio[2154]: time="2024-06-28 19:06:34.317240227Z" level=debug msg="Not looking for sigstore attachments: disabled by configuration" file="docker/docker_image_src.go:556"
      Jun 28 19:06:34.317335 ip-10-0-43-59 crio[2154]: time="2024-06-28 19:06:34.317277208Z" level=debug msg="Requirement 0: denied, done" file="signature/policy_eval.go:285"
      

      Expected results

      Something about "we're going to look for Sigstore signatures on quayio-pull-through-cache-us-west-2-ci.apps.ci.l2s4.p1.openshiftapps.com, since that's where we found the quay.io/openshift-release-dev/ocp-release@sha256:c17d4489c1b283ee71c76dda559e66a546e16b208a57eb156ef38fb30098903a image". At this point, it doesn't matter whether the retrieved signature is accepted or not, just that a signature lookup is attempted.

            qiwan233 Qi Wang
            trking W. Trevor King
            Min Li Min Li
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: