Background

• Supply chain security has moved into the spotlight of enterprise and cloud services ever since the SolarWinds incident
• Traditional long-term key management (as available in various KMS systems) is an option but adoption is low, especially upstream and also creates UX problems, challenges in automation and increases attack surface
• OpenShift so far has only support basic, simple GPG-based signing with keys stored on OpenShift nodes
• Red Hat has invested significant effort upstreams to create an easy to use a key-less signing toolchain, public signing service and signature trust root for the open source community at https://sigstore.dev, the scope initially being container images and compiled binaries. With support from IBM Research, Red Hat is also investing in the ability to sign ACM policies which are stored in Git. The UX is very much like Let's Encrypt for custom TLS certificates.
• the SigStore project creates a scalable tool chain for signing of artifacts that OpenShift can leverage

Feature Overview

• Quay should be enhanced to enable customers to add signatures to their custom images.
• OpenShift should be enhanced to validate the signatures for container images signed via the the cosign tool provided as part of the sigstore toolchain, either against KMS systems or SigStores key-less approach
• Out of the box Signature verification support in OpenShift is useful for restricting OpenShift clusters to only run container images that were signed by a trusted identity; the most likely implementation is an admission controller.
• cosign's support for existing PKI and KMS systems enables us to deliver a productized solution for customers to run and verify their own signed images in OpenShift
• Alignment is gained by Sigstore/cosign support being introduced in ACM, Quay, podman/buildah, RHEL and potentially other products in the future
• Eventually OpenShift payload images (core images, operator catalogs, operator bundles, operator images and related images, sample imagestream, cincinnati graph image) should be signed and verified using Sigstore infrastructure
• Eventually, other files such as deployments / deployment configs, config maps should be signed as well and the ability to validate the signature for these files provided. 

Goals

• Customers can leverage OpenShift Plus to create trust relationships for running container images (and later on potentially arbitrary artifacts or manifests) in a standardized way
• OpenShift Release Engineering can leverage a mature signing and signature verification stack instead of relying on simple signing
• Specifically, customers can trust signed images from a Red Hat registry or their own registry like Red Hat Quay and restrict running images to those with a trusted signature
• Specifically, customers can trust signed manifests served, distributed and applied via ACM and restrict acceptance of manifests to those with a trusted signature
• Specifically, customers can use OpenShift Pipelines to produce signed images
• Specifically, customers can use Red Hat Quay to sign images

Working document is available here: https://docs.google.com/document/d/11e4lcFPdKU0TFefk84JBOpeTykuZavy65qLrCugLBYk/edit?usp=sharing 

Requirements

Use Cases

• An image signed via cosign can be verified by the container runtime before a Pod is started in OpenShift
• OpenShift allows configuring of trusted sources for public keys (KMS, PKS) or WebPKI (fulcio)
• Trust can be enforced either cluster-wide or namespaces, so that execution of unsigned images is prohibited
• All Red Hat OpenShift images are signed via cosign and are verified OpenShift's runtime
• Customers can produce signed artifacts in OpenShift Pipelines and complete the verification chain on OpenShift
• Customers can sign their custom images in Quay and complete the verification chain on OpenShift

Questions to answer…

• Do we wait for RHEL to have cosign support before adding it to RHCOS?

Out of Scope

• Customers which currently lack a compatible PKI or KMS can leverage productized components of the SigStore project (rekor, fulcio) to ran WebPKI and transparency log on top of an OpenShift Hub/service cluster
• All OpenShift images (core and optional) are signed via cosign and RHCOS is configured out of the box with the correct source for their keys or potentially keyless signing

Background, and strategic fit

OpenShift currently only supports simple signing, primarily aimed at verifying signatures of OpenShift payload images from Red Hat Registries, where the public keys are already baked in the the RHCOS nodes. This is not scalable and leaves the onus on the customer to provide a mechanism to inject keys into the OCP worker nodes and implement rotation policies.

While other parts of the portfolio are adopting the SigStore toolchain (RHEL, podman/buildah, Quay, ACM, ...) OpenShift needs to follow suit.

Assumptions

• Customers have a supported PKI and private keys or are willing to use OIDC to generate a certificate for the signature
• Customers have a signing process in place to sign their images (will be supported in Quay later on)
• Customers have a registry that supports cosign signatures (Quay as of 3.6, most other registries are already supported)

Customer Considerations

• Customers looking for an end-to-end solution will need to wait until we productize the transparency log and WebPKI that is part of SigStore, namely rekor and fulcio 

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
  • docs for users / admins / security offers are needed
• Does this feature have doc impact?
  • yes
• New Content, Updates to existing content, Release Note, or No Doc Impact
  • new doc content is required to document the configuration of signature verification and supported PKI / KMS backends as well as registry requirements
• What concepts do customers need to understand to be successful in signature verification?
  • key pair creation and storage
  • signature creation and storage
  • verification configuration in OpenShift
• How do we expect customers will use the feature? For what purpose(s)?
  • customers want to enable this feature in OpenShift to validate signed images they produce or that are shipped with 
• What reference material might a customer want/need to complete signature verification?
  • SigStore WebPKI (and integration with existing OIDC providers)
  • Rekor transparency log
  • KMS integration
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
  • https://sigstore.dev/what_is_sigstore/
  • https://github.com/sigstore 
• What is the doc impact (New Content, Updates to existing content, or Release Note)?
  • New Content