Feature Overview

Products building on top of the Clair static vulnerability analyzer core library and/or distribution support a simple workflow to get Clair reporting to work correctly in offline environments by leveraging a self-contained, runnable offline bundle in the form of a container image that can be used to initially populate Clair's vulnerability data sources and keep them updated. In turn, Clair will run in "airplane mode" where it will seize to contact any out-of-cluster service to retrieve vulnerability data and operate solely off its internal database.

Goals

• users of Clair-based products like ACS or Quay leverage the same easy workflow to keep Clair's database populated with the latest vulnerability feeds and auxiliary data (CPE mapping files etc)
• we provide a portable offline bundle that our existing mirroring tool (oc-mirror) can pick up without further changes, i.e. a container image
• the offline bundle can be used in the environment without the need for manual preparation steps outside of the cluster
• the offline bundle is self-contained, so that no additional orchestration beyond basic Kubernetes lifecycle is required by either the ACS or Quay operator
• Clair does not produce noise in its logs with error messages about futile attempts to contact external APIs and endpoints to retrieve vuln data, when in offline mode
• the offline workflow can run periodically where it will automatically pick up any newer offline bundle, making offline updates for Clair completely automated

Requirements (aka. Acceptance Criteria):

• a runnable container image will be produced regularly by Red Hat, that contains both offline vulnerability data and the necessary software to load it into Clair
  • in case licensing terms prohibit us from re-distributing non-Red Hat vulnerability feeds this way, the bundle will only contain Red Hat's CSAF/VEX feeds
• we provide instructions in how to refresh the offline bundle based on the published image described above in the form of a simple Containerfile / Dockerfile
  • in case licensing terms prohibit us from re-distributing non-Red Hat vulnerability feeds this way, this will also be the way to create an offline bundle that contains non-Red Hat vuln data by the hand of the customer
• all data that Clair needs from external sources to do proper matching and reporting in an offline environment need to solely come from its internal Postgres database as opposed to additional file mounts etc, so that the offline bundle can simply be executed against the Clair database in an idempotent operation to load this data
  • this in particular pertains to the current use of the filesystem to get to the content of the CPE JSON files in a disconnected environment
• as a result, no additional orchestration around the offline bundle or awareness about its content should be required by the user or Kubernetes operators, it should simply be able to run periodically with the Clair config file as part of a cron jobs on Linux or CronJobs in Kubernetes
• the offline bundle itself makes no assumptions about how it is run (on Linux or Kubernetes), it solely requires the Clair Postgres database to be writeable
• Clair needs to be aware when it is in offline ("airplane") mode and refrain from any attempts to contact off-cluster APIs/endpoints during matching and updater operations so that no error messages should be produced in offline operations of Clair
• The above user experience should be available in equivalent form for ACS users

User Experience:

1. A Quay administrator installs Quay in a disconnected environment and enables Clair. They obtain the Clair offline bundle from registry.redhat.io via oc-mirror and push it to their offline container registry.
2. A Quay administrator points the Quay operator to the location of the image in their offline registry. As a result the operator runs a Job that runs the offline bundle which in turn populates Clair's database with all relevant offline data.
3. A Quay administrator optionally configures the offline bundle to be run periodically by providing an interval setting in the QuayRegistry custom resource. As a result, the operator creates a CronJob, which regularly executes the offline bundle so that a new offline image will be pulled automatically, if it have changed in the offline registry in the meantime.
4. The success of the (continuous) offline bundle execution is conveyed to the Quay administrator via the status condition of the QuayRegistry custom resource
5. Quay administrators running Quay on RHEL via podman can simply execute a one-off run of the offline bundle image to populate their Clair deployment with vulnerability data and leverage standard cron tooling to automate this task.
6. Offline users can rebuild the Clair offline themselves by using a podman build command with a Containerfile / Dockerfile provided by Red Hat product documentation, should they want more frequent vulnerability data refreshes compared to the intervals the offline bundle is published by Red Hat

Questions to Answer (Optional):

1. How often could be publish the offline bundle on registry.redhat.io?
2. Would we be allowed to redistribute 3rd party vuln data this way?

Prior work

This readme describes in more detailed the expected user experience: https://docs.google.com/document/d/1QQIJAoaCYrzYTgPFsr4h2f9mSSDeYmMKCKxyuE9ljh8/

Background

RFE-5030 describes the need for complex, manual orchestration to get Clair working properly in disconnected environments. Because the vulnerability is not provided in cloud-native form, multiple manual steps need to be executed by the user every time they want to update Clair's vuln databases, leading to a subpar user experience and often to Clair deployments with outdated vulnerability databases. Because not all data is supplied in the same format, additional manual orchestration is necessary on OpenShift, to supply Clair with additional supplementary files next to its vuln data in the database, which is not something trivial to automate in a Kubernetes environment and for which only a manual workaround exists today, that requires the customer to deploy and update Clair entirely manually.

Customer Considerations

Customers are conditioned to receive all data from Red Hat in container format, to run and update OpenShift and layered products in a disconnected fashion. All of OpenShift's disconnected tooling is centered around this fact. Meanwhile, with technologies like RHEL Image Mode, other Red Hat products are also pursuing this pattern. Any manual step that is required to operator our products in a disconnected environment is experienced by the customer as a regression compared to the connected experience and should be kept to a minimum.

Documentation Considerations

Both ACS and Quay product documentation should cover how to obtain, rebuild, and run the offline updater bundle periodically to keep Clair's vulnerability databases updated over time.

Interoperability Considerations

Ideally, there should be no differences of the offline updater bundle for ACS Clair and Quay Clair.