[Registry] Sparse Manifest Support for Multi-Architecture Filtering

Overview

Enable Quay to accept and serve sparse manifest lists - manifest lists where not all referenced child manifests are stored locally. This capability allows organizations to mirror only specific architectures (e.g., amd64, arm64) from multi-architecture images while preserving the original manifest list digest, which is critical for disconnected OpenShift deployments where image references are digest-based.

Context

Organizations running disconnected OpenShift environments must mirror extensive sets of operator images to enable layered product functionality. These images are typically distributed as multi-architecture manifest lists containing images for x86_64, ppc64le, s390x, and arm64. Currently, mirroring requires downloading all architectures even when only one is needed, resulting in hundreds of gigabytes of unnecessary data transfer and storage. Truncating manifest lists to only needed architectures changes the digest, breaking OpenShift's digest-based image references in disconnected environments.

The solution is sparse manifest support: the ability to store a manifest list with its original digest while only storing a subset of the referenced child manifests locally. This has been standardized upstream in the OCI distribution-spec and implemented in distribution/distribution.

Reference: PROJQUAY-3114

Scope

In Scope

• Global configuration option to enable sparse manifest list acceptance
• Registry core changes to accept manifest lists with missing child manifests
• API endpoint for clients to query sparse manifest support capability
• Database schema updates to track which manifests in a list are locally available
• Repository mirroring architecture filter configuration
• Mirror worker changes to perform sparse uploads of filtered architectures
• UI visual indication of sparse manifest lists and which architectures are present
• Integration with oc-mirror and other OpenShift tooling

Out of Scope

• Organization or repository-level sparse manifest configuration (global only for MVP)
• Automatic fetching of missing architectures on pull request
• Converting existing full manifest lists to sparse
• Quay-to-Quay mirroring of sparse manifests (future enhancement)

Child Stories

1. Database Schema for Architecture Tracking: Add fields to track which child manifests in a manifest list are locally present. Update RepoMirrorConfig to include architecture filter configuration. Create Alembic migration.
2. Registry Core Sparse Manifest Acceptance: Modify manifest PUT endpoint to accept manifest lists where not all child manifests exist locally when sparse mode is enabled. Leverage existing SparseManifestList class. Add global configuration option.
3. API Endpoint for Sparse Manifest Capability Detection: Create new API endpoint allowing clients (oc-mirror, skopeo) to query whether the registry supports sparse manifests. Return capability flags in response.
4. Mirror Worker Architecture Filtering: Update repository mirror worker to filter manifests by configured architectures. Modify skopeo integration to copy only selected architectures instead of using --all flag. Preserve original manifest list digest.
5. Mirror API Architecture Configuration: Extend mirror configuration API to accept and return architecture filter settings. Add validation for architecture values.
6. UI Architecture Filter for Mirroring: Add multi-select component in mirroring configuration UI to select which architectures to mirror. Display current architecture filter settings.
7. UI Sparse Manifest Visualization: Add visual indicators in tag/manifest views showing when a manifest list is sparse. Differentiate between locally-present and missing architecture entries in manifest list display.
8. Integration Testing with oc-mirror: Validate end-to-end workflow with oc-mirror v2 performing multi-arch filtered mirroring. Ensure digest preservation and sparse manifest handling works correctly.
9. Documentation and Release Enablement: Document sparse manifest feature, configuration options, and usage with oc-mirror. Create release enablement materials.

Dependencies

• Technical:
  • Upstream distribution-spec sparse manifest support (PR #310 - merged)
  • distribution/distribution sparse manifest implementation (PR #3536 - merged)
  • Existing SparseManifestList class in image/shared/types.py
  • Skopeo for manifest copying operations
• Cross-team:
  • oc-mirror team (OCPSTRAT-1808) for client integration
  • OpenShift installer team for disconnected workflow validation
  • Documentation team for user guides
  • QE for test plan development
• External:
  • oc-mirror v2 sparse manifest support
  • Skopeo manifest inspection capabilities

Success Criteria

• [ ] Sparse manifest lists can be pushed to Quay without all child manifests present
• [ ] Original manifest list digest is preserved when pushing sparse manifests
• [ ] Global configuration option enables/disables sparse manifest acceptance
• [ ] Clients can detect sparse manifest support via API endpoint
• [ ] Repository mirroring respects architecture filter configuration
• [ ] Only configured architectures are stored when mirroring with filter
• [ ] UI clearly indicates sparse manifest lists and present vs. missing architectures
• [ ] oc-mirror can successfully perform filtered multi-arch mirroring to Quay
• [ ] Storage savings of 50%+ achieved when mirroring single architecture from multi-arch images
• [ ] All existing mirroring functionality continues to work (backwards compatible)

Technical Approach

Components Affected

• data/database.py: Add architecture fields to RepoMirrorConfig, potentially add manifest presence tracking
• data/model/repo_mirror.py: Architecture filter management functions
• data/model/oci/manifest.py: Handle sparse manifest child relationships
• data/migrations/versions/: New migration for schema changes
• endpoints/v2/manifest.py: Accept sparse manifest lists when enabled
• endpoints/api/mirror.py: Expose architecture configuration in API
• workers/repomirrorworker/__init__.py: Architecture-filtered mirroring logic
• util/repomirror/skopeomirror.py: Filtered manifest copying
• image/shared/types.py: Leverage existing SparseManifestList class
• web/src/routes/RepositoryDetails/Mirroring/: Architecture selection UI
• web/src/routes/TagDetails/: Sparse manifest visualization
• config.py / features.py: Global configuration option

Key Technical Decisions

• Global configuration only for MVP: Simplifies implementation; org/repo-level can be added later
• Leverage existing SparseManifestList: Already implemented in codebase, used by proxy model
• Modify skopeo usage: Cannot use --all flag with filtering; must parse manifest lists and copy individual manifests
• Preserve manifest list digest: Critical requirement - digest must match upstream for OpenShift compatibility
• Opt-in via configuration: Sparse manifest acceptance disabled by default for safety

Risks and Mitigations

• Risk: Clients pulling sparse manifests may fail if requested architecture not present
  Mitigation: Clear error messages indicating missing architecture; UI shows which architectures are available
• Risk: Breaking existing mirroring workflows
  Mitigation: Architecture filter is optional; default behavior (all architectures) unchanged
• Risk: Complex skopeo integration for filtered copying
  Mitigation: Parse manifest list in Python, determine architectures, copy only matching manifests
• Risk: Digest verification failures when accepting sparse manifests
  Mitigation: Carefully implement to accept manifest list bytes as-is without modification
• Risk: Inconsistent state if mirror job fails mid-sync
  Mitigation: Transactional approach; mark manifests as available only after successful copy

Testing Strategy

• Unit testing: Test manifest list parsing, architecture filtering logic, sparse manifest handling
• Registry protocol tests: Validate sparse manifest push/pull operations
• Integration testing: End-to-end mirror workflow with architecture filtering
• E2E testing (Playwright/Cypress): UI components for architecture selection and sparse visualization
• oc-mirror integration testing: Validate with actual oc-mirror v2 workflows
• Performance testing: Measure storage savings with filtered mirroring
• Backwards compatibility testing: Ensure existing mirrors without architecture config work unchanged

Rollout Strategy

• Configuration-gated: Feature disabled by default via global config
• Documentation first: Document feature before enabling in production
• Staged enablement: Test in development/staging before production
• Rollback plan: Disable configuration option; existing manifests remain valid
• Client coordination: Coordinate with oc-mirror team for compatible release timing

Documentation Needs

• User guide: Enabling sparse manifest support
• User guide: Configuring architecture filters for repository mirroring
• User guide: Using oc-mirror with Quay sparse manifest support
• API documentation: Sparse manifest capability detection endpoint
• Admin guide: Global configuration options for sparse manifests
• Troubleshooting: Common issues with sparse manifests and missing architectures

Related Work

• Original Feature: PROJQUAY-3114
• oc-mirror Integration: OCPSTRAT-1808 (Multi-arch filtering via Sparse Manifests)
• Upstream Spec: OCI distribution-spec PR #310
• Reference Implementation: distribution/distribution PR #3536
• Related: PROJQUAY-2143 (Architecture selection in mirroring - incorporated)
• Related: RFE-3333 (oc-mirror multi-arch filtering)