Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-10262

[Testing] Integration Testing with oc-mirror

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • Security & Compliance
    • False
    • Hide

      None

      Show
      None
    • False
    • Not Selected

      [Testing] Integration Testing with oc-mirror

      Summary

      Validate the end-to-end workflow with oc-mirror v2 performing multi-architecture filtered mirroring. Ensure digest preservation and sparse manifest handling works correctly in real-world disconnected OpenShift deployment scenarios.

      Acceptance Criteria

      • [ ] oc-mirror can detect Quay sparse manifest support via capability endpoint
      • [ ] oc-mirror can perform architecture-filtered mirroring to Quay
      • [ ] Original manifest list digest is preserved after filtered mirroring
      • [ ] Filtered architectures are correctly stored in Quay
      • [ ] Non-filtered architectures are not stored (storage savings verified)
      • [ ] OpenShift can pull images using original digest references
      • [ ] Error handling works correctly for missing architectures
      • [ ] Performance metrics (storage savings) are measurable
      • [ ] Backwards compatibility with non-sparse registries verified

      Technical Requirements

      Test Environment Setup

      Requirements:
      - Quay instance with FEATURE_SPARSE_INDEX=True configured
      - oc-mirror v2 with sparse manifest support
      - Source registry with multi-architecture images
      - Test namespace/repository for mirroring
      - Metrics collection for storage comparison

      Test Scenarios

      Scenario 1: Basic Architecture-Filtered Mirror

      # Configure oc-mirror to use sparse manifest support
      # Mirror only amd64 from a multi-arch image
      
      # 1. Create imageset config for filtered mirror
      cat > imageset-config.yaml <<EOF
      kind: ImageSetConfiguration
      apiVersion: mirror.openshift.io/v1alpha2
      mirror:
        platform:
          architectures:
            - amd64
        operators:
          - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
            packages:
              - name: advanced-cluster-management
      EOF
      
      # 2. Run oc-mirror with sparse mode
      oc-mirror --config imageset-config.yaml \
        docker://quay.example.com/test-mirror \
        --sparse-index
      
      # 3. Verify results
      # - Original digest preserved
      # - Only amd64 manifests present
      # - Manifest list is marked as sparse
      

      Scenario 2: Multiple Architecture Filter

      # Mirror amd64 and arm64 (common cloud/edge combo)
      
      # 1. Configure for multiple architectures
      cat > imageset-config.yaml <<EOF
      kind: ImageSetConfiguration
      apiVersion: mirror.openshift.io/v1alpha2
      mirror:
        platform:
          architectures:
            - amd64
            - arm64
      EOF
      
      # 2. Run mirror and verify both architectures present
      # 3. Verify ppc64le and s390x are NOT present
      # 4. Verify manifest list digest matches source
      

      Scenario 3: Digest Preservation Verification

      # Test script to verify digest preservation
      
      import requests
      import hashlib
      
      def verify_digest_preservation(source_registry, dest_registry, image, tag):
          """Verify manifest list digest is preserved after sparse mirror."""
      
          # Get source manifest list
          src_response = requests.get(
              f"https://{source_registry}/v2/{image}/manifests/{tag}",
              headers={"Accept": "application/vnd.oci.image.index.v1+json"}
          )
          src_digest = hashlib.sha256(src_response.content).hexdigest()
      
          # Get destination manifest list
          dest_response = requests.get(
              f"https://{dest_registry}/v2/{image}/manifests/{tag}",
              headers={"Accept": "application/vnd.oci.image.index.v1+json"}
          )
          dest_digest = hashlib.sha256(dest_response.content).hexdigest()
      
          assert src_digest == dest_digest, \
              f"Digest mismatch: source={src_digest}, dest={dest_digest}"
      
          print(f"✓ Digest preserved: sha256:{src_digest}")
      
      # Run verification
      verify_digest_preservation(
          "registry.redhat.io",
          "quay.example.com",
          "test-mirror/advanced-cluster-management",
          "v2.9.0"
      )
      

      Scenario 4: OpenShift Pull Test

      # Verify OpenShift can pull using original digest
      
      # 1. Deploy pod using digest reference
      cat > test-pod.yaml <<EOF
      apiVersion: v1
      kind: Pod
      metadata:
        name: test-sparse-pull
      spec:
        containers:
        - name: test
          image: quay.example.com/test-mirror/image@sha256:abc123...
      EOF
      
      kubectl apply -f test-pod.yaml
      
      # 2. Verify pod starts successfully
      kubectl wait --for=condition=Ready pod/test-sparse-pull --timeout=60s
      
      # 3. Verify correct architecture was pulled
      kubectl exec test-sparse-pull -- uname -m
      

      Scenario 5: Storage Savings Measurement

      # Compare storage usage between full and sparse mirrors
      
      def measure_storage_savings(full_mirror_repo, sparse_mirror_repo):
          """Compare storage between full and sparse mirrors."""
      
          # Get repository storage from Quay API
          full_size = get_repo_storage(full_mirror_repo)
          sparse_size = get_repo_storage(sparse_mirror_repo)
      
          savings = (full_size - sparse_size) / full_size * 100
      
          print(f"Full mirror storage: {full_size / (1024**3):.2f} GB")
          print(f"Sparse mirror storage: {sparse_size / (1024**3):.2f} GB")
          print(f"Storage savings: {savings:.1f}%")
      
          # Epic requires 50%+ savings for single architecture
          assert savings >= 50, f"Expected 50%+ savings, got {savings:.1f}%"
      
      def get_repo_storage(repo):
          """Get total storage for a repository."""
          response = requests.get(
              f"https://quay.example.com/api/v1/repository/{repo}",
              headers={"Authorization": f"Bearer {token}"}
          )
          return response.json().get("quota_report", {}).get("configured_quota", 0)
      

      Scenario 6: Error Handling for Missing Architecture

      # Verify appropriate error when pulling missing architecture
      
      # 1. Mirror only amd64
      oc-mirror --config amd64-only.yaml docker://quay.example.com/test-mirror
      
      # 2. Attempt to pull arm64 (should fail gracefully)
      podman pull --arch arm64 quay.example.com/test-mirror/image:tag
      
      # Expected error:
      # Error: manifest unknown: Could not find manifest for architecture arm64
      

      Scenario 7: Backwards Compatibility

      # Verify non-sparse registries still work
      
      # 1. Disable sparse support on Quay
      # FEATURE_SPARSE_INDEX = False
      
      # 2. Attempt filtered mirror
      oc-mirror --config filtered.yaml docker://quay.example.com/test-mirror
      
      # 3. Verify oc-mirror falls back to full mirror
      # OR
      # 3. Verify oc-mirror detects lack of support and fails gracefully
      

      Test Data Requirements

      Use well-known multi-architecture images for testing:
      - registry.access.redhat.com/ubi8/ubi:latest (amd64, arm64, ppc64le, s390x)
      - OpenShift operator images with multiple architectures
      - Custom test images with known architecture content

      Automated Test Suite

      File: test/integration/test_sparse_manifest_oc_mirror.py (new file)

      import pytest
      import subprocess
      import requests
      
      class TestOcMirrorIntegration:
          """Integration tests for oc-mirror with sparse manifest support."""
      
          @pytest.fixture
          def quay_with_sparse(self):
              """Configure Quay with sparse manifest support enabled."""
              # Setup Quay with FEATURE_SPARSE_INDEX=True
              yield
              # Cleanup
      
          def test_capability_detection(self, quay_with_sparse):
              """Test oc-mirror can detect sparse support."""
              response = requests.get(
                  "https://quay.example.com/api/v1/registry/capabilities"
              )
              assert response.json()["sparse_manifests"]["supported"] == True
      
          def test_filtered_mirror_single_arch(self, quay_with_sparse):
              """Test mirroring single architecture."""
              result = subprocess.run([
                  "oc-mirror",
                  "--config", "test-amd64-only.yaml",
                  "--sparse-index",
                  "docker://quay.example.com/test"
              ], capture_output=True)
              assert result.returncode == 0
      
          def test_digest_preserved(self, quay_with_sparse):
              """Test original digest is preserved."""
              # Implementation as shown above
      
          def test_missing_arch_error(self, quay_with_sparse):
              """Test pulling missing architecture returns error."""
              result = subprocess.run([
                  "podman", "pull",
                  "--arch", "ppc64le",
                  "quay.example.com/test/image:tag"
              ], capture_output=True)
              assert result.returncode != 0
              assert b"manifest unknown" in result.stderr
      
          def test_storage_savings(self, quay_with_sparse):
              """Test 50%+ storage savings for single arch."""
              # Implementation as shown above
      

      Implementation Notes

      Test Environment Options

      1. Local development: Use Podman/Docker with local Quay instance
      2. CI/CD integration: Use ephemeral Quay instances in OpenShift
      3. Manual testing: Use dedicated QE environment

      Coordination with oc-mirror Team

      • Align on capability detection endpoint format
      • Confirm oc-mirror v2 sparse manifest support is available
      • Share test results and edge cases

      Metrics to Collect

      1. Storage usage comparison (full vs sparse)
      2. Mirror operation time (should be faster with fewer manifests)
      3. Pull success/failure rates by architecture
      4. Digest verification results

      Known Limitations

      • oc-mirror v2 required (v1 may not support sparse)
      • Skopeo version must support individual manifest copying
      • Source registry must be accessible during test

      Dependencies

      • Story 03: Mirror worker architecture filtering must be complete
      • Story 06: UI architecture filter for configuration
      • Story 07: UI sparse visualization for verification
      • Story 04: API capability detection endpoint

      Testing Requirements

      Pre-requisites

      • [ ] oc-mirror v2 with sparse support available
      • [ ] Test Quay instance configured
      • [ ] Test image data available in source registry
      • [ ] Network connectivity between components

      Test Execution Plan

      1. Unit Tests: Run automated test suite
      2. Manual Verification: Execute scenarios manually
      3. Performance Testing: Measure storage savings
      4. Edge Cases: Test error handling and fallbacks

      Documentation Updates

      After testing, update:
      - User documentation with tested workflows
      - Troubleshooting guide with common issues found
      - Release notes with verified capabilities

      Definition of Done

      • [ ] All test scenarios pass
      • [ ] Storage savings meet 50%+ target
      • [ ] Digest preservation verified
      • [ ] OpenShift pull workflow validated
      • [ ] Error handling verified
      • [ ] Backwards compatibility confirmed
      • [ ] Test results documented
      • [ ] oc-mirror team sign-off
      • [ ] QE test plan reviewed and approved

              marckok Marcus Kok
              marckok Marcus Kok
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: