Problem Statement

Quay repository and organization mirror features HARDCODE the --remove-signatures flag in skopeo operations, removing ALL image signatures (Cosign, Notary, Sigstore) during mirroring. This blocks enterprise adoption in air-gapped environments, regulated industries, and organizations with supply chain security requirements.

Current Behavior (Broken)

File: util/repomirror/skopeomirror.py:48

args = args + [
    "copy",
    "--all",
    "--remove-signatures",  # HARDCODED - Cannot be disabled
    "--src-tls-verify=%s" % src_tls_verify,
    "--dest-tls-verify=%s" % dest_tls_verify,
]

Impact:

• All Cosign signatures removed
• All Notary/Docker Content Trust signatures removed
• All Sigstore keyless signatures removed
• All attestations (SBOM, provenance, scan results) removed
• NO configuration option to preserve signatures

Critical Customer Use Cases

1. Air-Gapped/Disconnected Environments

Pattern: Internet → DMZ Quay (mirror) → Air-Gapped Quay → Production K8s

Requirements:

• MUST mirror signatures (cannot re-sign in air-gap - no Internet access to Rekor/Fulcio)
• Signature verification before deployment (Kyverno, OPA Gatekeeper)
• Cannot re-sign (signing keys not available in air-gap for security)

Impact if signatures removed:

• Admission controllers REJECT all unsigned images
• Compliance violations (NIST, FedRAMP, DoD)
• Cannot prove supply chain integrity
• BLOCKING: Cannot deploy ANY workloads

2. SLSA/Supply Chain Security Compliance

Requirements:

• Verify provenance attestations (signed by CI/CD)
• Validate SBOM signatures
• Prove artifacts have not been tampered with
• SLSA Level 3+ compliance

Example Policy (Kyverno):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:   name: require-cosign-signature
spec:   validationFailureAction: Enforce
  rules:   - name: verify-signature
    verifyImages:     - imageReferences:       - "quay.io/myorg/*"
      attestors:       - entries:         - keyless:             rekor:               url: https://rekor.sigstore.dev

Problem: If signatures removed → All pods REJECTED by admission controller

3. Third-Party/Vendor Image Verification

Scenario: Mirroring Red Hat, Google, AWS, CNCF vendor images

Examples:

• quay.io/redhat/ubi9:latest (signed by Red Hat)
• gcr.io/distroless/base (signed by Google)
• public.ecr.aws/aws-cli (signed by AWS)

Requirements:

• Verify vendor signatures before deployment (supply chain attacks like SolarWinds)
• CANNOT re-sign vendor images (customer does not have vendor private key)
• Compliance mandates require verifying original signatures

Impact:

• Cannot verify image authenticity
• Compliance failure (cannot prove provenance)
• Security risk (cannot detect tampering)

4. Regulatory Compliance

NIST SP 800-204D (DevSecOps): Container images MUST be signed and verified before deployment

FedRAMP/DoD: Supply chain artifacts require cryptographic verification of origin

PCI-DSS 4.0 (Requirement 6.3.3): Verify integrity of software components before use

Audit Problem:

• Without signatures: FAIL - Cannot prove image integrity
• With signatures: PASS - Cryptographically verified

Why Re-Signing Is NOT Acceptable

Breaking Chain of Custody

Original:
  quay.io/vendor/app:v1.0 → Signed by vendor (proves THEY built it)

After re-signing:
  internal-quay.io/vendor/app:v1.0 → Signed by customer (proves... what?)

Problems:

• Lost proof of origin
• Cannot prove vendor built the image
• Audit trail broken
• Compliance violation

Keyless Signatures Cannot Be Re-Created

Sigstore keyless signing (GitHub Actions, Google, Red Hat) uses ephemeral certificates:

• No long-lived private key
• Certificate expires in minutes
• Transparency log entry is immutable
• CANNOT re-create original signature

Real Customer Examples

Financial Services (Large Bank)

• Mirror 50,000+ images from Docker Hub, Quay.io, ECR
• MUST preserve signatures for compliance
• Use OPA Gatekeeper to enforce signature verification
• Audit requirement: Prove all container images are verified
• Current workaround: Use Harbor proxy (extra cost, complexity)

Defense Contractor

• Air-gapped Kubernetes in classified network
• Mirror CNCF/vendor images from Internet DMZ
• CANNOT re-sign (signing keys must not enter air-gap)
• Current blocker: Cannot deploy ANY workloads (admission controller rejects unsigned images)
• Current solution: Manual skopeo copy without automation

Healthcare (HIPAA Compliance)

• Verify software supply chain for PHI workloads
• Attestations required: SBOM, vulnerability scans, build provenance
• All attestations are Cosign-signed
• Problem: Audit failure - cannot prove compliance

Competitive Gap

Harbor (CNCF): Preserves Cosign/Notary signatures
Artifactory (JFrog): Preserves signatures during replication
AWS ECR: Preserves signatures during cross-region replication
Quay (Red Hat): REMOVES all signatures (no option to preserve)

Recommended Fix

# File: util/repomirror/skopeomirror.py
def copy(
    self,
    src_image,
    dest_image,
    timeout,
    src_tls_verify=True,
    dest_tls_verify=True,
    src_username=None,
    src_password=None,
    dest_username=None,
    dest_password=None,
    proxy=None,
    verbose_logs=False,
    unsigned_images=False,
    preserve_signatures=True,  # NEW: Default to preserving signatures
):
    args = ["/usr/bin/skopeo"]
    if verbose_logs:
        args = args + ["--debug"]
    if unsigned_images:
        args = args + ["--insecure-policy"]

    args = args + [
        "copy",
        "--all",
    ]
    
    # NEW: Only remove signatures if explicitly requested
    if not preserve_signatures:
        args = args + ["--remove-signatures"]
    
    args = args + [
        "--src-tls-verify=%s" % src_tls_verify,
        "--dest-tls-verify=%s" % dest_tls_verify,
    ]

Configuration API:

# endpoints/api/mirror.py
"preserve_signatures": {
    "type": "boolean",
    "description": "Preserve image signatures during mirroring (Cosign, Notary, etc.)",
    "default": True
}

Affected Features

• Repository mirror (FEATURE_REPO_MIRROR)
• Organization mirror (FEATURE_ORG_MIRROR)
• Proxy cache (if using skopeo backend)

Impact Assessment

Severity: CRITICAL (P0)
Customer Impact: BLOCKING for:

• Enterprise customers (80%+ of Quay revenue)
• Regulated industries (finance, healthcare, defense, government)
• Air-gapped environments
• Supply chain security compliance (SLSA, NIST, FedRAMP)

Business Risk:

• Customer churn to Harbor/Artifactory
• Lost sales in regulated industries
• Reputation damage (security product that breaks security features)

Success Criteria

• Signatures preserved by default during mirror operations
• Configuration option to disable (backward compatibility)
• Support Cosign, Notary v1/v2, Sigstore keyless signatures
• Preserve attestations (SBOM, provenance, scan results)
• No breaking changes to existing mirror configurations

Testing Requirements

• Test with Cosign-signed images
• Test with Notary-signed images
• Test with Sigstore keyless signatures
• Test with attestations (SBOM, provenance)
• Test air-gapped workflow
• Integration test with Kyverno/OPA Gatekeeper

Discovery Context

Discovered during comprehensive codebase review on 2026-03-06 while analyzing organization mirror signature handling.