Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-10504

Immutability bypass: Manifest deletion by digest skips immutable tags instead of blocking

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • quay-v3.17.0
    • quay
    • False
    • Hide

      None

      Show
      None
    • False

      Executive Summary

      This is a CRITICAL security and data integrity bug in the OCI/Docker Registry v2 API implementation.

      Problem: Manifest deletion behavior is inconsistent depending on whether deletion is requested by tag name or by digest.

      • Delete by tag name: CORRECTLY BLOCKS if tag is immutable
      • Delete by digest: SILENTLY SKIPS immutable tags and deletes manifest anyway

      Impact: Immutability can be bypassed, tags become orphaned, data integrity violated, compliance requirements broken.


      The Bug

      What You Would Expect

      When a tag is immutable (protected by immutability policy), you expect:

      • Cannot delete the tag
      • Cannot delete the manifest it points to
      • Tag and manifest are locked forever

      What Actually Happens

      Method 1: Try to delete by tag name
      DELETE /v2/myrepo/manifests/v1.0.0
      Result: Error: tag is immutable (CORRECTLY BLOCKED)

      Method 2: Try to delete by manifest digest
      DELETE /v2/myrepo/manifests/sha256:abc
      Result: Deleted: sha256:abc (IMMUTABILITY BYPASSED!)

      Result after digest deletion:

      • Manifest deleted from storage
      • Tag v1.0.0 still exists in database
      • Tag now points to NOTHING (orphaned)
      • Users cannot pull v1.0.0 anymore (manifest unknown error)

      Security Impact

      Severity: CRITICAL

      • Immutability bypass (security control circumvented)
      • Data integrity violation (orphaned tags)
      • Compliance violations (regulatory requirements broken)
      • Production outages (broken references)

      Attack Scenario:
      1. Policy requires production images tagged prod-* to be immutable
      2. Image pushed as prod-v1.0.0 (becomes immutable)
      3. Normal deletion blocked: docker manifest rm repo:prod-v1.0.0 (Error: immutable)
      4. Attacker bypasses: docker manifest rm repo@sha256:abc (Success!)
      5. Production deployment fails: Error: manifest unknown


      Steps to Reproduce

      1. Create repository and push image as v1.0.0
      2. Create immutability policy: tagPattern=v.*, tagPatternMatches=true
      3. Verify tag is immutable
      4. Try delete by tag name: DELETE /v2/repo/manifests/v1.0.0
      Expected: HTTP 422 (immutable) - CORRECT
      5. Get manifest digest
      6. Try delete by digest: DELETE /v2/repo/manifests/sha256:abc
      Expected: HTTP 422 (should match tag behavior)
      Actual: HTTP 202 Accepted - BUG!
      7. Try to pull image: docker pull repo:v1.0.0
      Expected: Success
      Actual: Error: manifest unknown (tag orphaned)


      Root Cause

      Two different code paths with inconsistent enforcement:

      Path 1: delete_manifest_by_tag() (endpoints/v2/manifest.py:439-472)

      • Calls delete_tag() which checks if tag.immutable
      • Raises ImmutableTagException if immutable
      • Returns HTTP 422 TagImmutable
      • CORRECT BEHAVIOR

      Path 2: delete_manifest_by_digest() (endpoints/v2/manifest.py:402-436)

      • Calls delete_tags_for_manifest() which SKIPS immutable tags
      • Returns HTTP 202 Accepted
      • Manifest deleted, immutable tags orphaned
      • INCORRECT BEHAVIOR

      The delete_tags_for_manifest() function (data/model/oci/tag.py:548-578) filters out immutable tags and logs Skipping deletion of immutable tag but proceeds with deletion anyway.


      Recommended Fix

      Make digest-based deletion also check for immutable tags:

      File: endpoints/v2/manifest.py:402-436

      Add before deletion:

      • Get all tags for manifest
      • Check if ANY tags are immutable
      • If yes: raise TagImmutable with error message listing immutable tag names
      • If no: proceed with deletion

      New helper function needed:

      • get_immutable_tags_for_manifest(manifest) in registry_oci_model.py
      • Returns list of immutable tags pointing to manifest

      Benefits:

      • Consistent behavior with tag-based deletion
      • Clear error message
      • No orphaned tags
      • Immutability guarantee preserved

      Testing Requirements

      Unit Tests:

      • test_delete_by_digest_blocks_if_immutable_tag_exists
      • test_delete_by_digest_succeeds_if_all_tags_mutable
      • test_delete_by_digest_with_mixed_tags
      • test_delete_by_tag_still_blocks_immutable (regression)

      Integration Tests:

      • E2E test with policy creation, push, and delete attempt
      • Verify error message includes immutable tag names
      • Verify image still pullable after failed delete

      Affected Files

      Primary:

      • endpoints/v2/manifest.py:402-436 (delete_manifest_by_digest)
      • data/model/oci/tag.py:548-578 (delete_tags_for_manifest)
      • data/registry_model/registry_oci_model.py (new helper needed)

      Related:

      • endpoints/v2/manifest.py:439-472 (delete_manifest_by_tag - works correctly)
      • data/model/oci/tag.py:504-518 (delete_tag with immutability check)

      Priority Justification

      Why CRITICAL:

      • Security control (immutability) can be bypassed
      • Data integrity violation (orphaned tags)
      • Compliance risk (regulatory violations)
      • Production outages possible
      • Simple fix (4-6 hours including tests)
      • Should block release

      Reported by: Claude Code Automated Security Analysis
      Discovery: Comprehensive immutability policy regression review
      Branch: redhat-3.17

              lzha1981 luffy zhang
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: