Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-10860

Quay 3.17 Immutability Missing audit logs when tags become immutable via policy enforcement

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • quay-v3.17.0
    • quay
    • False
    • Hide

      None

      Show
      None
    • False

      Summary

      When tags become immutable due to immutability policy enforcement (either automatically on push or retroactively when policies are created/updated), *no audit log entries are created* to record these state changes. This creates critical gaps in compliance audit trails and makes it impossible to trace when/why tags became immutable.

      Audit Logging Gaps

      What IS Logged ✅

      Immutability Policy CRUD Operations:

      • create_immutability_policy - When policy is created (endpoints/api/immutability_policy.py:103)
      • update_immutability_policy - When policy is updated (endpoints/api/immutability_policy.py:149)
      • delete_immutability_policy - When policy is deleted (endpoints/api/immutability_policy.py:165, 300)

      Manual Tag Immutability Changes:

      • change_tag_immutability - When user explicitly toggles tag immutability via REST API (endpoints/api/tag.py:272-278)

      What is NOT Logged ❌ (Critical Gaps)

      Gap 1: No Audit Log When New Tags Automatically Become Immutable

      File: data/model/oci/tag.py:481-498

      # Check if tag should be immutable based on policies
      immutable = False
      if features.IMMUTABLE_TAGS:
          from data.model import immutability
          
          immutable = immutability.evaluate_immutability_policies(
              manifest.repository_id, repo.namespace_user_id, tag_name
          )
      
      created = Tag.create(
          name=tag_name,
          repository=manifest.repository_id,
          manifest=manifest,
          immutable=immutable,  # ← Automatically set, NO LOG!
      )
      

      Impact:

      • When user pushes nginx:latest and it matches policy pattern ^latest$, tag becomes immutable
      • *NO audit log entry* showing this happened
      • User cannot determine when/why the tag became immutable
      • No traceability to which policy caused the immutability

      Gap 2: No Audit Log When Existing Tags Become Immutable Due to Policy Creation/Update

      File: data/model/immutability.py:161-207, 142-158

      def apply_immutability_policy_to_existing_tags(...):
          """Retroactively apply an immutability policy to existing tags."""
          total_marked = 0
          
          while True:
              tags = _fetch_candidate_tags_batch(namespace_id, repository_id, last_id, batch_size)
              
              tag_ids_to_mark = [
                  tag.id for tag in tags
                  if not tag.immutable and _matches_policy(tag.name, tag_pattern, tag_pattern_matches)
              ]
              
              if tag_ids_to_mark:
                  marked = _mark_tags_immutable_batch(tag_ids_to_mark)  # ← NO LOG!
                  total_marked += marked
          
          return total_marked
      
      def _mark_tags_immutable_batch(tag_ids: list[int]) -> int:
          """Mark a batch of tags as immutable."""
          return (
              Tag.update(immutable=True)  # ← Direct database UPDATE, NO audit log!
              .where(
                  Tag.id << tag_ids,
                  Tag.immutable == False,
              )
              .execute()
          )
      

      When This Happens:

      1. User creates new immutability policy with pattern ^v.*$
      2. Function create_namespace_immutability_policy() calls apply_immutability_policy_to_existing_tags() (line 289-294)
      3. *Could mark hundreds of existing tags as immutable*
      4. *NO audit log entries* for any of these changes

      Impact:

      • Administrator creates policy affecting 500 tags
      • All 500 tags silently become immutable
      • No audit trail showing which tags were affected
      • No record of when this happened (beyond the policy creation timestamp)

      Affected Code Files

      • data/model/oci/tag.py (lines 481-498) - Automatic immutability on tag creation
      • data/model/immutability.py (lines 161-207) - Retroactive policy application
      • data/model/immutability.py (lines 142-158) - Batch immutability marking
      • data/model/immutability.py (lines 269-311) - create_namespace_immutability_policy()
      • data/model/immutability.py (lines 314-367) - update_namespace_immutability_policy()
      • endpoints/api/tag.py (lines 272-278) - Manual immutability change (already logs correctly)

      Audit Trail Gaps Summary

      Scenario Current Logging What's Missing
      User creates immutability policy create_immutability_policy ❌ Which existing tags became immutable
      User updates immutability policy update_immutability_policy ❌ Which tags changed immutability state
      User pushes new tag matching policy ❌ None ❌ Tag created as immutable (no log)
      User manually toggles tag immutability change_tag_immutability N/A (working correctly)

      Example Audit Gap Scenario

      Timeline:

      1. 2026-03-06 10:00 - Admin creates namespace policy: {"tag_pattern": "^v.*$", "tag_pattern_matches": true}
      2. Audit Log Entry: create_immutability_policy (namespace: myorg, pattern: ^v.*$)
      3. Behind the scenes: 847 existing tags (v1.0.0, v2.3.1, etc.) are marked immutable
      4. Audit Log Entry:NONE - No record of these 847 tag changes
      5. 2026-03-06 10:05 - User tries to overwrite v1.0.0 tag
      6. Error: tag is immutable and cannot be overwritten
      7. User Investigation: Checks audit logs, sees policy creation but NO record showing this tag became immutable

      Compliance & Security Impact

      Compliance Issues:

      • SOC 2 Audit Trail: Cannot demonstrate when/why tags became immutable
      • Change Control: No record of mass immutability changes
      • Forensic Investigation: Cannot trace immutability state changes back to policy changes
      • ISO 27001 Compliance: Missing audit trail for critical asset state changes

      User Experience Issues:

      • Users cannot determine why their tag is immutable
      • No visibility into which policy caused immutability
      • Difficult to troubleshoot unexpected immutable tags
      • No way to audit which tags were affected by policy changes

      Scale of Impact:

      • Single policy creation could affect thousands of tags
      • No record of this mass change in audit logs
      • Only the policy creation is logged, not the consequences

      Recommended Fixes

      Fix 1: Log Retroactive Immutability Changes (Batch Summary)

      File: data/model/immutability.py:161-207

      def apply_immutability_policy_to_existing_tags(...):
          from data import model as data_model
          
          total_marked = 0
          affected_tags = []  # Track for audit log
          
          while True:
              tags = _fetch_candidate_tags_batch(...)
              
              tag_ids_to_mark = [...]
              
              if tag_ids_to_mark:
                  marked = _mark_tags_immutable_batch(tag_ids_to_mark)
                  total_marked += marked
                  
                  # Collect tag names for audit log
                  affected_tags.extend([tag.name for tag in tags if tag.id in tag_ids_to_mark])
              
              last_id = tags[-1].id
          
          # ✅ Log batch immutability change
          if total_marked > 0:
              from endpoints.api import log_action
              
              log_action(
                  "apply_immutability_policy_retroactive",
                  namespace_name,
                  {
                      "namespace": namespace_name,
                      "repo": repository_name if repository_id else None,
                      "tag_pattern": tag_pattern,
                      "tags_affected_count": total_marked,
                      "sample_tags": affected_tags[:10],  # First 10 for reference
                  },
                  repo_name=repository_name if repository_id else None,
              )
          
          return total_marked
      

      New LogEntryKind Needed:

      # In initdb.py or migration
      LogEntryKind.create(name="apply_immutability_policy_retroactive")
      

      Fix 2: Log Automatic Immutability on Tag Creation

      File: data/model/oci/tag.py:481-498

      # Check if tag should be immutable based on policies
      immutable = False
      matching_policy = None
      if features.IMMUTABLE_TAGS:
          from data.model import immutability
          
          immutable = immutability.evaluate_immutability_policies(
              manifest.repository_id, repo.namespace_user_id, tag_name
          )
          
          # ✅ Get which policy matched (requires enhancing evaluate_immutability_policies)
          if immutable:
              matching_policy = immutability.get_matching_policy_info(
                  manifest.repository_id, repo.namespace_user_id, tag_name
              )
      
      created = Tag.create(
          name=tag_name,
          repository=manifest.repository_id,
          manifest=manifest,
          immutable=immutable,
      )
      
      # ✅ Log if tag was created as immutable
      if immutable and matching_policy:
          from endpoints.api import log_action
          
          log_action(
              "tag_created_immutable_by_policy",
              namespace_name,
              {
                  "namespace": namespace_name,
                  "repo": repository_name,
                  "tag": tag_name,
                  "policy_uuid": matching_policy.get("uuid"),
                  "policy_pattern": matching_policy.get("pattern"),
                  "policy_scope": matching_policy.get("scope"),  # "namespace" or "repository"
              },
              repo_name=repository_name,
          )
      

      New LogEntryKind Needed:

      LogEntryKind.create(name="tag_created_immutable_by_policy")
      

      Alternative: Enhanced change_tag_immutability Log

      Instead of new log entry kinds, extend existing change_tag_immutability to include policy-driven changes:

      log_action(
          "change_tag_immutability",
          namespace,
          {
              "username": username or "system",  # "system" for policy-driven
              "repo": repository,
              "tag": tag_name,
              "immutable": True,
              "previous_immutable": False,
              "trigger": "policy",  # "manual" or "policy"
              "policy_uuid": policy_uuid if policy_driven else None,
              "policy_pattern": pattern if policy_driven else None,
          },
          repo_name=repository,
      )
      

      Impact Assessment

      Severity: High (Compliance/Audit Issue)

      • Compliance Risk: Cannot demonstrate change control for SOC 2, ISO 27001
      • User Impact: Confusion about immutability, difficult troubleshooting
      • Operational Impact: No visibility into mass immutability changes
      • Security Impact: Cannot forensically trace when/why tags became immutable

      Affected Users:

      • All users with immutability policies (common enterprise feature)
      • Compliance-sensitive organizations (finance, healthcare, government)
      • Administrators investigating immutability issues

      Frequency:

      • Occurs every time a tag is pushed that matches a policy
      • Occurs every time a policy is created or updated (mass retroactive changes)
      • High impact for organizations with active immutability policies

      Reproduction Steps

      Scenario 1: Tag Creation

      1. Create namespace immutability policy: pattern ^latest$
      2. Push new image tag: podman push quay.example.com/org/repo:latest
      3. Check audit logs for the repository
      4. Observe: No log entry showing tag became immutable
      5. Expected: Log entry showing tag created as immutable due to policy

      Scenario 2: Retroactive Application

      1. Create repository with 100 existing tags: v1.0.0, v1.1.0, v2.0.0, etc.
      2. Create immutability policy: pattern ^v.*$
      3. Check audit logs
      4. Observe: Only create_immutability_policy entry, no entries for 100 tags becoming immutable
      5. Expected: Batch log entry showing 100 tags were marked immutable by policy

      Testing Requirements

      1. Unit test: Verify audit log created when tag matches policy on creation
      2. Unit test: Verify batch audit log created when retroactively applying policies
      3. Integration test: Create policy affecting 1000+ tags, verify audit log
      4. Compliance test: Verify audit trail completeness for SOC 2 requirements
      5. Performance test: Ensure logging doesn't significantly impact tag push performance

      Related Features

      • Immutability policies (namespace and repository level)
      • Audit logging system
      • Tag lifecycle operations
      • Compliance and security auditing

              rhn-support-bpratt Brady Pratt
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: