Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-10859

Quay 3.17 Poor error message quality when pushing to immutable tags

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • quay-v3.17.0
    • quay
    • False
    • Hide

      None

      Show
      None
    • False

      Summary

      When users attempt to push/modify/delete an immutable tag, the Docker Registry v2 API error message is generic and lacks critical information, making it difficult for users to understand which tag failed, what operation was blocked, and how to resolve the issue.

      Example:

      podman push quayregistry-quay-quay-enterprise-15682.apps.quayfipstest-15682.qe.devcluster.openshift.com/quayqe/demo:ubuntu --tls-verify=false --creds ***:***
      Getting image source signatures
      Copying blob sha256:e5dae71ade4390c09123a86ada6c9bc64ac469d0495acae5b2216a627395050c
      Copying config sha256:350d40843c2436d6203ee1e35e9264fb08194c3455ec7462236e01adf542c00b
      Getting image source signatures
      Copying blob sha256:e5dae71ade4390c09123a86ada6c9bc64ac469d0495acae5b2216a627395050c
      Copying config sha256:350d40843c2436d6203ee1e35e9264fb08194c3455ec7462236e01adf542c00b
      Writing manifest to image destination
      Getting image source signatures
      Copying blob sha256:e5dae71ade4390c09123a86ada6c9bc64ac469d0495acae5b2216a627395050c
      Copying config sha256:350d40843c2436d6203ee1e35e9264fb08194c3455ec7462236e01adf542c00b
      Writing manifest to image destination
      Getting image source signatures
      Copying blob sha256:e5dae71ade4390c09123a86ada6c9bc64ac469d0495acae5b2216a627395050c
      Copying config sha256:350d40843c2436d6203ee1e35e9264fb08194c3455ec7462236e01adf542c00b
      Writing manifest to image destination
      Error: writing manifest: uploading manifest ubuntu to quayregistry-quay-quay-enterprise-15682.apps.quayfipstest-15682.qe.devcluster.openshift.com/quayqe/demo: unknown: tag is immutable and cannot be overwritten 

      Current Behavior

      Error Message Shown to Users:

      Error: writing manifest: uploading manifest ubuntu to ...: unknown: tag is immutable and cannot be overwritten
      

      Problems:

      1. Generic message - tag name not in main error message field
      2. Operation type (overwrite/delete/set expiration) not included
      3. No actionable guidance for users
      4. No policy information (which policy blocked, pattern matched, scope)
      5. Displays as "unknown:" prefix in podman/docker clients

      Root Cause Analysis

      Error Flow:

      1. data/model/oci/tag.py raises ImmutableTagException with tag_name, operation, repository_id
      2. endpoints/v2/manifest.py:529-530 catches exception and raises TagImmutable
      3. {}Operation and context information lost during exception conversion{}
      4. endpoints/v2/errors.py:124-128 always returns generic message

      Affected Code Files:

      • endpoints/v2/errors.py (lines 124-128) - TagImmutable error class
      • endpoints/v2/manifest.py (lines 529-530, 465-466) - Exception handling
      • data/model/oci/tag.py (lines 475, 516, 673) - ImmutableTagException raised with operation context

      Detailed Issues

      Issue 1: Generic Message - No Tag Name in Main Field

      Current Implementation (endpoints/v2/errors.py:124-128):

      class TagImmutable(V2RegistryException):
          def __init__(self, detail=None):
              super(TagImmutable, self).__init__(
                  "TAG_IMMUTABLE", "tag is immutable and cannot be overwritten", detail, 409
              )
      

      Problem: Main message field is always generic, tag name buried in detail.message

      Impact: Users don't immediately see which tag caused the error

      Issue 2: Operation Not Included

      Current Exception Handling (endpoints/v2/manifest.py:529-530):

      except ImmutableTagException as ite:
          raise TagImmutable(detail={"message": f"tag '{ite.tag_name}' is immutable"}) from ite
          # ← ite.operation is IGNORED!
      

      Problem: ImmutableTagException includes operation ("overwrite", "delete", "set expiration on") but it's not passed through to TagImmutable

      Impact: Same error message whether user is pushing, deleting, or setting expiration

      Different Operations Raise Same Error:

      • Pushing: data/model/oci/tag.py:475 - ImmutableTagException(tag_name, "overwrite", ...)
      • Deleting: data/model/oci/tag.py:516 - ImmutableTagException(tag_name, "delete", ...)
      • Setting expiration: data/model/oci/tag.py:673 - ImmutableTagException(tag.name, "set expiration on", ...)

      Issue 3: No Actionable Guidance

      Problem: Error doesn't tell users what they can do

      Missing Information:

      • Suggestion to use a different tag name
      • How to check immutability policies
      • Who can modify policies (admin/organization owner)

      Issue 4: No Policy Information

      Problem: Doesn't indicate which immutability policy is blocking the operation

      Missing Context:

      • Policy pattern that matched (e.g., ^ubuntu$)
      • Policy scope (namespace-level vs repository-level)
      • Policy name/ID for reference

      Issue 5: "unknown:" Prefix in Client Error

      Problem: Podman/Docker client doesn't recognize "TAG_IMMUTABLE" error code and displays it as "unknown:"

      Impact: Confusing error prefix suggesting the error type is unknown

      Expected Behavior

      Improved Error Message Examples:

      Harbor (current best practice):

      artifact ubuntu:latest is immutable, cannot be deleted
      

      Ideal Quay Error (with all improvements):

      TAG_IMMUTABLE: Cannot overwrite tag 'ubuntu' - tag matches immutability policy pattern '^ubuntu$' at repository level. Use a different tag name or contact your administrator to modify the immutability policy.
      

      Recommended Fixes

      Fix 1: Include Tag Name and Operation in Main Message

      # File: endpoints/v2/errors.py (lines 124-128)
      class TagImmutable(V2RegistryException):
          def __init__(self, tag_name=None, operation="modify", detail=None):
              if tag_name:
                  message = f"Cannot {operation} tag '{tag_name}' - tag is immutable"
              else:
                  message = "tag is immutable and cannot be overwritten"
              
              super(TagImmutable, self).__init__(
                  "TAG_IMMUTABLE", message, detail, 409
              )
      

      Fix 2: Pass Operation from ImmutableTagException

      # File: endpoints/v2/manifest.py (lines 529-530)
      except ImmutableTagException as ite:
          raise TagImmutable(
              tag_name=ite.tag_name,
              operation=ite.operation,
              detail={
                  "message": f"tag '{ite.tag_name}' is immutable",
                  "operation": ite.operation,
              }
          ) from ite
      

      Fix 3: Add Policy Information to Detail (Advanced)

      except ImmutableTagException as ite:
          policy_info = get_matching_immutability_policy(ite.repository_id, ite.tag_name)
          
          raise TagImmutable(
              tag_name=ite.tag_name,
              operation=ite.operation,
              detail={
                  "message": f"tag '{ite.tag_name}' is immutable",
                  "operation": ite.operation,
                  "policy_pattern": policy_info.get("pattern"),
                  "policy_scope": policy_info.get("scope"),
                  "suggestion": "Use a different tag name or contact your administrator to modify the immutability policy"
              }
          ) from ite
      

      Impact

      User Experience:

      • Confusing error messages for all users working with immutability policies
      • Difficult to troubleshoot which policy is blocking operations
      • No guidance on how to resolve the issue
      • Poor discoverability of immutability feature behavior

      Affected Users:

      • All users with namespace-level or repository-level immutability policies
      • Enterprise customers using immutability for compliance (common use case)
      • CI/CD pipelines that may unknowingly push to immutable tags

      Severity: Medium

      • Functional error detection works correctly (operations properly blocked)
      • Error message quality significantly impacts usability
      • Affects common enterprise feature (immutability policies)
      • No data loss or security issues

      Reproduction Steps

      1. Create repository with immutability policy (e.g., pattern: ^ubuntu$)
      2. Attempt to push image with immutable tag:
        podman push quay.example.com/org/repo:ubuntu --tls-verify=false --creds user:pass
        
      1. Observe: Generic error message:
        Error: writing manifest: uploading manifest ubuntu to ...: unknown: tag is immutable and cannot be overwritten
        
      1. Expected: Specific error with tag name, operation, and policy info

      Testing Requirements

      1. Unit test: Verify error message includes tag name and operation
      2. Integration test: Test all operations (push, delete, set expiration) return appropriate messages
      3. Regression test: Ensure existing error handling still works
      4. Manual test: Verify error message displayed correctly in podman/docker/skopeo clients

      Related Features

      • Immutability policies (namespace and repository level)
      • Docker Registry v2 API error handling
      • Tag lifecycle operations (create, delete, expire)

      Comparison with Other Registries

      Harbor:

      artifact ubuntu:latest is immutable, cannot be deleted
      

      ✓ Includes tag name
      ✓ Includes operation
      ✗ No policy information

      Docker Hub:

      denied: requested access to the resource is denied
      

      ✗ Generic error (worse than Quay)

      Quay can lead the industry with comprehensive error messages that include policy context.

              Unassigned Unassigned
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: