-
Bug
-
Resolution: Unresolved
-
Major
-
quay-v3.17.0
-
False
-
-
False
-
-
Overview
When pushing an image with LABEL quay.immutable=true in the Dockerfile or --label "quay.immutable=true" during build, the label is stored in the database and visible in the UI, but the tag does NOT become immutable. The immutability handler function exists and is properly registered but is never invoked during the manifest push flow.
Expected Behavior:
- quay.immutable=true manifest label support during push
Discovery: User report and code review (2026-03-04)
Severity: MAJOR (Feature Broken)
Impact: quay.immutable label feature completely non-functional for container push workflow
Issue Details
User Command (Doesn't Work)
# Build with immutable label podman build -t my-image:latest --label "quay.immutable=true" -f ./Dockerfile . # Or in Dockerfile: # FROM ubi9 # LABEL quay.immutable="true" # ... # Push to Quay podman push quay.io/myorg/demo:latest
Expected Behavior
1. Manifest created with quay.immutable=true label
2. Label stored in database
3. Label handler invoked to set tag immutability
4. Tag marked as immutable=true in database
5. Tag shows as immutable (locked) in UI
6. Tag cannot be overwritten or deleted
Actual Behavior
1. Manifest created with quay.immutable=true label ✓
2. Label stored in manifestlabel table ✓
3. Label handler NEVER invoked ✗
4. Tag created with immutable=false (default) ✗
5. Tag shows as mutable in UI ✗
6. Tag can be overwritten and deleted ✗
Database Evidence
After push with quay.immutable=true label:
-- Label exists in database ✓ SELECT ml.key, ml.value, m.digest FROM manifestlabel ml JOIN manifest m ON ml.manifest_id = m.id WHERE ml.key = 'quay.immutable'; -- Returns: quay.immutable | true | sha256:abc123... -- But tag is NOT immutable ✗ SELECT t.name, t.immutable, r.name as repo FROM tag t JOIN repository r ON t.repository_id = r.id WHERE t.name = 'latest' AND r.name = 'demo'; -- Returns: latest | f (false) | demo
Label is stored but tag immutability is not set.
Root Cause Analysis
Label Creation Without Handler Invocation
File: data/model/oci/manifest.py:430-437
During manifest creation, labels from the container image are extracted and stored:
for key, value in labels.items(): # NOTE: There can technically be empty label keys via Dockerfile's. if not key: continue media_type = "application/json" if is_json(value) else "text/plain" create_manifest_label(manifest, key, value, "manifest", media_type) # Line 437 # ✗ BUG: Handler is NOT called here!
The create_manifest_label() function only writes the label to the database. It does NOT invoke the label handler from label_handlers.py.
Handler Function Exists But Is Unused
File: data/registry_model/label_handlers.py:28-38
The immutable label handler is properly implemented and registered:
LABEL_IMMUTABLE_KEY = "quay.immutable" def _immutable(label_dict, manifest, model): """Sets immutability on manifest tags based on the quay.immutable label.""" if not features.IMMUTABLE_TAGS: return value = label_dict.get("value", "").strip().lower() if value == "true": logger.debug("Labeling manifest %s as immutable", manifest) model.set_tags_immutability_for_manifest(manifest, True) # ✗ NEVER EXECUTED _LABEL_HANDLERS = { LABEL_EXPIRY_KEY: _expires_after, LABEL_IMMUTABLE_KEY: _immutable, # Registered but never called }
This function exists, is registered, but is never invoked during podman push.
Where Handler IS Called (Not Used During Push)
File: data/registry_model/registry_oci_model.py:315-344
There is a context manager that properly calls handlers:
def batch_create_manifest_labels(self, manifest): labels_to_add = [] def add_label(key, value, source_type_name, media_type_name=None): labels_to_add.append(...) yield add_label # Process all labels for label_data in labels_to_add: with db_transaction(): # Create the label itself oci.label.create_manifest_label(manifest._db_id, **label_data) # ✓ Handler IS called here (line 344) apply_label_to_manifest(label_data, manifest, self)
Problem: This batch_create_manifest_labels() context manager is NEVER USED in the manifest push endpoint (endpoints/v2/manifest.py).
Why This Bug Exists - Historical Context
File: data/registry_model/registry_oci_model.py:502-506
Code comment explains the design decision:
# NOTE: Since there is currently only one special label on a manifest that has # an effect on its tags (expiration), it is just simpler to set that value at # tag creation time (plus it saves an additional query). # If we were to define more of these "special" labels in the future, we should # use the handlers from data/registry_model/label_handlers.py
Timeline:
1. Initially only quay.expires-after label existed
2. Expiration was hard-coded into tag creation (works correctly)
3. Comment states future labels should use handler system
4. quay.immutable feature added later (PROJQUAY-10157)
5. Handler created and registered in label_handlers.py
6. But handler was never integrated into manifest push flow ✗
Comparison: Expiration Works, Immutability Doesn't
File: data/registry_model/registry_oci_model.py:502-527
# ✓ Expiration label - SPECIAL HANDLING (WORKS) if not created_manifest.newly_created: label_dict = self._get_expiry_label_for_manifest(wrapped_manifest) else: # Extract quay.expires-after from labels_to_apply label_dict = next( (dict(key=label_key, value=label_value) for label_key, label_value in created_manifest.labels_to_apply.items() if label_key == LABEL_EXPIRY_KEY), None, ) expiration_seconds = None if label_dict is not None: expiration_td = convert_to_timedelta(label_dict["value"]) expiration_seconds = expiration_td.total_seconds() # Expiration applied during tag creation tag = oci.tag.retarget_tag( tag_name, created_manifest.manifest, expiration_seconds=expiration_seconds # ✓ Works )
quay.expires-after works because it has special hard-coded handling.
quay.immutable doesn't work because it has NO special handling and the handler system is not used.
Impact Assessment
Severity: MAJOR
Feature Impact
- quay.immutable manifest label feature is completely non-functional
- Users building images with LABEL quay.immutable=true get no protection
- CI/CD pipelines relying on this feature are not protected
- Documentation states this feature works, but it doesn't
Affected Workflows
1. Dockerfile builds:
Unable to find source-code formatter for language: dockerfile. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yamlFROM registry.access.redhat.com/ubi9/ubi-minimal:latest LABEL quay.immutable="true" COPY app /app CMD ["/app/start"]
2. Podman/Docker builds:
podman build --label quay.immutable=true -t quay.io/myorg/app:v1.0 . docker build --label quay.immutable=true -t quay.io/myorg/app:v1.0 .
3. Multi-stage builds:
Unable to find source-code formatter for language: dockerfile. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yamlFROM golang:1.21 AS builder WORKDIR /app COPY . . RUN go build -o myapp FROM scratch LABEL quay.immutable="true" COPY --from=builder /app/myapp / CMD ["/myapp"]
All of these workflows are affected - label is stored but tag remains mutable.
User Impact
- Users expect tags to be immutable but they are not
- Production tags can be accidentally overwritten
- Release tags (v1.0.0, v2.0.0) not protected as intended
- Silent failure - no error message, appears to work
Documentation Gap
Quay documentation likely states quay.immutable label works for container builds, creating false expectations.
Affected Code Files
Primary:
- data/model/oci/manifest.py:430-437 - Label creation without handler
- data/registry_model/registry_oci_model.py:502-549 - Manifest and tag creation
- data/registry_model/label_handlers.py:28-38 - Unused handler function
Related:
- endpoints/v2/manifest.py:308-311 - Push endpoint (calls create_manifest_and_retarget_tag)
- data/model/oci/tag.py:633-646 - set_tags_immutability_for_manifest() function
Recommended Fixes
Option 1: Add Special Handling Like Expiration (Easiest)
File: data/registry_model/registry_oci_model.py:527+
Add immutability handling after expiration handling:
# After line 527 (after expiration handling), add: # Handle quay.immutable label if features.IMMUTABLE_TAGS and created_manifest.newly_created: from data.registry_model.label_handlers import LABEL_IMMUTABLE_KEY immutable_label = created_manifest.labels_to_apply.get(LABEL_IMMUTABLE_KEY) if immutable_label and immutable_label.strip().lower() == "true": logger.debug("Setting tags for manifest %s as immutable from label", created_manifest.manifest.digest) # Re-target the tag to it tag = oci.tag.retarget_tag(...) # NEW: Apply immutability AFTER tag creation if features.IMMUTABLE_TAGS and created_manifest.newly_created: immutable_label = created_manifest.labels_to_apply.get(LABEL_IMMUTABLE_KEY) if immutable_label and immutable_label.strip().lower() == "true": oci.tag.set_tags_immutability_for_manifest( created_manifest.manifest.id, True )
Pros:
- Minimal code change (10 lines)
- Follows existing pattern (same as expiration)
- Low risk
- Easy to test
Cons:
- Continues technical debt of hard-coded special labels
- Doesn't use the handler system as originally intended
Option 2: Use Handler System Properly (Correct But Complex)
File: data/model/oci/manifest.py:437+
Call handler during label creation:
from data.registry_model.label_handlers import apply_label_to_manifest for key, value in labels.items(): if not key: continue media_type = "application/json" if is_json(value) else "text/plain" create_manifest_label(manifest, key, value, "manifest", media_type) # NEW: Apply label handler if one exists label_dict = {"key": key, "value": value} # Need to pass wrapped manifest and registry model instance # This requires refactoring to make these available in this context apply_label_to_manifest(label_dict, wrapped_manifest, registry_model)
Pros:
- Uses handler system as designed
- Extensible for future special labels
- Aligns with code comment's intent
Cons:
- Requires refactoring to pass manifest wrapper and registry model
- More complex change
- Higher risk
- Requires careful testing
Option 3: Hybrid Approach (Recommended)
Phase 1 (Quick Fix):
- Use Option 1 to fix immutability immediately
- Add to backlog: refactor to use handler system properly
Phase 2 (Technical Debt):
- Refactor manifest creation to use batch_create_manifest_labels()
- Remove hard-coded expiration and immutability handling
- Use handler system for all special labels
Testing Requirements
Integration Test (Must Add)
def test_immutable_label_from_manifest(initialized_db): # Create repository repository = create_repository('devtable', 'immutabletest', None) # Create manifest with quay.immutable label manifest_json = {...} # Manifest with Config containing Labels manifest = parse_manifest_from_bytes(Bytes.for_string_or_unicode(manifest_json), ...) # Push manifest created_manifest = get_or_create_manifest(repository, manifest, storage) # Create tag tag = retarget_tag('v1.0', created_manifest.manifest, ...) # Verify label was stored labels = list_manifest_labels(created_manifest.manifest) assert any(l.key == 'quay.immutable' and l.value == 'true' for l in labels) # ✓ CRITICAL: Verify tag is immutable assert tag.immutable == True # This should pass but currently FAILS # Verify tag cannot be overwritten with pytest.raises(ImmutableTagException): retarget_tag('v1.0', another_manifest, ...)
End-to-End Test
def test_podman_push_with_immutable_label(client, app): # Build image with quay.immutable label # Push to Quay # Verify tag is immutable via API response = client.get( '/api/v1/repository/devtable/immutabletest/tag/?specificTag=v1.0' ) assert response.json['tags'][0]['immutable'] == True
Regression Tests
- Verify quay.expires-after still works
- Verify manual label creation still works
- Verify API-based immutability still works
- Verify immutability policies still work
Workarounds
Until fixed, users must use API to set immutability after push:
Workaround 1: API Call After Push
# 1. Push image (label stored but tag not immutable) podman push quay.io/myorg/demo:v1.0 # 2. Set immutability via API TOKEN="your-oauth-token" curl -X PUT \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" \ -d '{"immutable": true}' \ "https://quay.io/api/v1/repository/myorg/demo/tag/v1.0"
Workaround 2: Immutability Policy
Create organization or repository policy instead of using label:
# Create policy to make all release tags immutable curl -X POST \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" \ -d '{"tagPattern": "^v[0-9]+\\.[0-9]+\\.[0-9]+$", "tagPatternMatches": true}' \ "https://quay.io/api/v1/organization/myorg/immutabilitypolicy/"
Workaround 3: CI/CD Automation
# .github/workflows/release.yml - name: Build and push run: | podman build -t quay.io/myorg/app:${VERSION} . podman push quay.io/myorg/app:${VERSION} - name: Make tag immutable run: | curl -X PUT \ -H "Authorization: Bearer ${{ secrets.QUAY_TOKEN }}" \ -H "Content-Type: application/json" \ -d '{"immutable": true}' \ "https://quay.io/api/v1/repository/myorg/app/tag/${VERSION}"
Verification Steps
After fix is deployed:
1. Build image with label:
podman build -t quay.io/testorg/testapp:v1.0 \ --label "quay.immutable=true" \ -f - <<EOF FROM ubi9 LABEL quay.immutable="true" CMD ["/bin/bash"] EOF
2. Push to Quay:
podman push quay.io/testorg/testapp:v1.0
3. Verify tag is immutable:
curl -H "Authorization: Bearer ${TOKEN}" \ "https://quay.io/api/v1/repository/testorg/testapp/tag/?specificTag=v1.0" \ | jq '.tags[0].immutable' # Expected: true ✓
4. Verify tag cannot be overwritten:
podman tag alpine:latest quay.io/testorg/testapp:v1.0
podman push quay.io/testorg/testapp:v1.0
# Expected: TAG_IMMUTABLE error ✓
5. Check database:
SELECT t.name, t.immutable, ml.key, ml.value FROM tag t JOIN manifest m ON t.manifest_id = m.id JOIN manifestlabel ml ON ml.manifest_id = m.id WHERE t.name = 'v1.0' AND ml.key = 'quay.immutable'; -- Expected: v1.0 | true | quay.immutable | true ✓
Related Issues
- PROJQUAY-10157 - Immutable Tag Support (Epic)
- PROJQUAY-10500 - Label error handling (fixed)
- PROJQUAY-10826 - Immutability policy rollback issues
References
- Code Review Date: 2026-03-04
- User Report: Quay 3.17 deployment
- Affected Versions: All versions with FEATURE_IMMUTABLE_TAGS (3.17+)
- Documentation: https://docs.quay.io/use_quay.html#immutable-tags
Reported by: User testing (lzha1981)
Analysis by: Claude Code
