Problem

Quay's CloudFront storage backend (CloudFrontedS3Storage) uses botocore's CloudFrontSigner which only supports legacy Trusted Signers method. It does not support modern Trusted Key Groups, which AWS recommends and actively promotes.

Current Implementation

File: storage/cloud.py
Class: CloudFrontedS3Storage (lines 1109-1233)
Signing Method: botocore.signers.CloudFrontSigner (line 1180)

@lru_cache(maxsize=1)
def _get_cloudfront_signer(self):
    return CloudFrontSigner(self.cloudfront_key_id, self._get_rsa_signer())

This implementation:

• Uses Key-Pair-Id parameter in signed URLs (legacy format)
• Requires CloudFront Key Pairs uploaded to AWS root account
• Only works with Trusted Signers CloudFront configuration

Limitations of Legacy Trusted Signers

AWS Limitations:

• Maximum 2 active key pairs per AWS account (global limit)
• Key pairs can ONLY be created by root account user (not IAM users)
• Cannot be created via AWS CLI/API
• Deprecated by AWS in favor of Trusted Key Groups
• Poor security practice (requires root account access)

Operational Impact:

• Cannot use modern CloudFront security features
• Requires manual root account operations for key rotation
• Blocks adoption of AWS security best practices
• Incompatible with Trusted Key Groups configurations

Modern Trusted Key Groups Benefits

AWS Recommended Method:

• IAM-based key management (no root account required)
• Unlimited public keys per account
• Fine-grained access control via Key Groups
• API/CLI support for key creation and rotation
• Better security and operational practices

Reproduction

1. Configure Quay with CloudFrontedS3Storage
2. Configure CloudFront distribution with Trusted Key Groups (modern method)
3. Upload public key to CloudFront
4. Attempt to pull container image
5. Result: HTTP 403 Forbidden from CloudFront
6. Reason: Quay generates signed URLs with Key-Pair-Id (legacy format) incompatible with Trusted Key Groups

CloudFront Error:

HTTP/2 403
server: CloudFront
x-cache: Error from cloudfront

Affected Configuration

DISTRIBUTED_STORAGE_CONFIG:   default:     - CloudFrontedS3Storage
    - cloudfront_distribution_domain: d3elkcgjmcsm5x.cloudfront.net
      cloudfront_key_id: K93772GV0BD4A
      cloudfront_privatekey_filename: /conf/stack/quay_private_key.pem

CloudFront Distribution:

• Behaviors → Restrict Viewer Access: Yes
• Trusted Key Groups: Enabled
• Trusted Signers: Disabled (modern configuration)

Root Cause

botocore.signers.CloudFrontSigner only supports legacy Trusted Signers method and cannot generate Trusted Key Groups compatible signatures.

Recommended Solution

Implement CloudFront Trusted Key Groups support using AWS CRT SDK or manual RSA signing with proper Key Groups format.

Workarounds

Short-term:
1. Switch to direct S3 presigned URLs (disable CloudFront)
2. Use legacy CloudFront Key Pairs (requires root account, max 2 keys)

Impact

Severity: Medium
Affected Users: Customers using CloudFront with Trusted Key Groups
User Impact: Cannot use modern CloudFront configurations, forced to use deprecated AWS features

References

• AWS Documentation: Trusted Key Groups vs Trusted Signers
• AWS Recommendation: Migrate from Trusted Signers to Trusted Key Groups

Discovery Context

Discovered: 2026-03-09
Environment: OCP Quay 3.17 with CloudFront storage
Root Cause Analysis: CloudFront 403 errors on image pulls