-
Bug
-
Resolution: Done
-
Blocker
-
quay-v3.7.0
-
None
Description:
This is an issue found when deploy quay with config(DEFAULT_SYSTEM_REJECT_QUOTA_BYTES: 9024000000), it will set the system default quota to 8GB, after complete quay deployment with quay operator, create new image repo user user account namespace, push image(500MB size) to this image repo, it was failed with error " Quota has been exceeded on namespace", but here the default quota is 8GB, Quay should not report error "Quota exceeded".
Note: this issue can also be reproduced when target Organization has enough quota size.
Quay Image: quay-operator-bundle-container-v3.7.0-123
[root@ip-10-0-1-88 fedora]# podman push quay370.apps.quayperf370.perfscale.devcluster.openshift.com/quay/demo --tls-verify=false --creds quay:password Getting image source signatures Copying blob a037458de4e0 [--------------------------------------] 8.0b / 18.4MiB Copying blob bafdbe68e4ae [--------------------------------------] 8.0b / 10.8MiB Copying blob 6d95196cbe50 [--------------------------------------] 8.0b / 18.2MiB Copying blob 3a0bf2cde803 [--------------------------------------] 8.0b / 4.5KiB Copying blob 45618b844972 [--------------------------------------] 8.0b / 55.1MiB Copying blob a13c519c6361 [--------------------------------------] 8.0b / 123.1MiB Copying blob 709cf5ff5dbe [--------------------------------------] 8.0b / 10.3MiB Copying blob 08fa02ce37eb [--------------------------------------] 8.0b / 149.3MiB Copying blob 5bdcc8e2060c [--------------------------------------] 8.0b / 513.1MiB WARN[0003] failed, retrying in 1s ... (1/3). Error: writing blob: initiating layer upload to /v2/quay/demo/blobs/uploads/ in quay370.apps.quayperf370.perfscale.devcluster.openshift.com: denied: Quota has been exceeded on namespace Getting image source signatures Copying blob a037458de4e0 [--------------------------------------] 8.0b / 18.4MiB Copying blob 6d95196cbe50 [--------------------------------------] 8.0b / 18.2MiB Copying blob bafdbe68e4ae [--------------------------------------] 8.0b / 10.8MiB Copying blob 3a0bf2cde803 [--------------------------------------] 8.0b / 4.5KiB Copying blob 709cf5ff5dbe [--------------------------------------] 8.0b / 10.3MiB Copying blob 45618b844972 [--------------------------------------] 8.0b / 55.1MiB Copying blob a13c519c6361 [--------------------------------------] 8.0b / 123.1MiB Copying blob 08fa02ce37eb [--------------------------------------] 8.0b / 149.3MiB Copying blob 5bdcc8e2060c [--------------------------------------] 8.0b / 513.1MiB WARN[0008] failed, retrying in 1s ... (2/3). Error: writing blob: initiating layer upload to /v2/quay/demo/blobs/uploads/ in quay370.apps.quayperf370.perfscale.devcluster.openshift.com: denied: Quota has been exceeded on namespace Getting image source signatures Copying blob 6d95196cbe50 [--------------------------------------] 8.0b / 18.2MiB Copying blob a037458de4e0 [--------------------------------------] 8.0b / 18.4MiB Copying blob bafdbe68e4ae [--------------------------------------] 8.0b / 10.8MiB Copying blob 3a0bf2cde803 [--------------------------------------] 8.0b / 4.5KiB Copying blob 709cf5ff5dbe [--------------------------------------] 8.0b / 10.3MiB Copying blob 45618b844972 [--------------------------------------] 8.0b / 55.1MiB Copying blob a13c519c6361 [--------------------------------------] 8.0b / 123.1MiB Copying blob 08fa02ce37eb [--------------------------------------] 8.0b / 149.3MiB Copying blob 5bdcc8e2060c [--------------------------------------] 8.0b / 513.1MiB WARN[0013] failed, retrying in 1s ... (3/3). Error: writing blob: initiating layer upload to /v2/quay/demo/blobs/uploads/ in quay370.apps.quayperf370.perfscale.devcluster.openshift.com: denied: Quota has been exceeded on namespace Getting image source signatures Copying blob a037458de4e0 [--------------------------------------] 8.0b / 18.4MiB Copying blob 6d95196cbe50 [--------------------------------------] 8.0b / 18.2MiB Copying blob bafdbe68e4ae [--------------------------------------] 8.0b / 10.8MiB Copying blob 3a0bf2cde803 [--------------------------------------] 8.0b / 4.5KiB Copying blob a13c519c6361 [--------------------------------------] 8.0b / 123.1MiB Copying blob 709cf5ff5dbe [--------------------------------------] 8.0b / 10.3MiB Copying blob 45618b844972 [--------------------------------------] 8.0b / 55.1MiB Copying blob 08fa02ce37eb [--------------------------------------] 8.0b / 149.3MiB Copying blob 5bdcc8e2060c [--------------------------------------] 8.0b / 513.1MiB Error: writing blob: initiating layer upload to /v2/quay/demo/blobs/uploads/ in quay370.apps.quayperf370.perfscale.devcluster.openshift.com: denied: Quota has been exceeded on namespace [root@ip-10-0-1-88 fedora]# podman push demo370-quay-quay370.apps.quayperf370.perfscale.devcluster.openshift.com/quotatest/demo4 --tls-verify=false --creds test:password Getting image source signatures Copying blob bafdbe68e4ae [--------------------------------------] 8.0b / 10.8MiB Copying blob a037458de4e0 [--------------------------------------] 8.0b / 18.4MiB Copying blob 6d95196cbe50 [--------------------------------------] 8.0b / 18.2MiB Copying blob 3a0bf2cde803 [--------------------------------------] 8.0b / 4.5KiB Copying blob 709cf5ff5dbe [--------------------------------------] 8.0b / 10.3MiB Copying blob 45618b844972 [--------------------------------------] 8.0b / 55.1MiB Copying blob a13c519c6361 [--------------------------------------] 8.0b / 123.1MiB Copying blob 08fa02ce37eb [--------------------------------------] 8.0b / 149.3MiB Copying blob 5bdcc8e2060c [--------------------------------------] 8.0b / 513.1MiB WARN[0003] failed, retrying in 1s ... (1/3). Error: writing blob: initiating layer upload to /v2/quotatest/demo4/blobs/uploads/ in demo370-quay-quay370.apps.quayperf370.perfscale.devcluster.openshift.com: denied: Quota has been exceeded on namespace Getting image source signatures Copying blob a037458de4e0 [--------------------------------------] 8.0b / 18.4MiB Copying blob 6d95196cbe50 [--------------------------------------] 8.0b / 18.2MiB Copying blob bafdbe68e4ae [--------------------------------------] 8.0b / 10.8MiB Copying blob 3a0bf2cde803 [--------------------------------------] 8.0b / 4.5KiB Copying blob 45618b844972 [--------------------------------------] 8.0b / 55.1MiB Copying blob 709cf5ff5dbe [--------------------------------------] 8.0b / 10.3MiB Copying blob a13c519c6361 [--------------------------------------] 8.0b / 123.1MiB Copying blob 08fa02ce37eb [--------------------------------------] 8.0b / 149.3MiB Copying blob 5bdcc8e2060c [--------------------------------------] 8.0b / 513.1MiB WARN[0007] failed, retrying in 1s ... (2/3). Error: writing blob: initiating layer upload to /v2/quotatest/demo4/blobs/uploads/ in demo370-quay-quay370.apps.quayperf370.perfscale.devcluster.openshift.com: denied: Quota has been exceeded on namespace Getting image source signatures Copying blob bafdbe68e4ae [--------------------------------------] 8.0b / 10.8MiB Copying blob a037458de4e0 [--------------------------------------] 8.0b / 18.4MiB Copying blob 6d95196cbe50 [--------------------------------------] 8.0b / 18.2MiB Copying blob 3a0bf2cde803 [--------------------------------------] 8.0b / 4.5KiB Copying blob 709cf5ff5dbe [--------------------------------------] 8.0b / 10.3MiB Copying blob 45618b844972 [--------------------------------------] 8.0b / 55.1MiB Copying blob 08fa02ce37eb [--------------------------------------] 8.0b / 149.3MiB Copying blob a13c519c6361 [--------------------------------------] 8.0b / 123.1MiB Copying blob 5bdcc8e2060c [--------------------------------------] 8.0b / 513.1MiB WARN[0011] failed, retrying in 1s ... (3/3). Error: writing blob: initiating layer upload to /v2/quotatest/demo4/blobs/uploads/ in demo370-quay-quay370.apps.quayperf370.perfscale.devcluster.openshift.com: denied: Quota has been exceeded on namespace Getting image source signatures Copying blob a037458de4e0 [--------------------------------------] 8.0b / 18.4MiB Copying blob 6d95196cbe50 [--------------------------------------] 8.0b / 18.2MiB Copying blob bafdbe68e4ae [--------------------------------------] 8.0b / 10.8MiB Copying blob 3a0bf2cde803 [--------------------------------------] 8.0b / 4.5KiB Copying blob 709cf5ff5dbe [--------------------------------------] 8.0b / 10.3MiB Copying blob a13c519c6361 [--------------------------------------] 8.0b / 123.1MiB Copying blob 45618b844972 [--------------------------------------] 8.0b / 55.1MiB Copying blob 08fa02ce37eb [--------------------------------------] 8.0b / 149.3MiB Copying blob 5bdcc8e2060c [--------------------------------------] 8.0b / 513.1MiB Error: writing blob: initiating layer upload to /v2/quotatest/demo4/blobs/uploads/ in demo370-quay-quay370.apps.quayperf370.perfscale.devcluster.openshift.com: denied: Quota has been exceeded on namespace
Default Quota size is 8GB
Quay config.yaml:
DEFAULT_SYSTEM_REJECT_QUOTA_BYTES: 9024000000 CREATE_PRIVATE_REPO_ON_PUSH: true CREATE_NAMESPACE_ON_PUSH: true FEATURE_EXTENDED_REPOSITORY_NAMES: true FEATURE_QUOTA_MANAGEMENT: true FEATURE_PROXY_CACHE: true FEATURE_USER_INITIALIZE: true SERVER_HOSTNAME: quay370.apps.quayperf370.perfscale.devcluster.openshift.com ALLOWED_OCI_ARTIFACT_TYPES: application/vnd.cncf.helm.config.v1+json: - application/tar+gzip application/vnd.oci.image.layer.v1.tar+gzip+encrypted: - application/vnd.oci.image.layer.v1.tar+gzip+encrypted DEFAULT_TAG_EXPIRATION: 4w TAG_EXPIRATION_OPTIONS: - 2w - 4w - 8w FEATURE_GENERAL_OCI_SUPPORT: true FEATURE_HELM_OCI_SUPPORT: true SUPER_USERS: - quay - admin USERFILES_LOCATION: default USERFILES_PATH: userfiles/ DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: - default DISTRIBUTED_STORAGE_PREFERENCE: - default DISTRIBUTED_STORAGE_CONFIG: default: - S3Storage - s3_bucket: quay370 storage_path: /quay370 s3_access_key: *** s3_secret_key: *** host: s3.us-east-2.amazonaws.com
Steps:
- Deploy Quay with Quay Operator, enabled quota management and set "DEFAULT_SYSTEM_REJECT_QUOTA_BYTES: 9024000000
", that means the default quota size is 8GB - Create new image repo user user account namespace
- Push image(500MB) to above new image repo
Expected Results:
Push image completed successfully
Actual Results:
Push image was failed with error "denied: Quota has been exceeded on namespace"
