-
Bug
-
Resolution: Done
-
Major
-
None
-
quay-v3.4.0
Description:
This is an issue found when re-save configurations with quay-config app on Azure, after config Quay with correct Azure blob storage configurations via config-app, save config and rollout deployment. Re-open quay-config app, modify some config not related to storage, like time machine settings, Click Save configurations, get error " Invalid storage configuration: azure_us_west: Exception when trying to put path", checked quay pod logs, found error message azure.core.exceptions.ResourceExistsError: The specified blob already exists , that means during validation, quay-config app tried to create new azure storage path, that's not correct.
Note: this issue can't be reproduced on AWS
Image: quay.io/projectquay/quay:0c65f88a
Steps:
- Deploy Quay with Operator by providing correct Azure Blob storage configurations
- Open Quay-config app
- Modify Time machine setting, like add "Allowed expiration periods"-3600s
- Click "Save configurations"
Expected Results:
Quay-config app can save configurations successfully.
Actual Results:
Quay-config app was failed to save configurations with error message "Invalid storage configuration: azure_us_west: Exception when trying to put path"
Quay Pod Logs:
For detailed Quay POD logs, see attached logs.
gunicorn-config stdout | 2020-08-21 06:30:18,908 [38] [ERROR] [storage.azurestorage] Exception when trying to put path _verify gunicorn-config stdout | Traceback (most recent call last): gunicorn-config stdout | File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_upload_helpers.py", line 106, in upload_block_blob gunicorn-config stdout | **kwargs) gunicorn-config stdout | File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_generated/operations/_block_blob_operations.py", line 222, in upload gunicorn-config stdout | raise models.StorageErrorException(response, self._deserialize) gunicorn-config stdout | azure.storage.blob._generated.models._models_py3.StorageErrorException: Operation returned an invalid status 'The specified blob already exists.' gunicorn-config stdout | During handling of the above exception, another exception occurred: gunicorn-config stdout | Traceback (most recent call last): gunicorn-config stdout | File "/quay-registry/storage/azurestorage.py", line 144, in put_content gunicorn-config stdout | self._blob(blob_name).upload_blob(content, blob_type=BlobType.BlockBlob) gunicorn-config stdout | File "/usr/local/lib/python3.6/site-packages/azure/core/tracing/decorator.py", line 83, in wrapper_use_tracer gunicorn-config stdout | return func(*args, **kwargs) gunicorn-config stdout | File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_blob_client.py", line 527, in upload_blob gunicorn-config stdout | return upload_block_blob(**options) gunicorn-config stdout | File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_upload_helpers.py", line 156, in upload_block_blob gunicorn-config stdout | process_storage_error(error) gunicorn-config stdout | File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_shared/response_handlers.py", line 147, in process_storage_error gunicorn-config stdout | raise error gunicorn-config stdout | azure.core.exceptions.ResourceExistsError: The specified blob already exists. gunicorn-config stdout | RequestId:755cc64e-901e-0033-5c84-777a46000000 gunicorn-config stdout | Time:2020-08-21T06:30:18.8748379Z gunicorn-config stdout | ErrorCode:BlobAlreadyExists gunicorn-config stdout | Error:None gunicorn-config stdout | 2020-08-21 06:30:18,909 [38] [ERROR] [util.config.validator] Validation exception gunicorn-config stdout | Traceback (most recent call last): gunicorn-config stdout | File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_upload_helpers.py", line 106, in upload_block_blob gunicorn-config stdout | **kwargs) gunicorn-config stdout | File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_generated/operations/_block_blob_operations.py", line 222, in upload gunicorn-config stdout | raise models.StorageErrorException(response, self._deserialize) gunicorn-config stdout | azure.storage.blob._generated.models._models_py3.StorageErrorException: Operation returned an invalid status 'The specified blob already exists.' gunicorn-config stdout | During handling of the above exception, another exception occurred: gunicorn-config stdout | Traceback (most recent call last): gunicorn-config stdout | File "/quay-registry/storage/azurestorage.py", line 144, in put_content gunicorn-config stdout | self._blob(blob_name).upload_blob(content, blob_type=BlobType.BlockBlob) gunicorn-config stdout | File "/usr/local/lib/python3.6/site-packages/azure/core/tracing/decorator.py", line 83, in wrapper_use_tracer gunicorn-config stdout | return func(*args, **kwargs) gunicorn-config stdout | File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_blob_client.py", line 527, in upload_blob gunicorn-config stdout | return upload_block_blob(**options) gunicorn-config stdout | File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_upload_helpers.py", line 156, in upload_block_blob gunicorn-config stdout | process_storage_error(error) gunicorn-config stdout | File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/_shared/response_handlers.py", line 147, in process_storage_error gunicorn-config stdout | raise error gunicorn-config stdout | azure.core.exceptions.ResourceExistsError: The specified blob already exists. gunicorn-config stdout | RequestId:755cc64e-901e-0033-5c84-777a46000000 gunicorn-config stdout | Time:2020-08-21T06:30:18.8748379Z gunicorn-config stdout | ErrorCode:BlobAlreadyExists gunicorn-config stdout | Error:None gunicorn-config stdout | During handling of the above exception, another exception occurred: gunicorn-config stdout | Traceback (most recent call last): gunicorn-config stdout | File "/quay-registry/util/config/validators/validate_storage.py", line 37, in validate gunicorn-config stdout | driver.validate(client) gunicorn-config stdout | File "/quay-registry/storage/azurestorage.py", line 129, in validate gunicorn-config stdout | super(AzureStorage, self).validate(client) gunicorn-config stdout | File "/quay-registry/storage/basestorage.py", line 54, in validate gunicorn-config stdout | self.put_content("_verify", b"testing 123") gunicorn-config stdout | File "/quay-registry/storage/azurestorage.py", line 147, in put_content gunicorn-config stdout | raise IOError("Exception when trying to put path") gunicorn-config stdout | OSError: Exception when trying to put path gunicorn-config stdout | During handling of the above exception, another exception occurred: gunicorn-config stdout | Traceback (most recent call last): gunicorn-config stdout | File "/quay-registry/util/config/validator.py", line 83, in validate_service_for_config gunicorn-config stdout | VALIDATORS[service](validator_context) gunicorn-config stdout | File "/quay-registry/util/config/validators/validate_storage.py", line 44, in validate gunicorn-config stdout | "Invalid storage configuration: %s: %s" % (name, msg) gunicorn-config stdout | util.config.validators.ConfigValidationException: Invalid storage configuration: azure_us_west: Exception when trying to put path nginx stdout | 10.129.2.4 () - quayconfig [21/Aug/2020:06:30:18 +0000] "POST /api/v1/superuser/config/validate/registry-storage HTTP/2.0" 200 111 "https://demo-quayecosystem-quay-config-quay-enterprise.apps.quay-317.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Firefox/68.0" (0.567 3161 0.567)
