Details
-
Bug
-
Resolution: Done
-
Normal
-
quay-v3.7.8
Description
We sometimes see KeyError exceptions in customer logs - it seems some storage apis don't consistently send "HTTPStatusCode" in the error dictionary. I have not been able to reproduce this, so I cannot provide steps for doing so.
Regression tests are probably a good way to ensure the fix for this doesn't break anything. Since this is a low risk change this should be enough.
Here's an example exception:
storagereplication stdout | 2022-06-03 20:36:59,292 [112] [ERROR] [__main__] Unknown exception when copying path sha256/1a/1ae3c93430d851c00608483bfe01f13a8f360bd434a90f8b98795c17c47219f3 of image storage 58efc129-f6bf-4824-8244- 350cbe16d212 to loc DRStoarge storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/storage/cloud.py", line 590, in _perform_action_with_retry storagereplication stdout | return action(*args, **kwargs) storagereplication stdout | File "/usr/local/lib/python3.8/site-packages/boto3/resources/factory.py", line 520, in do_action storagereplication stdout | response = action(self, *args, **kwargs) storagereplication stdout | File "/usr/local/lib/python3.8/site-packages/boto3/resources/action.py", line 83, in __call__ storagereplication stdout | response = getattr(parent.meta.client, operation_name)(*args, **params) storagereplication stdout | File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call storagereplication stdout | return self._make_api_call(operation_name, kwargs) storagereplication stdout | File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 676, in _make_api_call storagereplication stdout | raise error_class(parsed_response, operation_name) storagereplication stdout | botocore.exceptions.ClientError: An error occurred (OperationAborted) when calling the CompleteMultipartUpload operation: Object already exists. storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/workers/storagereplication.py", line 124, in replicate_storage storagereplication stdout | storage.copy_between(path_to_copy, existing_location, location) storagereplication stdout | File "/quay-registry/storage/distributedstorage.py", line 100, in copy_between storagereplication stdout | source_storage.copy_to(destination_storage, path) storagereplication stdout | File "/quay-registry/storage/cloud.py", line 508, in copy_to storagereplication stdout | destination.stream_write(path, fp) storagereplication stdout | File "/quay-registry/storage/cloud.py", line 315, in stream_write storagereplication stdout | _, write_error = self._stream_write_internal(path, fp, content_type, content_encoding) storagereplication stdout | File "/quay-registry/storage/cloud.py", line 396, in _stream_write_internal storagereplication stdout | self._perform_action_with_retry( storagereplication stdout | File "/quay-registry/storage/cloud.py", line 595, in _perform_action_with_retry storagereplication stdout | and s3re.response["Error"]["HTTPStatusCode"] == 200 storagereplication stdout | KeyError: 'HTTPStatusCode' storagereplication stdout | 2022-06-03 20:36:59,315 [112] [ERROR] [workers.queueworker] The worker has encountered an error via the job and will not take new jobs storagereplication stdout | 2022-06-03 20:36:59,315 [112] [ERROR] [workers.queueworker]
