Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-540

Azure georeplication is not working

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • quay-v3.3.0
    • None
    • quay
    • None

      When georeplication is attempted between two different geolocations in Azure, Quay appears to copy the file onto itself. This is seen in the Quay log:

      storagereplication stdout | 2020-03-31 09:49:28,843 [191] [DEBUG] [storage.azurestorage] Starting copying file from Azure quayregistry to Azure <storage.azurestorage.AzureStorage object at 0x7f78171a5fd0> via
 an Azure copy
2020-03-31 09:49:28,844 [191] [DEBUG] [azure.storage.common._auth] String_to_sign=PUT
x-ms-client-request-id:e9ee97ba-7334-11ea-a15c-0a580a830015
x-ms-copy-source:https://quaybackendbmwuskq7s7.blob.core.windows.net/quayregistry/datastorage/registry/sha256/bf/bf8192a67ca1d7a9973948a37dec6423f86468992bfeb1461578c587f36ded97?sr=..
x-ms-date:Tue, 31 Mar 2020 09:49:28 GMT
x-ms-version:2017-07-29
/quaybackendbmwuskq7s7/quayregistry/datastorage/registry/sha256/bf/bf8192a67ca1d7a9973948a37dec6423f86468992bfeb1461578c587f36ded97
2020-03-31 09:49:28,844 [191] [INFO] [azure.storage.common.storageclient] Client-Request-ID=e9ee97ba-7334-11ea-a15c-0a580a830015 Outgoing request: Method=PUT, Path=/quayregistry/datastorage/registry/sha256/bf/bf8192a67ca1d7a9973948a37dec6423f86468992bfeb1461578c587f36ded97, Query={'timeout': None}, Headers={'x-ms-version': '2017-07-29', 'x-ms-source-if-unmodified-since': None, 'x-ms-copy-source': 'https://quaybackendbmwuskq7s7.blob.core.windows.net/quayregistry/datastorage/registry/sha256/bf/bf8192a67ca1d7a9973948a37dec6423f86468992bfeb1461578c587f36ded97?sr=b...', 'x-ms-source-if-match': None, 'User-Agent': 'Azure-Storage/1.1.0-1.1.0 (Python CPython 2.7.17; Linux 4.18.0-147.5.1.el8_1.x86_64)', 'If-Unmodified-Since': None, 'x-ms-access-tier': None, 'Authorization': u'SharedKey quaybackendbmwuskq7s7:BK7m...', 'x-ms-source-if-modified-since': None, 'x-ms-client-request-id': 'e9ee97ba-7334-11ea-a15c-0a580a830015', 'x-ms-lease-id': None, 'If-Match': None, 'If-Modified-Since': None, 'If-None-Match': None, 'x-ms-source-lease-id': None, 'x-ms-source-if-none-match': None, 'x-ms-date': 'Tue, 31 Mar 2020 09:49:28 GMT'}.

      According to the log produced, it appears that the source and destination of the copy are the same. As a consequence, verification that the blob is inside the other container fails even though this operation succeeds:

      2020-03-31 09:49:28,876 [191] [DEBUG] [azure.storage.common._auth] String_to_sign=HEAD
x-ms-client-request-id:e9f3787a-7334-11ea-a15c-0a580a830015
x-ms-date:Tue, 31 Mar 2020 09:49:28 GMT
x-ms-version:2017-07-29
/ukquaybackendbmwuskq7s7/quayregistry/datastorage/registry/sha256/bf/bf8192a67ca1d7a9973948a37dec6423f86468992bfeb1461578c587f36ded97
timeout:20
2020-03-31 09:49:28,876 [191] [INFO] [azure.storage.common.storageclient] Client-Request-ID=e9f3787a-7334-11ea-a15c-0a580a830015 Outgoing request: Method=HEAD, Path=/quayregistry/datastorage/registry/sha256/bf/bf8192a67ca1d7a9973948a37dec6423f86468992bfeb1461578c587f36ded97, Query={'snapshot': None, 'timeout': '20'}, Headers={'x-ms-version': '2017-07-29', 'x-ms-client-request-id': 'e9f3787a-7334-11ea-a15c-0a580a830015', 'x-ms-lease-id': None, 'If-Match': None, 'If-Modified-Since': None, 'If-None-Match': None, 'User-Agent': 'Azure-Storage/1.1.0-1.1.0 (Python CPython 2.7.17; Linux 4.18.0-147.5.1.el8_1.x86_64)', 'If-Unmodified-Since': None, 'Authorization': u'SharedKey ukquaybackendbmwuskq7s7:qymz...', 'x-ms-date': 'Tue, 31 Mar 2020 09:49:28 GMT'}.
2020-03-31 09:49:28,959 [191] [DEBUG] [urllib3.connectionpool] https://ukquaybackendbmwuskq7s7.blob.core.windows.net:443 "HEAD /quayregistry/datastorage/registry/sha256/bf/bf8192a67ca1d7a9973948a37dec6423f86468992bfeb1461578c587f36ded97?timeout=20 HTTP/1.1" 404 0
storagereplication stdout | 2020-03-31 09:49:28,959 [191] [DEBUG] [urllib3.connectionpool] https://ukquaybackendbmwuskq7s7.blob.core.windows.net:443 "HEAD /quayregistry/datastorage/registry/sha256/bf/bf8192a67ca1d7a9973948a37dec6423f86468992bfeb1461578c587f36ded97?timeout=20 HTTP/1.1" 404 0
2020-03-31 09:49:28,960 [191] [INFO] [azure.storage.common.storageclient] Client-Request-ID=e9f3787a-7334-11ea-a15c-0a580a830015 Receiving Response: Server-Timestamp=Tue, 31 Mar 2020 09:49:28 GMT, Server-Request-ID=4778e325-501e-0041-1041-078056000000, HTTP Status Code=404, Message=The specified blob does not exist., Headers={'x-ms-version': '2017-07-29', 'x-ms-client-request-id': 'e9f3787a-7334-11ea-a15c-0a580a830015', 'x-ms-lease-id': None, 'If-Match': None, 'If-Modified-Since': None, 'If-None-Match': None, 'User-Agent': 'Azure-Storage/1.1.0-1.1.0 (Python CPython 2.7.17; Linux 4.18.0-147.5.1.el8_1.x86_64)', 'If-Unmodified-Since': None, 'Authorization': u'SharedKey ukquaybackendbmwuskq7s7:qymz9...', 'x-ms-date': 'Tue, 31 Mar 2020 09:49:28 GMT'}.
2020-03-31 09:49:28,960 [191] [INFO] [azure.storage.common.storageclient] Client-Request-ID=e9f3787a-7334-11ea-a15c-0a580a830015 Operation failed: checking if the operation should be retried. Current retry count=0, Server-Timestamp=Tue, 31 Mar 2020 09:49:28 GMT, Server-Request-ID=4778e325-501e-0041-1041-078056000000, HTTP status code=404, Exception=The specified blob does not exist.ErrorCode: BlobNotFound.
2020-03-31 09:49:28,960 [191] [ERROR] [azure.storage.common.storageclient] Client-Request-ID=e9f3787a-7334-11ea-a15c-0a580a830015 Retry policy did not allow for a retry: Server-Timestamp=Tue, 31 Mar 2020 09:49:28 GMT, Server-Request-ID=4778e325-501e-0041-1041-078056000000, HTTP status code=404, Exception=The specified blob does not exist.ErrorCode: BlobNotFound.
2020-03-31 09:49:28,960 [191] [DEBUG] [__main__] Cannot find path `sha256/bf/bf8192a67ca1d7a9973948a37dec6423f86468992bfeb1461578c587f36ded97` in location azure-uk (try #0). Sleeping for 2 seconds

      The result is that the blob is never copied between two different containers. I found another issue with Azure where Quay can't find the 2nd container in the 2nd geolocation at all. I attempted to replicate the issue by creating two separate containers and separate storage accounts under the same resource group, one storage account in the US, one in the UK. Verification of data goes through correctly but when storage replication is invoked, I get the following:

      2020-04-01 10:16:28,478 [195] [DEBUG] [storage.azurestorage] Starting copying file from Azure containerus to Azure <storage.azurestorage.AzureStorage object at 0x7f313fcbd490> via an Azure copy
2020-04-01 10:16:28,479 [195] [DEBUG] [azure.storage.common._auth] String_to_sign=PUT
x-ms-client-request-id:d9b8fbb0-7401-11ea-ac06-0a580a830224
x-ms-copy-source:https://quayus1.blob.core.windows.net/containerus/datastorage/registry/sha256/a1/a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72?sr=b...
x-ms-date:Wed, 01 Apr 2020 10:16:28 GMT
x-ms-version:2017-07-29
/quayus1/containereu/datastorage/registry/sha256/a1/a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72
2020-04-01 10:16:28,479 [195] [INFO] [azure.storage.common.storageclient] Client-Request-ID=d9b8fbb0-7401-11ea-ac06-0a580a830224 Outgoing request: Method=PUT, Path=/containereu/datastorage/registry/sha256/a1/a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72, Query={'timeout': None}, Headers={'x-ms-version': '2017-07-29', 'x-ms-source-if-unmodified-since': None, 'x-ms-copy-source': 'https://quayus1.blob.core.windows.net/containerus/datastorage/registry/sha256/a1/a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72?sr=...', 'x-ms-source-if-match': None, 'User-Agent': 'Azure-Storage/1.1.0-1.1.0 (Python CPython 2.7.17; Linux 4.18.0-147.5.1.el8_1.x86_64)', 'If-Unmodified-Since': None, 'x-ms-access-tier': None, 'Authorization': u'SharedKey quayus1:D/P1zC...', 'x-ms-source-if-modified-since': None, 'x-ms-client-request-id': 'd9b8fbb0-7401-11ea-ac06-0a580a830224', 'x-ms-lease-id': None, 'If-Match': None, 'If-Modified-Since': None, 'If-None-Match': None, 'x-ms-source-lease-id': None, 'x-ms-source-if-none-match': None, 'x-ms-date': 'Wed, 01 Apr 2020 10:16:28 GMT'}.
storagereplication stdout | 2020-04-01 10:16:28,479 [195] [DEBUG] [azure.storage.common._auth] String_to_sign=PUT
x-ms-client-request-id:d9b8fbb0-7401-11ea-ac06-0a580a830224
x-ms-copy-source:https://quayus1.blob.core.windows.net/containerus/datastorage/registry/sha256/a1/a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72?sr=b...
x-ms-date:Wed, 01 Apr 2020 10:16:28 GMT
x-ms-version:2017-07-29
/quayus1/containereu/datastorage/registry/sha256/a1/a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72
2020-04-01 10:16:28,479 [195] [INFO] [azure.storage.common.storageclient] Client-Request-ID=d9b8fbb0-7401-11ea-ac06-0a580a830224 Outgoing request: Method=PUT, Path=/containereu/datastorage/registry/sha256/a1/a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72, Query={'timeout': None}, Headers={'x-ms-version': '2017-07-29', 'x-ms-source-if-unmodified-since': None, 'x-ms-copy-source': 'https://quayus1.blob.core.windows.net/containerus/datastorage/registry/sha256/a1/a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72?sr=b...', 'x-ms-source-if-match': None, 'User-Agent': 'Azure-Storage/1.1.0-1.1.0 (Python CPython 2.7.17; Linux 4.18.0-147.5.1.el8_1.x86_64)', 'If-Unmodified-Since': None, 'x-ms-access-tier': None, 'Authorization': u'SharedKey quayus1:D/P1z...', 'x-ms-source-if-modified-since': None, 'x-ms-client-request-id': 'd9b8fbb0-7401-11ea-ac06-0a580a830224', 'x-ms-lease-id': None, 'If-Match': None, 'If-Modified-Since': None, 'If-None-Match': None, 'x-ms-source-lease-id': None, 'x-ms-source-if-none-match': None, 'x-ms-date': 'Wed, 01 Apr 2020 10:16:28 GMT'}.
2020-04-01 10:16:28,660 [195] [DEBUG] [urllib3.connectionpool] https://quayus1.blob.core.windows.net:443 "PUT /containereu/datastorage/registry/sha256/a1/a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72 HTTP/1.1" 404 225
storagereplication stdout | 2020-04-01 10:16:28,660 [195] [DEBUG] [urllib3.connectionpool] https://quayus1.blob.core.windows.net:443 "PUT /containereu/datastorage/registry/sha256/a1/a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72 HTTP/1.1" 404 225
2020-04-01 10:16:28,661 [195] [INFO] [azure.storage.common.storageclient] Client-Request-ID=d9b8fbb0-7401-11ea-ac06-0a580a830224 Receiving Response: Server-Timestamp=Wed, 01 Apr 2020 10:16:27 GMT, Server-Request-ID=183fb883-301e-00d4-320e-08075a000000, HTTP Status Code=404, Message=The specified container does not exist., Headers={'x-ms-version': '2017-07-29', 'x-ms-source-if-unmodified-since': None, 'x-ms-copy-source': 'https://quayus1.blob.core.windows.net/containerus/datastorage/registry/sha256/a1/a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72?sr=b&...', 'x-ms-source-if-match': None, 'User-Agent': 'Azure-Storage/1.1.0-1.1.0 (Python CPython 2.7.17; Linux 4.18.0-147.5.1.el8_1.x86_64)', 'If-Unmodified-Since': None, 'x-ms-access-tier': None, 'Authorization': u'SharedKey quayus1:D/P1....', 'x-ms-source-if-modified-since': None, 'x-ms-client-request-id': 'd9b8fbb0-7401-11ea-ac06-0a580a830224', 'x-ms-lease-id': None, 'If-Match': None, 'If-Modified-Since': None, 'If-None-Match': None, 'x-ms-source-lease-id': None, 'x-ms-source-if-none-match': None, 'x-ms-date': 'Wed, 01 Apr 2020 10:16:28 GMT'}.
2020-04-01 10:16:28,661 [195] [INFO] [azure.storage.common.storageclient] Client-Request-ID=d9b8fbb0-7401-11ea-ac06-0a580a830224 Operation failed: checking if the operation should be retried. Current retry count=0, Server-Timestamp=Wed, 01 Apr 2020 10:16:27 GMT, Server-Request-ID=183fb883-301e-00d4-320e-08075a000000, HTTP status code=404, Exception=The specified container does not exist.ErrorCode: ContainerNotFound<?xml version="1.0" encoding="utf-8"?><Error><Code>ContainerNotFound</Code><Message>The specified container does not exist.RequestId:183fb883-301e-00d4-320e-08075a000000Time:2020-04-01T10:16:28.6007095Z</Message></Error>.
storagereplication stdout | 2020-04-01 10:16:28,661 [195] [INFO] [azure.storage.common.storageclient] Client-Request-ID=d9b8fbb0-7401-11ea-ac06-0a580a830224 Receiving Response: Server-Timestamp=Wed, 01 Apr 2020 10:16:27 GMT, Server-Request-ID=183fb883-301e-00d4-320e-08075a000000, HTTP Status Code=404, Message=The specified container does not exist., Headers={'x-ms-version': '2017-07-29', 'x-ms-source-if-unmodified-since': None, 'x-ms-copy-source': 'https://quayus1.blob.core.windows.net/containerus/datastorage/registry/sha256/a1/a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72?sr=b&...', 'x-ms-source-if-match': None, 'User-Agent': 'Azure-Storage/1.1.0-1.1.0 (Python CPython 2.7.17; Linux 4.18.0-147.5.1.el8_1.x86_64)', 'If-Unmodified-Since': None, 'x-ms-access-tier': None, 'Authorization': u'SharedKey quayus1:D/P1z...', 'x-ms-source-if-modified-since': None, 'x-ms-client-request-id': 'd9b8fbb0-7401-11ea-ac06-0a580a830224', 'x-ms-lease-id': None, 'If-Match': None, 'If-Modified-Since': None, 'If-None-Match': None, 'x-ms-source-lease-id': None, 'x-ms-source-if-none-match': None, 'x-ms-date': 'Wed, 01 Apr 2020 10:16:28 GMT'}.
2020-04-01 10:16:28,661 [195] [INFO] [azure.storage.common.storageclient] Client-Request-ID=d9b8fbb0-7401-11ea-ac06-0a580a830224 Operation failed: checking if the operation should be retried. Current retry count=0, Server-Timestamp=Wed, 01 Apr 2020 10:16:27 GMT, Server-Request-ID=183fb883-301e-00d4-320e-08075a000000, HTTP status code=404, Exception=The specified container does not exist.ErrorCode: ContainerNotFound<?xml version="1.0" encoding="utf-8"?><Error><Code>ContainerNotFound</Code><Message>The specified container does not exist.RequestId:183fb883-301e-00d4-320e-08075a000000Time:2020-04-01T10:16:28.6007095Z</Message></Error>.
2020-04-01 10:16:28,661 [195] [ERROR] [azure.storage.common.storageclient] Client-Request-ID=d9b8fbb0-7401-11ea-ac06-0a580a830224 Retry policy did not allow for a retry: Server-Timestamp=Wed, 01 Apr 2020 10:16:27 GMT, Server-Request-ID=183fb883-301e-00d4-320e-08075a000000, HTTP status code=404, Exception=The specified container does not exist.ErrorCode: ContainerNotFound<?xml version="1.0" encoding="utf-8"?><Error><Code>ContainerNotFound</Code><Message>The specified container does not exist.RequestId:183fb883-301e-00d4-320e-08075a000000Time:2020-04-01T10:16:28.6007095Z</Message></Error>.
storagereplication stdout | 2020-04-01 10:16:28,661 [195] [ERROR] [azure.storage.common.storageclient] Client-Request-ID=d9b8fbb0-7401-11ea-ac06-0a580a830224 Retry policy did not allow for a retry: Server-Timestamp=Wed, 01 Apr 2020 10:16:27 GMT, Server-Request-ID=183fb883-301e-00d4-320e-08075a000000, HTTP status code=404, Exception=The specified container does not exist.ErrorCode: ContainerNotFound<?xml version="1.0" encoding="utf-8"?><Error><Code>ContainerNotFound</Code><Message>The specified container does not exist.RequestId:183fb883-301e-00d4-320e-08075a000000Time:2020-04-01T10:16:28.6007095Z</Message></Error>.

      In the above, it's clear that the containereu container is searched under the quayus1 account, but that should not be, it should be searched through quayeu1 account, as defined in the config.yaml file. If I create a containereu container under quayus1 account, then georeplication works, but this just copies blob within the same geolocation and not between two separate geolocations. This has been tested on OpenShift 4.3.8 and Quay was deployed via an operator. The same behaviour can be seen in local deployments as well.

      Please check, thank you!

        1. 02619805_config.yaml
          2 kB
        2. 02619805_example-quayecosystem-quay-864d4b9cb8-fkzzx.log
          15.61 MB
        3. ibazulic_config.yaml
          3 kB
        4. ibazulic_quay_enterprise.log
          1.93 MB

              jschorr Joseph Schorr (Inactive)
              rhn-support-ibazulic Ivan Bazulic
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: