Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-5073

Quay pull image was failed with 400 error code when backend storage is Nutanix Object Storage

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • quay-v3.7.11, quay-v3.8.1, quay-v3.8.6
    • quay
    • False
    • None
    • False

      Description:

      This is an issue found when deployed Quay with Nutanix Object Storage, after push image to Quay successfully, pull image was failed with 400 error code, checked Quay App Pod logs, get error message "Error parsing image configuration: Error fetching blob: invalid status code from registry 400 (Bad Request)",  see detailed logs attached.

      Quay Version: 3.7.11/3.8.1

      Note:

      • This issue can be reproduced in Quay 3.8.1 and 3.7.11
      • This issue only existed when enabled "Proxy storage via Quay "
      Quay 3.7.11:

      podman pull quay3711-quay-qua37.apps.quay38nutnx01.qe.devcluster.openshift.com/quay/demo:rhel79-747 --creds quay:password --tls-verify=false
Trying to pull quay3711-quay-qua37.apps.quay38nutnx01.qe.devcluster.openshift.com/quay/demo:rhel79-747...
  Error fetching blob: invalid status code from registry 400 (Bad Request)
Error: unable to pull quay3711-quay-qua37.apps.quay38nutnx01.qe.devcluster.openshift.com/quay/demo:rhel79-747: Error parsing image configuration: Error fetching blob: invalid status code from registry 400 (Bad Request)

      Quay Logs:

      securityworker stdout | 2023-02-10 07:28:15,395 [87] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7f6ee658fa90>>
securityworker stdout | 2023-02-10 07:28:15,395 [87] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7f6ef6899790>
securityworker stdout | 2023-02-10 07:28:15,395 [87] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x7f6ee658fb20>>
securityworker stdout | 2023-02-10 07:28:15,395 [87] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f6eecec7d30>>
securityworker stdout | 2023-02-10 07:28:15,395 [87] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f6ee658fb20>>
securityworker stdout | 2023-02-10 07:28:15,395 [87] [DEBUG] [botocore.auth] Calculating signature using hmacv1 auth.
securityworker stdout | 2023-02-10 07:28:15,395 [87] [DEBUG] [botocore.auth] HTTP request method: GET
securityworker stdout | 2023-02-10 07:28:15,395 [87] [DEBUG] [botocore.auth] StringToSign:
securityworker stdout | GET
securityworker stdout | 1676014695
securityworker stdout | /quay/quay3711/sha256/5b/5b8d16cfb98918b19072cedfeb8e4978fd60635c705623c0417ca28328df674f
securityworker stdout | 2023-02-10 07:28:15,400 [87] [DEBUG] [storage.downloadproxy] Proxying via URL https://quay3711-quay-qua37.apps.quay38nutnx01.qe.devcluster.openshift.com/_storage_proxy/ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKU1V6STFOaUlzSW10cFpDSTZJalJYYWxSWlEzRTFVbUZtUTJZemNsVjNRbEpTUWkxWVpUaFpPRkpQVVV0bk1WbE5lWEJHY1dkRFdrMGlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVUTTNNVEV0Y1hWaGVTMXhkV0V6Tnk1aGNIQnpMbkYxWVhrek9HNTFkRzU0TURFdWNXVXVaR1YyWTJ4MWMzUmxjaTV2Y0dWdWMyaHBablF1WTI5dElpd2libUptSWpveE5qYzJNREUwTURrMUxDSnBZWFFpT2pFMk56WXdNVFF3T1RVc0ltVjRjQ0k2TVRZM05qQXhOREV5TlN3aWMzVmlJam9pYzNSdmNtRm5aWEJ5YjNoNUlpd2lZV05qWlhOeklqcGJleUowZVhCbElqb2ljM1J2Y21GblpYQnliM2g1SWl3aWRYSnBJam9pY1hWaGVTOXhkV0Y1TXpjeE1TOXphR0V5TlRZdk5XSXZOV0k0WkRFMlkyWmlPVGc1TVRoaU1Ua3dOekpqWldSbVpXSTRaVFE1TnpobVpEWXdOak0xWXpjd05UWXlNMk13TkRFM1kyRXlPRE15T0dSbU5qYzBaajlCVjFOQlkyTmxjM05MWlhsSlpEMUxYM1J3YVRCU2RGQjVjbXhTUjNkYU9XOVpTR28xZFZKalJ6VXlZVmhrVGlaVGFXZHVZWFIxY21VOVZraHZaRTl3UzB0QlVHOUxaVE5zYVhkUFYwWkdaRTV4ZEhrMEpUTkVKa1Y0Y0dseVpYTTlNVFkzTmpBeE5EWTVOU0lzSW1odmMzUWlPaUp2Y3k1c2RITXRZMngxYzNSbGNpNXBiblJsY201aGJDNXVkWFJoYm1sNExXUmxkaTVrWlhaamJIVnpkR1Z5TG05d1pXNXphR2xtZEM1amIyMDZORFF6SWl3aWMyTm9aVzFsSWpvaWFIUjBjSE1pZlYwc0ltTnZiblJsZUhRaU9udDlmUS5XZUxra3gyRV9mYlBhT1hDcjFESTcwWHJYOEIyNjlzNElwUV9pcEtTelJYZnBsNlgzVGZFTTNNdlk2MS02VFQ1Q0VzRDhyS0JVTzJuWDdCbEY2S2Z6aWtlczBadFhPbFFQRW5zVlltX0x5QnFIVTU4NENFUzcyakdRMVJaY2JaMTJmQVpRb0FxUU9LS0ItWTBIYVA2UlNYWGJIV2VjNzdzNHhTd2hObUQ1ZGlQWnE0T0JQaEJickpwdHVpR2dvOXJGVC1BcFdDcmNLWUFrYjhTZkduVmZoUXhwT1djYUhuUTZLSzEtZE54UjBKd1BmemZxQUFxZEJGajd3YWwwUmxJOTYtTHg4cHNYUlZpOWxkZ2pFMXJQYXZRVGRmaXhleUpiQ3RDVGMxeHhTSXd3MUdaQm56WU5uSm0tT3JiV0xJeFZSVG11TDFVYjI2V3UwSWQ5bnY3MFE=/https/os.lts-cluster.internal.nutanix-dev.devcluster.openshift.com:443/quay/quay3711/sha256/5b/5b8d16cfb98918b19072cedfeb8e4978fd60635c705623c0417ca28328df674f?AWSAccessKeyId=K_tpi0RtPyrlRGwZ9oYHj5uRcG52aXdN&Signature=VHodOpKKAPoKe3liwOWFFdNqty4%3D&Expires=1676014695
securityworker stdout | 2023-02-10 07:28:15,400 [87] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7f6ef68999d0>

      Quay config.yaml:

      ALLOW_PULLS_WITHOUT_STRICT_LOGGING: false
AUTHENTICATION_TYPE: Database
AVATAR_KIND: local
BUILDLOGS_REDIS:
  host: quay3711-quay-redis
  port: 6379
DATABASE_SECRET_KEY: IN4FE1dcwX1o7NJujsNHz8L-UdwHvZHTiqqHjfbzb9VhgVgyq4TTJkXyheW8P1-xaL321korkrZ3sOiN
DB_CONNECTION_ARGS:
  autorollback: true
  threadlocals: true
DB_URI: postgresql://quay3711-quay-database:AlaRhFgqJt0JeF-Zv9VYmK1ztwViCxPkVFDyEfT9saA2rKJ27EpdusIEKfUBzKGqPjYFejiDyrADYzbI@quay3711-quay-database:5432/quay3711-quay-database
DEFAULT_TAG_EXPIRATION: 2w
DISTRIBUTED_STORAGE_CONFIG:
  local_us:
  - RadosGWStorage
  - access_key: K_tpi0RtPyrlRGwZ9oYHj5uRcG52aXdN
    bucket_name: quay
    hostname: os.lts-cluster.internal.nutanix-dev.devcluster.openshift.com
    is_secure: true
    port: 443
    secret_key: bjRuc_t45wFry2DjdTJz7i0xn_BiI6AP
    storage_path: /quay3711
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
- local_us
DISTRIBUTED_STORAGE_PREFERENCE:
- local_us
ENTERPRISE_LOGO_URL: /static/img/RH_Logo_Quay_Black_UX-horizontal.svg
EXTERNAL_TLS_TERMINATION: true
FEATURE_ACTION_LOG_ROTATION: false
FEATURE_ANONYMOUS_ACCESS: true
FEATURE_APP_SPECIFIC_TOKENS: true
FEATURE_BITBUCKET_BUILD: false
FEATURE_BLACKLISTED_EMAILS: false
FEATURE_BUILD_SUPPORT: false
FEATURE_CHANGE_TAG_EXPIRATION: true
FEATURE_DIRECT_LOGIN: true
FEATURE_EXTENDED_REPOSITORY_NAMES: true
FEATURE_FIPS: false
FEATURE_GITHUB_BUILD: false
FEATURE_GITHUB_LOGIN: false
FEATURE_GITLAB_BUILD: false
FEATURE_GOOGLE_LOGIN: false
FEATURE_INVITE_ONLY_USER_CREATION: false
FEATURE_MAILING: false
FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
FEATURE_PARTIAL_USER_AUTOCOMPLETE: true
FEATURE_PROXY_STORAGE: true
FEATURE_REPO_MIRROR: true
FEATURE_SECURITY_NOTIFICATIONS: true
FEATURE_SECURITY_SCANNER: true
FEATURE_STORAGE_REPLICATION: false
FEATURE_TEAM_SYNCING: false
FEATURE_USER_CREATION: true
FEATURE_USER_LAST_ACCESSED: true
FEATURE_USER_LOG_ACCESS: false
FEATURE_USER_METADATA: false
FEATURE_USER_RENAME: false
FEATURE_USERNAME_CONFIRMATION: true
FRESH_LOGIN_TIMEOUT: 10m
GITHUB_LOGIN_CONFIG: {}
GITHUB_TRIGGER_CONFIG: {}
GITLAB_TRIGGER_KIND: {}
LDAP_ALLOW_INSECURE_FALLBACK: false
LDAP_EMAIL_ATTR: mail
LDAP_UID_ATTR: uid
LDAP_URI: ldap://localhost
LOGS_MODEL: database
LOGS_MODEL_CONFIG: {}
MAIL_DEFAULT_SENDER: support@quay.io
MAIL_PORT: 587
MAIL_USE_AUTH: false
MAIL_USE_TLS: false
PREFERRED_URL_SCHEME: https
REGISTRY_TITLE: Red Hat Quay
REGISTRY_TITLE_SHORT: Red Hat Quay
REPO_MIRROR_INTERVAL: 30
REPO_MIRROR_TLS_VERIFY: true
SEARCH_MAX_RESULT_PAGE_COUNT: 10
SEARCH_RESULTS_PER_PAGE: 10
SECRET_KEY: Y13seAl98-KZ2g64sB5VJ7iM-IsSjC7oALyJnR9z0pPpLKGmHZ41mKBXHTybq0ES096FSGqI1sUMPIrL
SECURITY_SCANNER_INDEXING_INTERVAL: 30
SECURITY_SCANNER_V4_ENDPOINT: http://quay3711-clair-app.qua37.svc.cluster.local
SECURITY_SCANNER_V4_NAMESPACE_WHITELIST:
- admin
SECURITY_SCANNER_V4_PSK: RndDd29Wd1ZOMnVYWlVFVUtZd0VobG9vbzFiY1JBdjI=
SERVER_HOSTNAME: quay3711-quay-qua37.apps.quay38nutnx01.qe.devcluster.openshift.com
SETUP_COMPLETE: true
TAG_EXPIRATION_OPTIONS:
- 2w
TEAM_RESYNC_STALE_TIME: 60m
TESTING: false
USER_EVENTS_REDIS:
  host: quay3711-quay-redis
  port: 6379
USER_RECOVERY_TOKEN_LIFETIME: 30m

        1. image-2023-02-10-16-02-52-936.png
          image-2023-02-10-16-02-52-936.png
          391 kB
        2. quay381_app2.logs
          26.57 MB
        3. quay381_app1.logs
          43.39 MB

              Unassigned Unassigned
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: