Normal Description:
This is an issue when deployed Quay 3.8.1 with operator using Nutanix Object Storage, after pushed image to Quay, found Clair can't scan the image vulnerability, checked Clair logs get error message as following, more detailed logs see attachments.
Note:
- This issue can be reproduced in Quay 3.8.1 and 3.7.11
- This issue only existed when enabled "Proxy storage via Quay "
{"level":"warn","component":"indexer/controller/Controller.Index","manifest":"sha256:11b75612febb90ba0315dc90d9f5a1c62806b659680f8863779e7bd508d19d5a","state":"FetchLayers","error":"encountered error while fetching a layer: error realizing layer sha256:735bcfbfe54d91ba5b32637542e7275f35d349a7efb28bee50dc5ebf9759dbe2: fetcher: unexpected status code: 400 Bad Request (body starts: \"\")","time":"2023-02-08T07:19:32Z","message":"layers fetch failure"}
Quay Version: Quay 3.8.1
Quay Image:
Quay Logs:
nginx stdout | 10.131.0.2 (-) - - [08/Feb/2023:07:30:36 +0000] "GET /_storage_proxy/ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKU1V6STFOaUlzSW10cFpDSTZJakIxVFZBdE1FdFJXVkZtWjB0Q2FqaEpVbGR5Vmt4UlpIWmpaMU10V25Ga2N6Qnhja05IYkROMVowRWlmUS5leUpwYzNNaU9pSnhkV0Y1SWl3aVlYVmtJam9pY1hWaGVUTTRNUzF4ZFdGNUxYRjFZWGt6T0RFdVlYQndjeTV4ZFdGNWJuVjBZVzVwZUM1eFpTNWtaWFpqYkhWemRHVnlMbTl3Wlc1emFHbG1kQzVqYjIwaUxDSnVZbVlpT2pFMk56VTROREUwTXpVc0ltbGhkQ0k2TVRZM05UZzBNVFF6TlN3aVpYaHdJam94TmpjMU9EUXhORFkxTENKemRXSWlPaUp6ZEc5eVlXZGxjSEp2ZUhraUxDSmhZMk5sYzNNaU9sdDdJblI1Y0dVaU9pSnpkRzl5WVdkbGNISnZlSGtpTENKMWNta2lPaUp4ZFdGNUwzRjFZWGt6T0RFdmMyaGhNalUyTHpsbUx6bG1aRFk1WlRobFkyWmpNamRsTldZNE1UWmhOMlZsTVRZeFpUUmtaak15TTJVMU56UTJabVpoWXpreFl6WTFNRGhsWXpnek9UQmxZMlpoWlROa1pUWV9RVmRUUVdOalpYTnpTMlY1U1dROVMxOTBjR2t3VW5SUWVYSnNVa2QzV2psdldVaHFOWFZTWTBjMU1tRllaRTRtVTJsbmJtRjBkWEpsUFZsUVV6WTNNU1V5Um10MGQxTjFhMVEwWldoeVUyWTNVSFZDT1dGekpUTkVKa1Y0Y0dseVpYTTlNVFkzTlRnME1qQXpOU0lzSW1odmMzUWlPaUp2Y3k1c2RITXRZMngxYzNSbGNpNXBiblJsY201aGJDNXVkWFJoYm1sNExXUmxkaTVrWlhaamJIVnpkR1Z5TG05d1pXNXphR2xtZEM1amIyMDZORFF6SWl3aWMyTm9aVzFsSWpvaWFIUjBjSE1pZlYwc0ltTnZiblJsZUhRaU9udDlmUS5yWEVHZjgzdlFGbUc2aVg4T25EMHV4bW9scV9tM2JocDZBZ1BJWnlQN1Q0Y3FoYm1RcmZic0RvOXdzR1hfMDZpcFkyUkFNa09aNlRLLTJKN3dOa3d4ZTFYRVpWdG1aR0dhWnJySU55R2M0NVktTkNrTVF6REg5Y2J4dkpxc2dXck5MTVAyNTNpSXNmX2NCTGNra05vR21qcUluUHhnZC1VZVZzX1dZOHhveTJTWHNQRW5KUFkyWHA5SDVYNktCX296TWhTaVJUVnJ5Y0ExNWpFSXR1Szk0YU9GQU5qTG9yZHZDWTctTWxlQ1Q5S3FGTlpla05wQ3ZKaTl6eU1EMktJajIwQ1hFYUJvQ1dFQTJVSC1lSWtkNWphOTZENFBZdVhucDFsdmVoVWpNSmItOHNPZzFmSlMyVTRqN29PQWplcS1TY1ZodTZ6TmZGcTJua1Z5LXRNRVE=/https/os.lts-cluster.internal.nutanix-dev.devcluster.openshift.com:443/quay/quay381/sha256/9f/9fd69e8ecfc27e5f816a7ee161e4df323e5746ffac91c6508ec8390ecfae3de6?AWSAccessKeyId=K_tpi0RtPyrlRGwZ9oYHj5uRcG52aXdN&Signature=YPS671%2FktwSukT4ehrSf7PuB9as%3D&Expires=1675842035 HTTP/1.1" 400 5 "-" "clair/v4" (0.024 2223 0.015)
Clair Logs:
{"level":"warn","component":"indexer/controller/Controller.Index","manifest":"sha256:11b75612febb90ba0315dc90d9f5a1c62806b659680f8863779e7bd508d19d5a","state":"FetchLayers","error":"encountered error while fetching a layer: error realizing layer sha256:735bcfbfe54d91ba5b32637542e7275f35d349a7efb28bee50dc5ebf9759dbe2: fetcher: unexpected status code: 400 Bad Request (body starts: \"\")","time":"2023-02-08T07:19:32Z","message":"layers fetch failure"}
{"level":"info","component":"indexer/controller/Controller.Index","manifest":"sha256:11b75612febb90ba0315dc90d9f5a1c62806b659680f8863779e7bd508d19d5a","state":"FetchLayers","time":"2023-02-08T07:19:32Z","message":"layers fetch done"}
{"level":"error","component":"indexer/controller/Controller.Index","manifest":"sha256:11b75612febb90ba0315dc90d9f5a1c62806b659680f8863779e7bd508d19d5a","state":"FetchLayers","error":"failed to fetch layers: encountered error while fetching a layer: error realizing layer sha256:735bcfbfe54d91ba5b32637542e7275f35d349a7efb28bee50dc5ebf9759dbe2: fetcher: unexpected status code: 400 Bad Request (body starts: \"\")","time":"2023-02-08T07:19:32Z","message":"error during scan"}
Quay config.yaml:
ALLOW_PULLS_WITHOUT_STRICT_LOGGING: false AUTHENTICATION_TYPE: Database AVATAR_KIND: local BUILDLOGS_REDIS: host: quay381-quay-redis port: 6379 DATABASE_SECRET_KEY: qUMoTdrZQBxDBPo5oMyHbnkdic931PsSiAva2B5-AjddIB7qBM1Q4qgRrYX15KxnqMMKeW48vuNEw4df DB_CONNECTION_ARGS: autorollback: true threadlocals: true DB_URI: postgresql://quay381-quay-database:QdeB882-ZUp8e25dglGRDg0iXnqwmPkI4rhBP2okfl3N3Tz84Tb5kAeglUFmGYBIORtSFARVWfdCgn3M@quay381-quay-database:5432/quay381-quay-database DEFAULT_TAG_EXPIRATION: 2w DISTRIBUTED_STORAGE_CONFIG: local_us: - RadosGWStorage - access_key: K_tpi0RtPyrlRGwZ9oYHj5uRcG52aXdN bucket_name: quay hostname: os.lts-cluster.internal.nutanix-dev.devcluster.openshift.com is_secure: true port: "443" secret_key: bjRuc_t45wFry2DjdTJz7i0xn_BiI6AP storage_path: /quay381 DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: - local_us DISTRIBUTED_STORAGE_PREFERENCE: - local_us ENTERPRISE_LOGO_URL: /static/img/RH_Logo_Quay_Black_UX-horizontal.svg EXTERNAL_TLS_TERMINATION: true FEATURE_ACTION_LOG_ROTATION: false FEATURE_ANONYMOUS_ACCESS: true FEATURE_APP_SPECIFIC_TOKENS: true FEATURE_BITBUCKET_BUILD: false FEATURE_BLACKLISTED_EMAILS: false FEATURE_BUILD_SUPPORT: false FEATURE_CHANGE_TAG_EXPIRATION: true FEATURE_DIRECT_LOGIN: true FEATURE_EXTENDED_REPOSITORY_NAMES: true FEATURE_FIPS: false FEATURE_GITHUB_BUILD: false FEATURE_GITHUB_LOGIN: false FEATURE_GITLAB_BUILD: false FEATURE_GOOGLE_LOGIN: false FEATURE_INVITE_ONLY_USER_CREATION: false FEATURE_MAILING: false FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false FEATURE_PARTIAL_USER_AUTOCOMPLETE: true FEATURE_PROXY_STORAGE: true FEATURE_REPO_MIRROR: true FEATURE_SECURITY_NOTIFICATIONS: true FEATURE_SECURITY_SCANNER: true FEATURE_STORAGE_REPLICATION: false FEATURE_TEAM_SYNCING: false FEATURE_USER_CREATION: true FEATURE_USER_LAST_ACCESSED: true FEATURE_USER_LOG_ACCESS: false FEATURE_USER_METADATA: false FEATURE_USER_RENAME: false FEATURE_USERNAME_CONFIRMATION: true FRESH_LOGIN_TIMEOUT: 10m GITHUB_LOGIN_CONFIG: {} GITHUB_TRIGGER_CONFIG: {} GITLAB_TRIGGER_KIND: {} LDAP_ALLOW_INSECURE_FALLBACK: false LDAP_EMAIL_ATTR: mail LDAP_UID_ATTR: uid LDAP_URI: ldap://localhost LOGS_MODEL: database LOGS_MODEL_CONFIG: {} MAIL_DEFAULT_SENDER: support@quay.io MAIL_PORT: 587 MAIL_USE_AUTH: false MAIL_USE_TLS: false PREFERRED_URL_SCHEME: https REGISTRY_TITLE: Red Hat Quay REGISTRY_TITLE_SHORT: Red Hat Quay REPO_MIRROR_INTERVAL: 30 REPO_MIRROR_TLS_VERIFY: true SEARCH_MAX_RESULT_PAGE_COUNT: 10 SEARCH_RESULTS_PER_PAGE: 10 SECRET_KEY: JH6Tuww2OYpc0d5kJCWDyg83TUEAdLh2ttVIUEbfx6j9DSawgwXrRvJ8kJIhhA-BeyIJG1Vp9OVIuQWC SECURITY_SCANNER_INDEXING_INTERVAL: 30 SECURITY_SCANNER_V4_ENDPOINT: http://quay381-clair-app.quay381.svc.cluster.local SECURITY_SCANNER_V4_NAMESPACE_WHITELIST: - admin SECURITY_SCANNER_V4_PSK: VGxmb3UxRmhBNVlURmo4VnhtbHp1UktaR1BGd2FZNlM= SERVER_HOSTNAME: quay381-quay-quay381.apps.quaynutanix.qe.devcluster.openshift.com SETUP_COMPLETE: true TAG_EXPIRATION_OPTIONS: - 2w TEAM_RESYNC_STALE_TIME: 60m TESTING: false USER_EVENTS_REDIS: host: quay381-quay-redis port: 6379 USER_RECOVERY_TOKEN_LIFETIME: 30m
