-
Story
-
Resolution: Unresolved
-
Normal
-
None
-
None
-
None
-
False
-
None
-
False
This is an issue found when use podman to pull image from quay, when the backend registry storage is Veritas S3, pull image was failed with error message "invalid status code from registry 403 ". pls check attached Quay logs
podman pull ocp4-quay.arbetsformedlingen.se/rendi/test1:eap --log-level=debug INFO[0000] podman filtering at log level debug DEBU[0000] Called pull.PersistentPreRunE(podman pull ocp4-quay.arbetsformedlingen.se/rendi/test1:eap --log-level=debug) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that metacopy is being used DEBU[0000] Cached value indicated that native-diff is not being used INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true DEBU[0000] Initializing event backend file DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" INFO[0000] Setting parallel job count to 25 DEBU[0000] Pulling image ocp4-quay.arbetsformedlingen.se/rendi/test1:eap (policy: always) DEBU[0000] Looking up image "ocp4-quay.arbetsformedlingen.se/rendi/test1:eap" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "ocp4-quay.arbetsformedlingen.se/rendi/test1:eap" ... DEBU[0000] Trying "ocp4-quay.arbetsformedlingen.se/rendi/test1:eap" ... DEBU[0000] Trying "ocp4-quay.arbetsformedlingen.se/rendi/test1:eap" ... DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf" DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate ocp4-quay.arbetsformedlingen.se/rendi/test1:eap for ocp4-quay.arbetsformedlingen.se/rendi/test1:eap DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]ocp4-quay.arbetsformedlingen.se/rendi/test1:eap" Trying to pull ocp4-quay.arbetsformedlingen.se/rendi/test1:eap... DEBU[0000] Copying source image //ocp4-quay.arbetsformedlingen.se/rendi/test1:eap to destination image [overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]ocp4-quay.arbetsformedlingen.se/rendi/test1:eap DEBU[0000] Using registries.d directory /etc/containers/registries.d DEBU[0000] Trying to access "ocp4-quay.arbetsformedlingen.se/rendi/test1:eap" DEBU[0000] Found credentials for ocp4-quay.arbetsformedlingen.se/rendi/test1 in credential helper containers-auth.json in file /run/user/0/containers/auth.json DEBU[0000] No signature storage configuration found for ocp4-quay.arbetsformedlingen.se/rendi/test1:eap, using built-in default file:///var/lib/containers/sigstore DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/ocp4-quay.arbetsformedlingen.se DEBU[0000] GET https://ocp4-quay.arbetsformedlingen.se/v2/ DEBU[0000] Ping https://ocp4-quay.arbetsformedlingen.se/v2/ status 401 DEBU[0000] GET https://ocp4-quay.arbetsformedlingen.se/v2/auth?account=%24app&scope=repository%3Arendi%2Ftest1%3Apull&service=ocp4-quay.arbetsformedlingen.se DEBU[0000] Increasing token expiration to: 60 seconds DEBU[0000] GET https://ocp4-quay.arbetsformedlingen.se/v2/rendi/test1/manifests/eap DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" DEBU[0000] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb DEBU[0000] IsRunningImageAllowed for image docker:ocp4-quay.arbetsformedlingen.se/rendi/test1:eap DEBU[0000] Using default policy section DEBU[0000] Requirement 0: allowed DEBU[0000] Overall: allowed DEBU[0000] Downloading /v2/rendi/test1/blobs/sha256:594c1a2053d4a4766cfeaf59ee18475317f4f084a80df9caaf293670449dddf8 DEBU[0000] GET https://ocp4-quay.arbetsformedlingen.se/v2/rendi/test1/blobs/sha256:594c1a2053d4a4766cfeaf59ee18475317f4f084a80df9caaf293670449dddf8 DEBU[0000] Error pulling candidate ocp4-quay.arbetsformedlingen.se/rendi/test1:eap: parsing image configuration: Error fetching blob: invalid status code from registry 403 (Forbidden) Error: parsing image configuration: Error fetching blob: invalid status code from registry 403 (Forbidden)
Config: ALLOW_PULLS_WITHOUT_STRICT_LOGGING: true AUTHENTICATION_TYPE: AppToken AVATAR_KIND: local BUILDLOGS_REDIS: host: 10.142.2.37 port: 6380 CIAM_LOGIN_CONFIG: DEBUGGING : False CLIENT_ID: 71b29a5c-03d1-438d-850d-2e8e066f4c17 CLIENT_SECRET: vOsyh6LrY_F2AvxbKtY6T52nyUKCJqMS OIDC_SERVER: https://ciam.arbetsformedlingen.se/uas/ SERVICE_ICON: https://arbetsformedlingen.se/webdav/files/logo/logo.svg LOGIN_SCOPES: - openid SERVICE_NAME: ARBETSFORMEDLINGEN DATABASE_SECRET_KEY: xxx DB_CONNECTION_ARGS: {} DB_URI: xxx/quaydb?sslmode=disable DEFAULT_TAG_EXPIRATION: 2w DISTRIBUTED_STORAGE_CONFIG: default: - RadosGWStorage - access_key: xxx bucket_name: ocp4-quay-s3 is_secure: true hostname: vrts-s3.mgmt.ams.se secret_key: xxxx storage_path: /datastorage/storage # pull: # - S3Storage # - host: vrts-s3.mgmt.ams.se # s3_region: us-east-1 # s3_access_key: xxxx # s3_bucket: ocp4-quay-s3 # s3_secret_key: xxxx # storage_path: /datastorage/storageDISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: [default] DISTRIBUTED_STORAGE_PREFERENCE: - defaultEXTERNAL_TLS_TERMINATION: true FEATURE_PROXY_CACHE: true FEATURE_ACI_CONVERSION: false FEATURE_ACTION_LOG_ROTATION: false FEATURE_ANONYMOUS_ACCESS: true FEATURE_APP_REGISTRY: true FEATURE_APP_SPECIFIC_TOKENS: true FEATURE_BITBUCKET_BUILD: false FEATURE_BLACKLISTED_EMAILS: false FEATURE_BUILD_SUPPORT: false FEATURE_CHANGE_TAG_EXPIRATION: true FEATURE_DIRECT_LOGIN: false FEATURE_EXTENDED_REPOSITORY_NAMES: true FEATURE_FIPS: false FEATURE_GITHUB_BUILD: false FEATURE_GITHUB_LOGIN: false FEATURE_GITLAB_BUILD: false FEATURE_GOOGLE_LOGIN: false FEATURE_INVITE_ONLY_USER_CREATION: false FEATURE_MAILING: false FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false FEATURE_PARTIAL_USER_AUTOCOMPLETE: true FEATURE_PROXY_STORAGE: true FEATURE_REPO_MIRROR: true FEATURE_REQUIRE_TEAM_INVITE: true FEATURE_RESTRICTED_V1_PUSH: true FEATURE_SECURITY_NOTIFICATIONS: false FEATURE_SECURITY_SCANNER: true FEATURE_STORAGE_REPLICATION: false FEATURE_TEAM_SYNCING: true FEATURE_USER_CREATION: true FEATURE_USER_LAST_ACCESSED: true FEATURE_USER_LOG_ACCESS: true FEATURE_USER_METADATA: true FEATURE_USER_RENAME: true FEATURE_USERNAME_CONFIRMATION: true FRESH_LOGIN_TIMEOUT: 10m GITHUB_LOGIN_CONFIG: {} GITHUB_TRIGGER_CONFIG: {} GITLAB_TRIGGER_KIND: {} LDAP_ALLOW_INSECURE_FALLBACK: false LDAP_EMAIL_ATTR: mail LDAP_UID_ATTR: uid LDAP_URI: ldap://localhost LOG_ARCHIVE_LOCATION: default LOGS_MODEL: database LOGS_MODEL_CONFIG: {} MAIL_DEFAULT_SENDER: support@quay.io MAIL_PORT: 587 MAIL_USE_AUTH: false MAIL_USE_TLS: false PREFERRED_URL_SCHEME: https REGISTRY_TITLE: Project Quay REGISTRY_TITLE_SHORT: Project Quay REPO_MIRROR_INTERVAL: 900 REPO_MIRROR_TLS_VERIFY: true SEARCH_MAX_RESULT_PAGE_COUNT: 10 SEARCH_RESULTS_PER_PAGE: 10 SECRET_KEY: f25d2057-1081-408b-ba84-96572a1b2163 SECURITY_SCANNER_INDEXING_INTERVAL: 30 SECURITY_SCANNER_V4_ENDPOINT: http://10.142.2.37:6060 SECURITY_SCANNER_V4_PSK: NTI1MTdlaDVqZWE5Ng== SERVER_HOSTNAME: ocp4-quay.arbetsformedlingen.se SETUP_COMPLETE: true SUPER_USERS: - rendi - jooda - jonju TAG_EXPIRATION_OPTIONS: - 0s - 1d - 1w - 2w - 4w TEAM_RESYNC_STALE_TIME: 30m TESTING: false USE_CDN: false USER_EVENTS_REDIS: host: 10.142.2.37 port: 6380 USER_RECOVERY_TOKEN_LIFETIME: 30m USERFILES_LOCATION: default
Expected Results:
Use docker or podman can pull image successfully from quay.
Actual Results:
Unable to pull image from quay.