Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-4754

Podman pull image failed when the backend storage registry is Veritas S3

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • None
    • None
    • False
    • None
    • False

      This is an issue found when use podman to pull image from quay, when the backend registry storage is Veritas S3, pull image was failed with error message "invalid status code from registry 403 ". pls check attached Quay logs 

      podman pull ocp4-quay.arbetsformedlingen.se/rendi/test1:eap --log-level=debug
      INFO[0000] podman filtering at log level debug
      DEBU[0000] Called pull.PersistentPreRunE(podman pull ocp4-quay.arbetsformedlingen.se/rendi/test1:eap --log-level=debug)
      DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
      DEBU[0000] Using conmon: "/usr/bin/conmon"
      DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
      DEBU[0000] Using graph driver overlay
      DEBU[0000] Using graph root /var/lib/containers/storage
      DEBU[0000] Using run root /run/containers/storage
      DEBU[0000] Using static dir /var/lib/containers/storage/libpod
      DEBU[0000] Using tmp dir /run/libpod
      DEBU[0000] Using volume path /var/lib/containers/storage/volumes
      DEBU[0000] Set libpod namespace to ""
      DEBU[0000] [graphdriver] trying provided driver "overlay"
      DEBU[0000] Cached value indicated that overlay is supported
      DEBU[0000] Cached value indicated that overlay is supported
      DEBU[0000] Cached value indicated that metacopy is being used
      DEBU[0000] Cached value indicated that native-diff is not being used
      INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
      DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
      DEBU[0000] Initializing event backend file
      DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
      DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
      DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
      DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
      DEBU[0000] Using OCI runtime "/usr/bin/runc"
      INFO[0000] Setting parallel job count to 25
      DEBU[0000] Pulling image ocp4-quay.arbetsformedlingen.se/rendi/test1:eap (policy: always)
      DEBU[0000] Looking up image "ocp4-quay.arbetsformedlingen.se/rendi/test1:eap" in local containers storage
      DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
      DEBU[0000] Trying "ocp4-quay.arbetsformedlingen.se/rendi/test1:eap" ...
      DEBU[0000] Trying "ocp4-quay.arbetsformedlingen.se/rendi/test1:eap" ...
      DEBU[0000] Trying "ocp4-quay.arbetsformedlingen.se/rendi/test1:eap" ...
      DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
      DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
      DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf"
      DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf"
      DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
      DEBU[0000] Attempting to pull candidate ocp4-quay.arbetsformedlingen.se/rendi/test1:eap for ocp4-quay.arbetsformedlingen.se/rendi/test1:eap
      DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]ocp4-quay.arbetsformedlingen.se/rendi/test1:eap"
      Trying to pull ocp4-quay.arbetsformedlingen.se/rendi/test1:eap...
      DEBU[0000] Copying source image //ocp4-quay.arbetsformedlingen.se/rendi/test1:eap to destination image [overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]ocp4-quay.arbetsformedlingen.se/rendi/test1:eap
      DEBU[0000] Using registries.d directory /etc/containers/registries.d
      DEBU[0000] Trying to access "ocp4-quay.arbetsformedlingen.se/rendi/test1:eap"
      DEBU[0000] Found credentials for ocp4-quay.arbetsformedlingen.se/rendi/test1 in credential helper containers-auth.json in file /run/user/0/containers/auth.json
      DEBU[0000]  No signature storage configuration found for ocp4-quay.arbetsformedlingen.se/rendi/test1:eap, using built-in default file:///var/lib/containers/sigstore
      DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/ocp4-quay.arbetsformedlingen.se
      DEBU[0000] GET https://ocp4-quay.arbetsformedlingen.se/v2/
      DEBU[0000] Ping https://ocp4-quay.arbetsformedlingen.se/v2/ status 401
      DEBU[0000] GET https://ocp4-quay.arbetsformedlingen.se/v2/auth?account=%24app&scope=repository%3Arendi%2Ftest1%3Apull&service=ocp4-quay.arbetsformedlingen.se
      DEBU[0000] Increasing token expiration to: 60 seconds
      DEBU[0000] GET https://ocp4-quay.arbetsformedlingen.se/v2/rendi/test1/manifests/eap
      DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
      DEBU[0000] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb
      DEBU[0000] IsRunningImageAllowed for image docker:ocp4-quay.arbetsformedlingen.se/rendi/test1:eap
      DEBU[0000]  Using default policy section
      DEBU[0000]  Requirement 0: allowed
      DEBU[0000] Overall: allowed
      DEBU[0000] Downloading /v2/rendi/test1/blobs/sha256:594c1a2053d4a4766cfeaf59ee18475317f4f084a80df9caaf293670449dddf8
      DEBU[0000] GET https://ocp4-quay.arbetsformedlingen.se/v2/rendi/test1/blobs/sha256:594c1a2053d4a4766cfeaf59ee18475317f4f084a80df9caaf293670449dddf8
      DEBU[0000] Error pulling candidate ocp4-quay.arbetsformedlingen.se/rendi/test1:eap: parsing image configuration: Error fetching blob: invalid status code from registry 403 (Forbidden)
      Error: parsing image configuration: Error fetching blob: invalid status code from registry 403 (Forbidden) 
      Config:
      
      
      ALLOW_PULLS_WITHOUT_STRICT_LOGGING: true
      AUTHENTICATION_TYPE: AppToken
      AVATAR_KIND: local
      BUILDLOGS_REDIS:
          host: 10.142.2.37
          port: 6380
      CIAM_LOGIN_CONFIG:
          DEBUGGING : False
          CLIENT_ID: 71b29a5c-03d1-438d-850d-2e8e066f4c17
          CLIENT_SECRET: vOsyh6LrY_F2AvxbKtY6T52nyUKCJqMS
          OIDC_SERVER: https://ciam.arbetsformedlingen.se/uas/
          SERVICE_ICON: https://arbetsformedlingen.se/webdav/files/logo/logo.svg
          LOGIN_SCOPES:
              - openid
          SERVICE_NAME: ARBETSFORMEDLINGEN
      DATABASE_SECRET_KEY: xxx
      DB_CONNECTION_ARGS: {}
      DB_URI: xxx/quaydb?sslmode=disable
      DEFAULT_TAG_EXPIRATION: 2w
      DISTRIBUTED_STORAGE_CONFIG:
          default:
              - RadosGWStorage
              - access_key: xxx
                bucket_name: ocp4-quay-s3
                is_secure: true
                hostname: vrts-s3.mgmt.ams.se
                secret_key: xxxx
                storage_path: /datastorage/storage
      
      #    pull:
      #        - S3Storage
      #        - host: vrts-s3.mgmt.ams.se
      #          s3_region: us-east-1
      #          s3_access_key: xxxx
      #          s3_bucket: ocp4-quay-s3
      #          s3_secret_key: xxxx
      #          storage_path: /datastorage/storageDISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: [default]
      DISTRIBUTED_STORAGE_PREFERENCE:
          - defaultEXTERNAL_TLS_TERMINATION: true
      FEATURE_PROXY_CACHE: true
      FEATURE_ACI_CONVERSION: false
      FEATURE_ACTION_LOG_ROTATION: false
      FEATURE_ANONYMOUS_ACCESS: true
      FEATURE_APP_REGISTRY: true
      FEATURE_APP_SPECIFIC_TOKENS: true
      FEATURE_BITBUCKET_BUILD: false
      FEATURE_BLACKLISTED_EMAILS: false
      FEATURE_BUILD_SUPPORT: false
      FEATURE_CHANGE_TAG_EXPIRATION: true
      FEATURE_DIRECT_LOGIN: false
      FEATURE_EXTENDED_REPOSITORY_NAMES: true
      FEATURE_FIPS: false
      FEATURE_GITHUB_BUILD: false
      FEATURE_GITHUB_LOGIN: false
      FEATURE_GITLAB_BUILD: false
      FEATURE_GOOGLE_LOGIN: false
      FEATURE_INVITE_ONLY_USER_CREATION: false
      FEATURE_MAILING: false
      FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
      FEATURE_PARTIAL_USER_AUTOCOMPLETE: true
      FEATURE_PROXY_STORAGE: true
      FEATURE_REPO_MIRROR: true
      FEATURE_REQUIRE_TEAM_INVITE: true
      FEATURE_RESTRICTED_V1_PUSH: true
      FEATURE_SECURITY_NOTIFICATIONS: false
      FEATURE_SECURITY_SCANNER: true
      FEATURE_STORAGE_REPLICATION: false
      FEATURE_TEAM_SYNCING: true
      FEATURE_USER_CREATION: true
      FEATURE_USER_LAST_ACCESSED: true
      FEATURE_USER_LOG_ACCESS: true
      FEATURE_USER_METADATA: true
      FEATURE_USER_RENAME: true
      FEATURE_USERNAME_CONFIRMATION: true
      FRESH_LOGIN_TIMEOUT: 10m
      GITHUB_LOGIN_CONFIG: {}
      GITHUB_TRIGGER_CONFIG: {}
      GITLAB_TRIGGER_KIND: {}
      LDAP_ALLOW_INSECURE_FALLBACK: false
      LDAP_EMAIL_ATTR: mail
      LDAP_UID_ATTR: uid
      LDAP_URI: ldap://localhost
      LOG_ARCHIVE_LOCATION: default
      LOGS_MODEL: database
      LOGS_MODEL_CONFIG: {}
      MAIL_DEFAULT_SENDER: support@quay.io
      MAIL_PORT: 587
      MAIL_USE_AUTH: false
      MAIL_USE_TLS: false
      PREFERRED_URL_SCHEME: https
      REGISTRY_TITLE: Project Quay
      REGISTRY_TITLE_SHORT: Project Quay
      REPO_MIRROR_INTERVAL: 900
      REPO_MIRROR_TLS_VERIFY: true
      SEARCH_MAX_RESULT_PAGE_COUNT: 10
      SEARCH_RESULTS_PER_PAGE: 10
      SECRET_KEY: f25d2057-1081-408b-ba84-96572a1b2163
      SECURITY_SCANNER_INDEXING_INTERVAL: 30
      SECURITY_SCANNER_V4_ENDPOINT: http://10.142.2.37:6060
      SECURITY_SCANNER_V4_PSK: NTI1MTdlaDVqZWE5Ng==
      SERVER_HOSTNAME: ocp4-quay.arbetsformedlingen.se
      SETUP_COMPLETE: true
      SUPER_USERS:
          - rendi
          - jooda
          - jonju
      TAG_EXPIRATION_OPTIONS:
          - 0s
          - 1d
          - 1w
          - 2w
          - 4w
      TEAM_RESYNC_STALE_TIME: 30m
      TESTING: false
      USE_CDN: false
      USER_EVENTS_REDIS:
          host: 10.142.2.37
          port: 6380
      USER_RECOVERY_TOKEN_LIFETIME: 30m
      USERFILES_LOCATION: default 

      Expected Results:

      Use docker or podman can pull image successfully from quay.

      Actual Results:

      Unable to pull image from quay.

              Unassigned Unassigned
              dilip.renkila@arbetsformedlingen.se Dilip Renkila (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: