Loading...

Details

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: z-stream
Affects Version/s: quay-v3.6.0
Component/s: quay
Labels:
- quay-enterprise
- triaged

Blocked:
False
Ready:
False

RICE Score:
0

SFDC Cases Links:
SFDC Cases Counter:

Description

Description:

This is an issue found when use Quay 3.6.0 to build image from Github repo, but found the build job was failed to start on OCP baremetal Cluster, checked Quay APP POD found error "TypeError: quote_from_bytes() expected bytes", see attached Quay APP POD logs quay360_app_pod1.logs

Quay image: "quay-operator-bundle-container-v3.6.0-32"

Note: This issue is not existed with Quay 3.5.6

builder stdout | 2021-09-15 03:57:10,555 [60] [ERROR] [buildman.manager.ephemeral] Exception when starting builder for job: buildman/production/building/73a7a334-e43f-4524-b712-bf25807318d2 - Falling back to next configured executor
builder stdout | Traceback (most recent call last):
builder stdout |   File "/quay-registry/buildman/manager/ephemeral.py", line 650, in schedule
builder stdout |     execution_id = executor.start_builder(registration_token, build_job.build_uuid)
builder stdout |   File "/quay-registry/buildman/manager/executor.py", line 56, in wrapper
builder stdout |     rv = func(*args, **kwargs)
builder stdout |   File "/quay-registry/buildman/manager/executor.py", line 638, in start_builder
builder stdout |     user_data = self.generate_cloud_config(token, build_uuid, self.manager_hostname)
builder stdout |   File "/quay-registry/buildman/manager/executor.py", line 197, in generate_cloud_config
builder stdout |     TEMPLATE.render(
builder stdout |   File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
builder stdout |     self.environment.handle_exception()
builder stdout |   File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
builder stdout |     reraise(*rewrite_traceback_stack(source=source))
builder stdout |   File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
builder stdout |     raise value.with_traceback(tb)
builder stdout |   File "/quay-registry/buildman/templates/cloudconfig.json", line 82, in top-level template code
builder stdout |     "source": {{ ca_cert | dataurl | jsonify }}
builder stdout |   File "/quay-registry/buildman/container_cloud_config.py", line 115, in data_url
builder stdout |     data = "," + urlquote(content)
builder stdout |   File "/usr/lib64/python3.8/urllib/parse.py", line 839, in quote
builder stdout |     return quote_from_bytes(string, safe)
builder stdout |   File "/usr/lib64/python3.8/urllib/parse.py", line 864, in quote_from_bytes
builder stdout |     raise TypeError("quote_from_bytes() expected bytes")
builder stdout | TypeError: quote_from_bytes() expected bytes
builder stdout | 2021-09-15 03:57:10,556 [60] [ERROR] [buildman.manager.ephemeral] Could not start ephemeral worker for build 73a7a334-e43f-4524-b712-bf25807318d2

Config.yaml:

ALLOW_PULLS_WITHOUT_STRICT_LOGGING: false
ALLOWED_OCI_ARTIFACT_TYPES:
  application/vnd.cncf.helm.config.v1+json:
  - application/tar+gzip
  application/vnd.oci.image.layer.v1.tar+gzip+encrypted:
  - application/vnd.oci.image.layer.v1.tar+gzip+encrypted
AUTHENTICATION_TYPE: Database
AVATAR_KIND: local
BROWSER_API_CALLS_XHR_ONLY: false
BUILDLOGS_REDIS:
  host: quay360-quay-redis
  port: 6379
CREATE_REPOSITORY_ON_PUSH_PUBLIC: true
DATABASE_SECRET_KEY: eTAI6czN1zpnzP2jhE1hChXHIWETXzoKoX0Yf2sLkK1xwkroAOfU3a71q0MsNoqkM8uEd1eeBmh1vlHN
DB_CONNECTION_ARGS:
  autorollback: true
  threadlocals: true
DB_URI: postgresql://quay360-quay-database:UwbR2guNmm7lfarAHYqnzPEoTaA07h3qn7uyp2SSQIhJfL9aA3Oc5Ye--pat2Zu-qp51QWClmcdWvuiQ@quay360-quay-database:5432/quay360-quay-database
DEFAULT_TAG_EXPIRATION: 4w
DISTRIBUTED_STORAGE_CONFIG:
  default:
  - S3Storage
  - host: s3.us-east-2.amazonaws.com
    s3_access_key: ******
    s3_bucket: quay360
    s3_secret_key: ******
    storage_path: /quay360
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
- default
DISTRIBUTED_STORAGE_PREFERENCE:
- default
ENTERPRISE_LOGO_URL: /static/img/quay-horizontal-color.svg
EXTERNAL_TLS_TERMINATION: true
FEATURE_ACTION_LOG_ROTATION: false
FEATURE_ANONYMOUS_ACCESS: true
FEATURE_APP_SPECIFIC_TOKENS: true
FEATURE_BITBUCKET_BUILD: false
FEATURE_BLACKLISTED_EMAILS: false
FEATURE_BUILD_SUPPORT: true
FEATURE_CHANGE_TAG_EXPIRATION: true
FEATURE_DIRECT_LOGIN: true
FEATURE_EXTENDED_REPOSITORY_NAMES: true
FEATURE_FIPS: false
FEATURE_GENERAL_OCI_SUPPORT: true
FEATURE_GITHUB_BUILD: true
FEATURE_GITHUB_LOGIN: false
FEATURE_GITLAB_BUILD: false
FEATURE_GOOGLE_LOGIN: false
FEATURE_HELM_OCI_SUPPORT: false
FEATURE_INVITE_ONLY_USER_CREATION: false
FEATURE_MAILING: false
FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
FEATURE_PARTIAL_USER_AUTOCOMPLETE: true
FEATURE_PROXY_STORAGE: false
FEATURE_REPO_MIRROR: true
FEATURE_SECURITY_NOTIFICATIONS: true
FEATURE_SECURITY_SCANNER: true
FEATURE_SIGNING: false
FEATURE_STORAGE_REPLICATION: false
FEATURE_TEAM_SYNCING: false
FEATURE_USER_CREATION: true
FEATURE_USER_INITIALIZE: true
FEATURE_USER_LAST_ACCESSED: true
FEATURE_USER_LOG_ACCESS: false
FEATURE_USER_METADATA: false
FEATURE_USER_RENAME: false
FEATURE_USERNAME_CONFIRMATION: true
FRESH_LOGIN_TIMEOUT: 10m
GITHUB_LOGIN_CONFIG: {}
GITHUB_TRIGGER_CONFIG:
  API_ENDPOINT: https://api.github.com/
  CLIENT_ID: 904cb364cd6e9ac7b130
  CLIENT_SECRET: ******
  GITHUB_ENDPOINT: https://github.com/
GITLAB_TRIGGER_KIND: {}
GPG2_PRIVATE_KEY_FILENAME: signing-private.gpg
GPG2_PUBLIC_KEY_FILENAME: signing-public.gpg
LDAP_ALLOW_INSECURE_FALLBACK: false
LDAP_EMAIL_ATTR: mail
LDAP_UID_ATTR: uid
LDAP_URI: ldap://localhost
LOGS_MODEL: database
LOGS_MODEL_CONFIG: {}
MAIL_DEFAULT_SENDER: support@quay.io
MAIL_PORT: 587
MAIL_USE_AUTH: false
MAIL_USE_TLS: false
PREFERRED_URL_SCHEME: https
REGISTRY_TITLE: Quay
REGISTRY_TITLE_SHORT: Quay
REPO_MIRROR_INTERVAL: 30
REPO_MIRROR_TLS_VERIFY: true
SEARCH_MAX_RESULT_PAGE_COUNT: 10
SEARCH_RESULTS_PER_PAGE: 10
SECRET_KEY: SC0Su6zaq6TAnmXkGWgLwdfR6KXLOPcTTA8E12LyoGNEOkd3hzPvIn0fra-cUKH3h-sAqa0-sFwXCcQm
SECURITY_SCANNER_INDEXING_INTERVAL: 30
SECURITY_SCANNER_V4_ENDPOINT: http://quay360-clair-app:80
SECURITY_SCANNER_V4_NAMESPACE_WHITELIST:
- admin
SECURITY_SCANNER_V4_PSK: c0xuZFlGcEdoS3NqNENJZktmQldJRk14cEs0MUpCR0c=
SERVER_HOSTNAME: quayv360.apps.quay-perf-738.perfscale.devcluster.openshift.com
SETUP_COMPLETE: true
SUPER_USERS:
- quay
- admin
TAG_EXPIRATION_OPTIONS:
- 2w
- 4w
- 8w
TEAM_RESYNC_STALE_TIME: 60m
TESTING: false
USER_EVENTS_REDIS:
  host: quay360-quay-redis
  port: 6379
USER_RECOVERY_TOKEN_LIFETIME: 30m
BUILDMAN_HOSTNAME: quay360-quay-builder-quay360.apps.quay-perf-738.perfscale.devcluster.openshift.com:443
BUILD_MANAGER:
- ephemeral
- ALLOWED_WORKER_COUNT: 10
  ORCHESTRATOR_PREFIX: buildman/production/
  ORCHESTRATOR:
    REDIS_HOST: quay360-quay-redis
  EXECUTORS:
  - EXECUTOR: kubernetes
    DEBUG: true   
    BUILDER_NAMESPACE: builder
    K8S_API_SERVER: api.ci-ln-5bb0jdb-86010.origin-ci-int-aws.dev.rhcloud.com:6443
    K8S_API_TLS_CA: /conf/stack/extra_ca_cert_K8S_API_TLS_CA.crt
    VOLUME_SIZE: 8G
    KUBERNETES_DISTRIBUTION: openshift
    CONTAINER_MEMORY_LIMITS: 5120Mi
    CONTAINER_CPU_LIMITS: 1000m
    CONTAINER_MEMORY_REQUEST: 3968Mi
    CONTAINER_CPU_REQUEST: 500m
    NODE_SELECTOR_LABEL_KEY: node-role.kubernetes.io/worker
    NODE_SELECTOR_LABEL_VALUE: ''
    CONTAINER_RUNTIME: podman
    SERVICE_ACCOUNT_NAME: quay-builder
    SERVICE_ACCOUNT_TOKEN: eyJhbGciOiJSUzI1NiIsImtpZCI6InNDdUFxRXJPWnMwT0FVYjZ3ZUN6U1BJMXh4SGRhOUFkU1l2ZWdONHQ4TlUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJidWlsZGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InF1YXktYnVpbGRlci10b2tlbi16cW1oOCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJxdWF5LWJ1aWxkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI3YTRmMWYwYi0xOGYxLTQ1ZWEtYWVjOC0wOTQ5YmI0YjkwYzIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6YnVpbGRlcjpxdWF5LWJ1aWxkZXIifQ.atfFMTB-SCD-d-5ExRM-FfTmjSMgLv6A_TnmkswuYJmnmoeY_rOWC-v2UZr9ePtypbjMBac5UW_jX0ZlOQ95OySPtoasgY7Hw5oAdf6JF1o2kVRIhtBQuSZMUxLu2K045PacvlnGsZX3f1uKHocnHXJi-Xf7UU72cCAPbAAmrL04HPRomvxihy7YCNijx8Yn0VlYNKNJTKBeDXHmrV8rX541K-F2StMAbcw8U5mzrOPYxOhU-QylwRp1kBAnTjm1v58_jg0HdxEYoDxCz5I62eAHvEJbZdYSeYCU7332KBA9P20UNdM1Dzfz8SGWVc7PBsFJ3OUX6tcccEZNizgGxX1G-BIa4ROrzKAXM_4wceU30cspgnmfb2iYSsTeiA50A_D5KWTi4geYXhn3xZweC3sPTWyyiFcygNNBFiHd0VMIRR85EyGiVc_oeVGdMCL7HaSJY5kLVrcMRO6FL809KDRcDtOz0YCYJAs5W46rdSIWNtLTy7bsVWIkyOlxbYC82nOJucAe-3FAyyu4rLs2Rt3J7q1IKBmxCmOW_U58-pU7kiZGz3bZPkG95YQAvJc69N0zzUkTWa_a3CDfHFVEBk6wZZYOvE4kg9-PpbR_a6uLQovWHjwRAGNxpoPewRrk65cgPXWNmdKC87apnNGiRh5FBGDD1ZU70WJwqm0I2lI
    QUAY_USERNAME: '|76db245a-0d75-11e7-a544-28d244ea5a6d.j9is.751956'
    QUAY_PASSWORD: ******
    WORKER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8
    WORKER_TAG: v3.6.0-31
    BUILDER_VM_CONTAINER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-qemu-rhcos-rhel8:v3.6.0-32
    SETUP_TIME: 180
    MINIMUM_RETRY_THRESHOLD: 0
    SSH_AUTHORIZED_KEYS:
    - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN8nUzLnPHq9o6Crika8brT4i5CL0a0azoHJoHe02BH8/vgDyhgHin+1qDrHA414t6smDIhYRM/L503J0kD2/jUTPVqeFNmbxbzXnEXWv2RaAyKChMzw2PkrKiLntY4CxcukdSN6lqtJa8TH3/Vmy/YUOMJOKWEsYkg6qojDWPYbFHMubm6JWPydiEJJYPYCH7tHPaq4Y3CWNw+jx2sL69Sltnsdc/oj5Icl+u/ClF7lm0LPXkrkUF745ktCg6r06dLju3Ap+A0HJ/doTpCymZrt88eEy0RqW9koDYPJsRm380caT0J4wux3HlZiHP0b1mhx9pp7DB0FuhZHxeQawGs4V3aYDisBE27YMoMBqoCmBOqkVqC7uY47HOYiS15YHpriCXSnflE628e6a7zfFVV+CcrcqtcqPltZlXmbm2PeQY547VphB1nivinALOVM+CcSgOchX1Phmj63nXKt/IbsUJhUnZQicFhh2bJzXWKBtCQkodwTnu90RaKJN2pn8= lizhang@lzha-mac

Steps:

Deploy Quay with Operator, choose managed route and tls, and choose unmanaged objectstorage to use AWS S3
Login Quay config editor to enable builder
Following Quay docs https://access.redhat.com/documentation/en-us/red_hat_quay/3.5/html-single/use_red_hat_quay/index#setting-up-builders to create new config.yaml and other required config on OCP Baremetal Cluster to run build job
Login Quay to create Github trigger
Trigger new image build job

Expected Results:

Image build job completed successfully.

Actual Results:

Image build job was failed at start builder job.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

image-2021-09-26-20-37-46-044.png
308 kB
2021/09/26 12:37 PM
image-2021-10-09-10-54-47-429.png
800 kB
2021/10/09 2:54 AM
quay360_app_pod1.logs
95 kB
2021/09/15 9:09 AM
quayregistry-quay-app-6d5445778f-pqw25.log
2.33 MB
2022/05/19 6:31 AM

Issue Links

is related to

PROJQUAY-2657 Quay 3.6.0 docs should guide how to setup builder when use unmanaged tls component

Closed

PROJQUAY-2668 Quay 3.6.0 docs should provide solution for build image when using managed tls

Closed

PROJQUAY-3690 TypeError: quote_from_bytes() expected bytes

Closed

relates to

PROJQUAY-2559 Expose a non-secure gRPC port for externally TLS terminated Quay deployment

New

PROJQUAY-2423 Document known issues in release notes for v3.6

Closed

Quay 3.6.0 was failed to start image builder job

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates