Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-2545

Quay 3.6.0 was failed to start image builder job

    XMLWordPrintable

Details

    • ToDo

    Description

      Description:

      This is an issue found when use Quay 3.6.0 to build image from Github repo, but found the build job was failed to start on OCP baremetal Cluster, checked Quay APP POD found error "TypeError: quote_from_bytes() expected bytes", see attached Quay APP POD logs quay360_app_pod1.logs 

      Quay image: "quay-operator-bundle-container-v3.6.0-32"

      Note: This issue is not existed with Quay 3.5.6

      builder stdout | 2021-09-15 03:57:10,555 [60] [ERROR] [buildman.manager.ephemeral] Exception when starting builder for job: buildman/production/building/73a7a334-e43f-4524-b712-bf25807318d2 - Falling back to next configured executor
      builder stdout | Traceback (most recent call last):
      builder stdout |   File "/quay-registry/buildman/manager/ephemeral.py", line 650, in schedule
      builder stdout |     execution_id = executor.start_builder(registration_token, build_job.build_uuid)
      builder stdout |   File "/quay-registry/buildman/manager/executor.py", line 56, in wrapper
      builder stdout |     rv = func(*args, **kwargs)
      builder stdout |   File "/quay-registry/buildman/manager/executor.py", line 638, in start_builder
      builder stdout |     user_data = self.generate_cloud_config(token, build_uuid, self.manager_hostname)
      builder stdout |   File "/quay-registry/buildman/manager/executor.py", line 197, in generate_cloud_config
      builder stdout |     TEMPLATE.render(
      builder stdout |   File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
      builder stdout |     self.environment.handle_exception()
      builder stdout |   File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
      builder stdout |     reraise(*rewrite_traceback_stack(source=source))
      builder stdout |   File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
      builder stdout |     raise value.with_traceback(tb)
      builder stdout |   File "/quay-registry/buildman/templates/cloudconfig.json", line 82, in top-level template code
      builder stdout |     "source": {{ ca_cert | dataurl | jsonify }}
      builder stdout |   File "/quay-registry/buildman/container_cloud_config.py", line 115, in data_url
      builder stdout |     data = "," + urlquote(content)
      builder stdout |   File "/usr/lib64/python3.8/urllib/parse.py", line 839, in quote
      builder stdout |     return quote_from_bytes(string, safe)
      builder stdout |   File "/usr/lib64/python3.8/urllib/parse.py", line 864, in quote_from_bytes
      builder stdout |     raise TypeError("quote_from_bytes() expected bytes")
      builder stdout | TypeError: quote_from_bytes() expected bytes
      builder stdout | 2021-09-15 03:57:10,556 [60] [ERROR] [buildman.manager.ephemeral] Could not start ephemeral worker for build 73a7a334-e43f-4524-b712-bf25807318d2
      

      Config.yaml:

      ALLOW_PULLS_WITHOUT_STRICT_LOGGING: false
      ALLOWED_OCI_ARTIFACT_TYPES:
        application/vnd.cncf.helm.config.v1+json:
        - application/tar+gzip
        application/vnd.oci.image.layer.v1.tar+gzip+encrypted:
        - application/vnd.oci.image.layer.v1.tar+gzip+encrypted
      AUTHENTICATION_TYPE: Database
      AVATAR_KIND: local
      BROWSER_API_CALLS_XHR_ONLY: false
      BUILDLOGS_REDIS:
        host: quay360-quay-redis
        port: 6379
      CREATE_REPOSITORY_ON_PUSH_PUBLIC: true
      DATABASE_SECRET_KEY: eTAI6czN1zpnzP2jhE1hChXHIWETXzoKoX0Yf2sLkK1xwkroAOfU3a71q0MsNoqkM8uEd1eeBmh1vlHN
      DB_CONNECTION_ARGS:
        autorollback: true
        threadlocals: true
      DB_URI: postgresql://quay360-quay-database:UwbR2guNmm7lfarAHYqnzPEoTaA07h3qn7uyp2SSQIhJfL9aA3Oc5Ye--pat2Zu-qp51QWClmcdWvuiQ@quay360-quay-database:5432/quay360-quay-database
      DEFAULT_TAG_EXPIRATION: 4w
      DISTRIBUTED_STORAGE_CONFIG:
        default:
        - S3Storage
        - host: s3.us-east-2.amazonaws.com
          s3_access_key: ******
          s3_bucket: quay360
          s3_secret_key: ******
          storage_path: /quay360
      DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
      - default
      DISTRIBUTED_STORAGE_PREFERENCE:
      - default
      ENTERPRISE_LOGO_URL: /static/img/quay-horizontal-color.svg
      EXTERNAL_TLS_TERMINATION: true
      FEATURE_ACTION_LOG_ROTATION: false
      FEATURE_ANONYMOUS_ACCESS: true
      FEATURE_APP_SPECIFIC_TOKENS: true
      FEATURE_BITBUCKET_BUILD: false
      FEATURE_BLACKLISTED_EMAILS: false
      FEATURE_BUILD_SUPPORT: true
      FEATURE_CHANGE_TAG_EXPIRATION: true
      FEATURE_DIRECT_LOGIN: true
      FEATURE_EXTENDED_REPOSITORY_NAMES: true
      FEATURE_FIPS: false
      FEATURE_GENERAL_OCI_SUPPORT: true
      FEATURE_GITHUB_BUILD: true
      FEATURE_GITHUB_LOGIN: false
      FEATURE_GITLAB_BUILD: false
      FEATURE_GOOGLE_LOGIN: false
      FEATURE_HELM_OCI_SUPPORT: false
      FEATURE_INVITE_ONLY_USER_CREATION: false
      FEATURE_MAILING: false
      FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
      FEATURE_PARTIAL_USER_AUTOCOMPLETE: true
      FEATURE_PROXY_STORAGE: false
      FEATURE_REPO_MIRROR: true
      FEATURE_SECURITY_NOTIFICATIONS: true
      FEATURE_SECURITY_SCANNER: true
      FEATURE_SIGNING: false
      FEATURE_STORAGE_REPLICATION: false
      FEATURE_TEAM_SYNCING: false
      FEATURE_USER_CREATION: true
      FEATURE_USER_INITIALIZE: true
      FEATURE_USER_LAST_ACCESSED: true
      FEATURE_USER_LOG_ACCESS: false
      FEATURE_USER_METADATA: false
      FEATURE_USER_RENAME: false
      FEATURE_USERNAME_CONFIRMATION: true
      FRESH_LOGIN_TIMEOUT: 10m
      GITHUB_LOGIN_CONFIG: {}
      GITHUB_TRIGGER_CONFIG:
        API_ENDPOINT: https://api.github.com/
        CLIENT_ID: 904cb364cd6e9ac7b130
        CLIENT_SECRET: ******
        GITHUB_ENDPOINT: https://github.com/
      GITLAB_TRIGGER_KIND: {}
      GPG2_PRIVATE_KEY_FILENAME: signing-private.gpg
      GPG2_PUBLIC_KEY_FILENAME: signing-public.gpg
      LDAP_ALLOW_INSECURE_FALLBACK: false
      LDAP_EMAIL_ATTR: mail
      LDAP_UID_ATTR: uid
      LDAP_URI: ldap://localhost
      LOGS_MODEL: database
      LOGS_MODEL_CONFIG: {}
      MAIL_DEFAULT_SENDER: support@quay.io
      MAIL_PORT: 587
      MAIL_USE_AUTH: false
      MAIL_USE_TLS: false
      PREFERRED_URL_SCHEME: https
      REGISTRY_TITLE: Quay
      REGISTRY_TITLE_SHORT: Quay
      REPO_MIRROR_INTERVAL: 30
      REPO_MIRROR_TLS_VERIFY: true
      SEARCH_MAX_RESULT_PAGE_COUNT: 10
      SEARCH_RESULTS_PER_PAGE: 10
      SECRET_KEY: SC0Su6zaq6TAnmXkGWgLwdfR6KXLOPcTTA8E12LyoGNEOkd3hzPvIn0fra-cUKH3h-sAqa0-sFwXCcQm
      SECURITY_SCANNER_INDEXING_INTERVAL: 30
      SECURITY_SCANNER_V4_ENDPOINT: http://quay360-clair-app:80
      SECURITY_SCANNER_V4_NAMESPACE_WHITELIST:
      - admin
      SECURITY_SCANNER_V4_PSK: c0xuZFlGcEdoS3NqNENJZktmQldJRk14cEs0MUpCR0c=
      SERVER_HOSTNAME: quayv360.apps.quay-perf-738.perfscale.devcluster.openshift.com
      SETUP_COMPLETE: true
      SUPER_USERS:
      - quay
      - admin
      TAG_EXPIRATION_OPTIONS:
      - 2w
      - 4w
      - 8w
      TEAM_RESYNC_STALE_TIME: 60m
      TESTING: false
      USER_EVENTS_REDIS:
        host: quay360-quay-redis
        port: 6379
      USER_RECOVERY_TOKEN_LIFETIME: 30m
      BUILDMAN_HOSTNAME: quay360-quay-builder-quay360.apps.quay-perf-738.perfscale.devcluster.openshift.com:443
      BUILD_MANAGER:
      - ephemeral
      - ALLOWED_WORKER_COUNT: 10
        ORCHESTRATOR_PREFIX: buildman/production/
        ORCHESTRATOR:
          REDIS_HOST: quay360-quay-redis
        EXECUTORS:
        - EXECUTOR: kubernetes
          DEBUG: true   
          BUILDER_NAMESPACE: builder
          K8S_API_SERVER: api.ci-ln-5bb0jdb-86010.origin-ci-int-aws.dev.rhcloud.com:6443
          K8S_API_TLS_CA: /conf/stack/extra_ca_cert_K8S_API_TLS_CA.crt
          VOLUME_SIZE: 8G
          KUBERNETES_DISTRIBUTION: openshift
          CONTAINER_MEMORY_LIMITS: 5120Mi
          CONTAINER_CPU_LIMITS: 1000m
          CONTAINER_MEMORY_REQUEST: 3968Mi
          CONTAINER_CPU_REQUEST: 500m
          NODE_SELECTOR_LABEL_KEY: node-role.kubernetes.io/worker
          NODE_SELECTOR_LABEL_VALUE: ''
          CONTAINER_RUNTIME: podman
          SERVICE_ACCOUNT_NAME: quay-builder
          SERVICE_ACCOUNT_TOKEN: eyJhbGciOiJSUzI1NiIsImtpZCI6InNDdUFxRXJPWnMwT0FVYjZ3ZUN6U1BJMXh4SGRhOUFkU1l2ZWdONHQ4TlUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJidWlsZGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InF1YXktYnVpbGRlci10b2tlbi16cW1oOCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJxdWF5LWJ1aWxkZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI3YTRmMWYwYi0xOGYxLTQ1ZWEtYWVjOC0wOTQ5YmI0YjkwYzIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6YnVpbGRlcjpxdWF5LWJ1aWxkZXIifQ.atfFMTB-SCD-d-5ExRM-FfTmjSMgLv6A_TnmkswuYJmnmoeY_rOWC-v2UZr9ePtypbjMBac5UW_jX0ZlOQ95OySPtoasgY7Hw5oAdf6JF1o2kVRIhtBQuSZMUxLu2K045PacvlnGsZX3f1uKHocnHXJi-Xf7UU72cCAPbAAmrL04HPRomvxihy7YCNijx8Yn0VlYNKNJTKBeDXHmrV8rX541K-F2StMAbcw8U5mzrOPYxOhU-QylwRp1kBAnTjm1v58_jg0HdxEYoDxCz5I62eAHvEJbZdYSeYCU7332KBA9P20UNdM1Dzfz8SGWVc7PBsFJ3OUX6tcccEZNizgGxX1G-BIa4ROrzKAXM_4wceU30cspgnmfb2iYSsTeiA50A_D5KWTi4geYXhn3xZweC3sPTWyyiFcygNNBFiHd0VMIRR85EyGiVc_oeVGdMCL7HaSJY5kLVrcMRO6FL809KDRcDtOz0YCYJAs5W46rdSIWNtLTy7bsVWIkyOlxbYC82nOJucAe-3FAyyu4rLs2Rt3J7q1IKBmxCmOW_U58-pU7kiZGz3bZPkG95YQAvJc69N0zzUkTWa_a3CDfHFVEBk6wZZYOvE4kg9-PpbR_a6uLQovWHjwRAGNxpoPewRrk65cgPXWNmdKC87apnNGiRh5FBGDD1ZU70WJwqm0I2lI
          QUAY_USERNAME: '|76db245a-0d75-11e7-a544-28d244ea5a6d.j9is.751956'
          QUAY_PASSWORD: ******
          WORKER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8
          WORKER_TAG: v3.6.0-31
          BUILDER_VM_CONTAINER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-qemu-rhcos-rhel8:v3.6.0-32
          SETUP_TIME: 180
          MINIMUM_RETRY_THRESHOLD: 0
          SSH_AUTHORIZED_KEYS:
          - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN8nUzLnPHq9o6Crika8brT4i5CL0a0azoHJoHe02BH8/vgDyhgHin+1qDrHA414t6smDIhYRM/L503J0kD2/jUTPVqeFNmbxbzXnEXWv2RaAyKChMzw2PkrKiLntY4CxcukdSN6lqtJa8TH3/Vmy/YUOMJOKWEsYkg6qojDWPYbFHMubm6JWPydiEJJYPYCH7tHPaq4Y3CWNw+jx2sL69Sltnsdc/oj5Icl+u/ClF7lm0LPXkrkUF745ktCg6r06dLju3Ap+A0HJ/doTpCymZrt88eEy0RqW9koDYPJsRm380caT0J4wux3HlZiHP0b1mhx9pp7DB0FuhZHxeQawGs4V3aYDisBE27YMoMBqoCmBOqkVqC7uY47HOYiS15YHpriCXSnflE628e6a7zfFVV+CcrcqtcqPltZlXmbm2PeQY547VphB1nivinALOVM+CcSgOchX1Phmj63nXKt/IbsUJhUnZQicFhh2bJzXWKBtCQkodwTnu90RaKJN2pn8= lizhang@lzha-mac
      

      Steps:

      1. Deploy Quay with Operator, choose managed route and tls, and choose unmanaged objectstorage to use AWS S3
      2. Login Quay config editor to enable builder
      3. Following Quay docs https://access.redhat.com/documentation/en-us/red_hat_quay/3.5/html-single/use_red_hat_quay/index#setting-up-builders to create new config.yaml and other required config on OCP Baremetal Cluster to run build job
      4. Login Quay to create Github trigger
      5. Trigger new image build job

      Expected Results:

      Image build job completed successfully.

      Actual Results:

      Image build job was failed at start builder job.

      Attachments

        Issue Links

          Activity

            People

              sleesinc Kenny Lee Sin Cheong
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: