Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-1283

Quay build manager didn't launch the expected EC2 instance and build worker

XMLWordPrintable

    • False
    • False
    • Undefined

      Description:

      This is an issue found when config Quay build manager to launch quay builder worker, following docs deploy Quay with TNG Operator, provided required configuration "FEATURE_BUILD_SUPPORT: true" and "Github trigger config", see config,yaml as following. After quay was deployed successfully, edit the quay-config-secret, to add required BUIILD_MANAGER configuration, redeploy Quay POD. The result after create new build Job, the Job is stuck at "Waiting for available build worker", checked AWS Console, the expected EC2 instance was not provisioning.

       Docs: https://github.com/quay/quay/blob/master/buildman/README.md 

       Quay Image:

      lizhang@lzha-mac quay3.4 % oc get pod quaybuilder-quay-app-565cf8c5f5-7w9m2 -o json | jq '.spec.containers[0].image'
"registry.redhat.io/quay/quay@sha256:12434420fa3ca1df4654bf6d08c8980b1209b13fbfe8336fd9b1ef6f04cc4c75"

       

      Quay config.yaml to deploy quay with TNG Operator:

      SUPER_USERS:
  - quay
  - admin
DISTRIBUTED_STORAGE_CONFIG:
    default:
      - S3Storage
      - host: s3.us-east-2.amazonaws.com
        port: "443"
        s3_access_key: ****
        s3_bucket: quay340
        s3_secret_key: ****
        storage_path: /quay3401113
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
  - default
DISTRIBUTED_STORAGE_PREFERENCE:
  - default
FEATURE_BUILD_SUPPORT: true
FEATURE_GITHUB_BUILD: true
GITHUB_TRIGGER_CONFIG:
    API_ENDPOINT: https://api.github.com/
    CLIENT_ID: ****
    CLIENT_SECRET: ****
    GITHUB_ENDPOINT: https://github.com/

      Updated quay config.yaml from quay-config-secret:

      ALLOW_PULLS_WITHOUT_STRICT_LOGGING: false
AUTHENTICATION_TYPE: Database
BUILDLOGS_REDIS:
  host: quaybuilder-quay-redis
  port: 6379
DATABASE_SECRET_KEY: 1AC0q8i4voTx2vb3-XfVkFGTJbSENTJlfqonX1EeyUnsL6ROHagGDfB2WGha00NPUmkqc333lpBSUdlV
DB_CONNECTION_ARGS:
  autorollback: true
  threadlocals: true
DB_URI: postgresql://quaybuilder-quay-database:postgres@quaybuilder-quay-database:5432/quaybuilder-quay-database
DEFAULT_TAG_EXPIRATION: 2w
DISTRIBUTED_STORAGE_CONFIG:
  default:
  - S3Storage
  - host: s3.us-east-2.amazonaws.com
    port: "443"
    s3_access_key: ****
    s3_bucket: quay340
    s3_secret_key: ****
    storage_path: /quay3401113
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
- default
DISTRIBUTED_STORAGE_PREFERENCE:
- default
ENTERPRISE_LOGO_URL: /static/img/quay-horizontal-color.svg
FEATURE_BUILD_SUPPORT: true
FEATURE_DIRECT_LOGIN: true
FEATURE_GITHUB_BUILD: true
FEATURE_MAILING: false
FEATURE_REPO_MIRROR: true
FEATURE_SECURITY_NOTIFICATIONS: true
FEATURE_SECURITY_SCANNER: true
GITHUB_TRIGGER_CONFIG:
  API_ENDPOINT: https://api.github.com/
  CLIENT_ID: ****
  CLIENT_SECRET: ****
  GITHUB_ENDPOINT: https://github.com/
PREFERRED_URL_SCHEME: https
REGISTRY_TITLE: Quay
REGISTRY_TITLE_SHORT: Quay
REPO_MIRROR_INTERVAL: 30
REPO_MIRROR_TLS_VERIFY: true
SECRET_KEY: M-b4mukLnJGoD6PjzQuBsNjYar0cQCJylvK5NKXaubQtixStW3JnKMYSvYeWeBy7gRi1fhC2vjwhgY2V
SECURITY_SCANNER_INDEXING_INTERVAL: 30
SECURITY_SCANNER_V4_ENDPOINT: http://quaybuilder-clair:80
SECURITY_SCANNER_V4_NAMESPACE_WHITELIST:
- admin
SECURITY_SCANNER_V4_PSK: RHZvMkZma1hjS1lpNjVIV0hPS2dTczVlRTA0Y0plNTc=
SERVER_HOSTNAME: quaybuilder-quay-quaybuilder.apps.quay-perf-399.perf-testing.devcluster.openshift.com
SETUP_COMPLETE: true
SUPER_USERS:
- quay
- admin
TAG_EXPIRATION_OPTIONS:
- 2w
TEAM_RESYNC_STALE_TIME: 60m
TESTING: false
USER_EVENTS_REDIS:
  host: quaybuilder-quay-redis
  port: 6379
BUIILD_MANAGER:
- ephemeral
- ORCHESTRATOR_PREFIX: buildman/production/
  ORCHESTRATOR:
    REDIS_HOST: quayldap.perf-testing.devcluster.openshift.com
    REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
ALLOWED_WORKER_COUNT: 10
EXECUTORS:
  - EXECUTOR: ec2
    QUAY_USERNAME: quay
    QUAY_PASSWORD: password
    WORKER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder@sha256:33edfa733680e05a2e0a5db6ece3493f561738d528f2cc766dc2f9dd75b8bb6c
    WORKER_TAG: master
    SETUP_TIME: 180 
    MINIMUM_RETRY_THRESHOLD: 1 
    EC2_REGION: us-east-1
    COREOS_AMI: ami-02545325b519192df
    AWS_ACCESS_KEY: ****
    AWS_SECRET_KEY: ****
    EC2_INSTANCE_TYPE: t2.large
    EC2_VPC_SUBNET_ID: subnet-0af5f148f51e27743
    EC2_SECURITY_GROUP_IDS:
    - sg-08f48936517b03a70
    EC2_KEY_NAME: quaybuild1113
    BLOCK_DEVICE_SIZE: 58

      Steps:

      1. Deploy Quay with TNG Operator with including GitHub Trigger Config settings
      2. Login Quay and create new super user quay
      3. decode the config.yaml from quay-config-secret
      4. add BUIILD_MANAGER to the config.yaml and encode with base64
      5. Update quay-config-secret to include "BUIILD_MANAGER" with "oc edit secretquaybuilder-quay-config-secret-k98khc4kf5"
      6. Redeploy Quay POD
      7. Create new image repository
      8. Create new Github build trigger, fill all required settings
      9. Submit commit to the Github repository
      10. Wait to see new build generated
      11. The build is stuck at "Waiting for available build worker"
      12. Check AWS EC2 console

      Expected Results:

      New AWS EC2 instance should be provisioning and new build worker was running to process the build job

      Actual Results:

      not found the expected EC2 instance was provisioning, and build job is stuck at "Waiting for available build worker" 

              sleesinc Kenny Lee Sin Cheong
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: