Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-5910

The latest build worker image(v3.9.0-18, v3.8.11-3) can't start in bare metal build environment

    XMLWordPrintable

Details

    • False
    • None
    • False
    • 0

    Description

      Description of problem:

      The latest build worker image (quay-builder-container-v3.9.0-18) can't start in bare metal build environment.

      Version-Release number of selected component (if applicable):

      quay-operator-bundle-container-v3.9.0-155
      Index image v4.13: registry-proxy.engineering.redhat.com/rh-osbs/iib:548254
      registry.redhat.io/quay/quay-operator-rhel8@sha256:984cebc8817cf8617052bca3932444ef761ef2d06fbbe15d2048ea26d2b8aea3
      registry.redhat.io/quay/quay-rhel8@sha256:e802dd2c6cf63bb20aaded3f419dab86ee9fa069b3d5ff7931c1a0ca4d11ed6b
      quay-builder-container-v3.9.0-18

      How reproducible:

      always

      Steps to Reproduce:
      1. set up a bare metal build environment 

      FEATURE_BUILD_SUPPORT: True
      FEATURE_GITLAB_BUILD: true
      GITLAB_TRIGGER_CONFIG:
          CLIENT_ID: e...e 
          CLIENT_SECRET: c...9 
      GITLAB_TRIGGER_KIND:
          GITLAB_ENDPOINT: https://gitlab.com/
      FEATURE_BITBUCKET_BUILD: true
      BITBUCKET_TRIGGER_CONFIG:
          CONSUMER_KEY: p...e 
          CONSUMER_SECRET: 7..J 
      FEATURE_GITHUB_BUILD: true
      GITHUB_TRIGGER_CONFIG:
        API_ENDPOINT: https://api.github.com/
        CLIENT_ID: 2...d 
        CLIENT_SECRET:  3...8 
        GITHUB_ENDPOINT: https://github.com/
      BUILDMAN_HOSTNAME: quayregistry-quay-builder-quay-enterprise.apps.whu413osp02.qe.devcluster.openshift.com:443
      BUILD_MANAGER:
      - ephemeral
      - ALLOWED_WORKER_COUNT: 3 
        ORCHESTRATOR_PREFIX: buildman/production/
        JOB_REGISTRATION_TIMEOUT: 600
        ORCHESTRATOR:
          REDIS_HOST: quayregistry-quay-redis
          REDIS_PASSWORD: "" 
          REDIS_SSL: false
          REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
        EXECUTORS:
        - EXECUTOR: kubernetes
          DEBUG: true
          K8S_API_SERVER: api.whu412bm01.qe.devcluster.openshift.com:6443 
          K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build_cluster.crt
          VOLUME_SIZE: 8G
          KUBERNETES_DISTRIBUTION: openshift
          CONTAINER_MEMORY_LIMITS: 5120Mi
          CONTAINER_CPU_LIMITS: 1000m
          CONTAINER_MEMORY_REQUEST: 3968Mi
          CONTAINER_CPU_REQUEST: 500m
          NODE_SELECTOR_LABEL_KEY: type 
          NODE_SELECTOR_LABEL_VALUE: builder 
          CONTAINER_RUNTIME: podman
          BUILDER_NAMESPACE: builder 
          SERVICE_ACCOUNT_NAME:  quay-builder  
          SERVICE_ACCOUNT_TOKEN: e...w
          QUAY_USERNAME: '..6'
          QUAY_PASSWORD: ey...8 
          WORKER_IMAGE:  brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8
          WORKER_TAG: v3.9.0 
          BUILDER_VM_CONTAINER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-qemu-rhcos-rhel8:v3.9.0
          SETUP_TIME: 180
          MINIMUM_RETRY_THRESHOLD: 0
          SSH_AUTHORIZED_KEYS:
          - ssh-rsa A.....hine
      USERFILES_LOCATION: default
      USERFILES_PATH: userfiles
      DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: 
        - default
      DISTRIBUTED_STORAGE_PREFERENCE:
        - default
      DISTRIBUTED_STORAGE_CONFIG:
        default:
            - SwiftStorage
            - auth_url: https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13000/v3/
              auth_version: "3"
              os_options:
                  region_name: regionOne
                  tenant_id: 54.....0
                  user_domain_name: redhat.com
              storage_path: /datastorage/registry
              swift_container: quay
              swift_password: e.....e
              swift_user: p...s
      SUPER_USERS:
        - whuquay
      FEATURE_ANONYMOUS_ACCESS: true 
      FEATURE_QUOTA_MANAGEMENT: true
      FEATURE_PROXY_CACHE: true
      BROWSER_API_CALLS_XHR_ONLY: false
      CREATE_PRIVATE_REPO_ON_PUSH: true
      CREATE_NAMESPACE_ON_PUSH: true
      FEATURE_USERNAME_CONFIRMATION: false
      FEATURE_USER_INITIALIZE: true 
      AUTHENTICATION_TYPE: Database
      FEATURE_UI_V2: true
      FEATURE_LISTEN_IP_VERSION: IPv4
      FEATURE_SUPERUSERS_FULL_ACCESS: true
      QUOTA_BACKFILL: true
      FEATURE_GARBAGE_COLLECTION: true
      GARBAGE_COLLECTION_FREQUENCY: 10
      DEFAULT_TAG_EXPIRATION: 2w
      TAG_EXPIRATION_OPTIONS:
      - 2w
      - 10s
      FEATURE_SECURITY_SCANNER_NOTIFY_ON_NEW_INDEX: True
      FEATURE_GENERAL_OCI_SUPPORT: true

       

      2. configure a github build trigger

      3. trigger the build process

      Actual results:

      The build worker container can't start in builder VM container. 

      $ journalctl -f -u quay-builder
      -- Logs begin at Thu 2023-08-03 08:56:46 UTC. --
      Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=info msg="pinging buildmanager..."
      Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=info msg="starting log stream to buildmanager"
      Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=info msg="registering job for registration token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImdKaVBTYXNZMzAtNGpxbnBkS2tXSzhYWnZoUHdqOUQ5aXRSTU51SWdGZG8ifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktYnVpbGRlci1xdWF5LWVudGVycHJpc2UuYXBwcy53aHU0MTNvc3AwMi5xZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb206NDQzIiwibmJmIjoxNjkxMDUyOTkwLCJpYXQiOjE2OTEwNTI5OTAsImV4cCI6MTY5MTA1MzYyMCwic3ViIjoiKGFub255bW91cykiLCJhY2Nlc3MiOnt9LCJjb250ZXh0Ijp7InRva2VuX3R5cGUiOiJidWlsZF9qb2JfcmVnaXN0cmF0aW9uIiwiYnVpbGRfaWQiOiJmMGU5YTY2Yy1iMTcyLTRkN2YtODEwYi0xODk5YTZmM2Y3NWEiLCJqb2JfaWQiOiJidWlsZG1hbi9wcm9kdWN0aW9uL2J1aWxkaW5nL2YwZTlhNjZjLWIxNzItNGQ3Zi04MTBiLTE4OTlhNmYzZjc1YSIsImV4cGlyYXRpb24iOjYzMH19.CtfoAnbzqQDWkheMGauZ3_zVpnngHTz4vL6rTiRCzQpAfTsqlGUyQwbbhp3OI7smgYiWeGEGHdOyerkkFdTlBmejiCRhXVNF-A_UmwH8Ly72UyB7XgCb1Odn2Zi9PilSPlHrCKY-LYWEpKb6RVogNhJ5Ws3tBMcicPfV8hJJjd1tCPEb91jSbqX6BN-9MYvMZlomSHFUHDkumucX4-g2uQXX5c6SzP8kF99YuJKgD3n1kSPn66cr5fyluR3fu63UeeLuDLpQnliPi3zU1yiC4ZOPTTFLkrIhImUSAMbtUklfqzRZO1FVbVWf4mydplPgwPxs-OdNFFIm-UeW50MyKg"
      Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=info msg="starting heartbeat to buildmanager"
      Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=info msg="starting build"
      Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=info msg="connecting to docker host: unix:/var/run/podman/podman.sock"
      Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=fatal msg="invalid endpoint"
      Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a systemd[1]: quay-builder.service: Main process exited, code=exited, status=1/FAILURE
      Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a systemd[1]: quay-builder.service: Failed with result 'exit-code'.
      Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a systemd[1]: Failed to start quay-builder.service.

      Expected results:

      The build process should finish successfully.

      Additional info:

      Don't change any configuration, just use brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.9.0-13 to replace the latest version v3.9.0-18,  everything work well. 

       

      The latest builder worker image v3.9.0 is v3.9.0-18

      $ skopeo inspect --raw --tls-verify=false --creds='....8 docker://brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.9.0  |jq .
      {
        "manifests": [
          {
            "digest": "sha256:677137792803636377413ec8d058d9b8ece2b75c91bb2a30fc435e78a9ee335e",
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "platform": {
              "architecture": "amd64",
              "os": "linux"
            },
            "size": 760
          }
        ],
        "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
        "schemaVersion": 2
      }
      
      $ skopeo inspect --raw --tls-verify=false --creds='....8 docker://brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.9.0-18  |jq .
      {
        "manifests": [
          {
            "digest": "sha256:677137792803636377413ec8d058d9b8ece2b75c91bb2a30fc435e78a9ee335e",
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "platform": {
              "architecture": "amd64",
              "os": "linux"
            },
            "size": 760
          }
        ],
        "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
        "schemaVersion": 2
      }
      

       

      Attachments

        Activity

          People

            obulatov@redhat.com Oleg Bulatov
            rhwhu Weihua Hu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: