Loading...

Type: Bug
Resolution: Done
Priority: Blocker
Fix Version/s: quay-v3.8.11, quay-v3.9.0, quay-v3.10.0
Affects Version/s: quay-v3.8.11, quay-v3.9.0
Component/s: quay-builder
Labels:
- release-blocker
- triaged

Blocked:
False
Blocked Reason:
None
Ready:
False
Intelligence Requested:
Market:

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem:

The latest build worker image (quay-builder-container-v3.9.0-18) can't start in bare metal build environment.

Version-Release number of selected component (if applicable):

quay-operator-bundle-container-v3.9.0-155
Index image v4.13: registry-proxy.engineering.redhat.com/rh-osbs/iib:548254
registry.redhat.io/quay/quay-operator-rhel8@sha256:984cebc8817cf8617052bca3932444ef761ef2d06fbbe15d2048ea26d2b8aea3
registry.redhat.io/quay/quay-rhel8@sha256:e802dd2c6cf63bb20aaded3f419dab86ee9fa069b3d5ff7931c1a0ca4d11ed6b
quay-builder-container-v3.9.0-18

How reproducible:

always

Steps to Reproduce:
1. set up a bare metal build environment

FEATURE_BUILD_SUPPORT: True
FEATURE_GITLAB_BUILD: true
GITLAB_TRIGGER_CONFIG:
    CLIENT_ID: e...e 
    CLIENT_SECRET: c...9 
GITLAB_TRIGGER_KIND:
    GITLAB_ENDPOINT: https://gitlab.com/
FEATURE_BITBUCKET_BUILD: true
BITBUCKET_TRIGGER_CONFIG:
    CONSUMER_KEY: p...e 
    CONSUMER_SECRET: 7..J 
FEATURE_GITHUB_BUILD: true
GITHUB_TRIGGER_CONFIG:
  API_ENDPOINT: https://api.github.com/
  CLIENT_ID: 2...d 
  CLIENT_SECRET:  3...8 
  GITHUB_ENDPOINT: https://github.com/
BUILDMAN_HOSTNAME: quayregistry-quay-builder-quay-enterprise.apps.whu413osp02.qe.devcluster.openshift.com:443
BUILD_MANAGER:
- ephemeral
- ALLOWED_WORKER_COUNT: 3 
  ORCHESTRATOR_PREFIX: buildman/production/
  JOB_REGISTRATION_TIMEOUT: 600
  ORCHESTRATOR:
    REDIS_HOST: quayregistry-quay-redis
    REDIS_PASSWORD: "" 
    REDIS_SSL: false
    REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
  EXECUTORS:
  - EXECUTOR: kubernetes
    DEBUG: true
    K8S_API_SERVER: api.whu412bm01.qe.devcluster.openshift.com:6443 
    K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build_cluster.crt
    VOLUME_SIZE: 8G
    KUBERNETES_DISTRIBUTION: openshift
    CONTAINER_MEMORY_LIMITS: 5120Mi
    CONTAINER_CPU_LIMITS: 1000m
    CONTAINER_MEMORY_REQUEST: 3968Mi
    CONTAINER_CPU_REQUEST: 500m
    NODE_SELECTOR_LABEL_KEY: type 
    NODE_SELECTOR_LABEL_VALUE: builder 
    CONTAINER_RUNTIME: podman
    BUILDER_NAMESPACE: builder 
    SERVICE_ACCOUNT_NAME:  quay-builder  
    SERVICE_ACCOUNT_TOKEN: e...w
    QUAY_USERNAME: '..6'
    QUAY_PASSWORD: ey...8 
    WORKER_IMAGE:  brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8
    WORKER_TAG: v3.9.0 
    BUILDER_VM_CONTAINER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-qemu-rhcos-rhel8:v3.9.0
    SETUP_TIME: 180
    MINIMUM_RETRY_THRESHOLD: 0
    SSH_AUTHORIZED_KEYS:
    - ssh-rsa A.....hine
USERFILES_LOCATION: default
USERFILES_PATH: userfiles
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: 
  - default
DISTRIBUTED_STORAGE_PREFERENCE:
  - default
DISTRIBUTED_STORAGE_CONFIG:
  default:
      - SwiftStorage
      - auth_url: https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13000/v3/
        auth_version: "3"
        os_options:
            region_name: regionOne
            tenant_id: 54.....0
            user_domain_name: redhat.com
        storage_path: /datastorage/registry
        swift_container: quay
        swift_password: e.....e
        swift_user: p...s
SUPER_USERS:
  - whuquay
FEATURE_ANONYMOUS_ACCESS: true 
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_PROXY_CACHE: true
BROWSER_API_CALLS_XHR_ONLY: false
CREATE_PRIVATE_REPO_ON_PUSH: true
CREATE_NAMESPACE_ON_PUSH: true
FEATURE_USERNAME_CONFIRMATION: false
FEATURE_USER_INITIALIZE: true 
AUTHENTICATION_TYPE: Database
FEATURE_UI_V2: true
FEATURE_LISTEN_IP_VERSION: IPv4
FEATURE_SUPERUSERS_FULL_ACCESS: true
QUOTA_BACKFILL: true
FEATURE_GARBAGE_COLLECTION: true
GARBAGE_COLLECTION_FREQUENCY: 10
DEFAULT_TAG_EXPIRATION: 2w
TAG_EXPIRATION_OPTIONS:
- 2w
- 10s
FEATURE_SECURITY_SCANNER_NOTIFY_ON_NEW_INDEX: True
FEATURE_GENERAL_OCI_SUPPORT: true

2. configure a github build trigger

3. trigger the build process

Actual results:

The build worker container can't start in builder VM container.

$ journalctl -f -u quay-builder
-- Logs begin at Thu 2023-08-03 08:56:46 UTC. --
Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=info msg="pinging buildmanager..."
Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=info msg="starting log stream to buildmanager"
Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=info msg="registering job for registration token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImdKaVBTYXNZMzAtNGpxbnBkS2tXSzhYWnZoUHdqOUQ5aXRSTU51SWdGZG8ifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktYnVpbGRlci1xdWF5LWVudGVycHJpc2UuYXBwcy53aHU0MTNvc3AwMi5xZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb206NDQzIiwibmJmIjoxNjkxMDUyOTkwLCJpYXQiOjE2OTEwNTI5OTAsImV4cCI6MTY5MTA1MzYyMCwic3ViIjoiKGFub255bW91cykiLCJhY2Nlc3MiOnt9LCJjb250ZXh0Ijp7InRva2VuX3R5cGUiOiJidWlsZF9qb2JfcmVnaXN0cmF0aW9uIiwiYnVpbGRfaWQiOiJmMGU5YTY2Yy1iMTcyLTRkN2YtODEwYi0xODk5YTZmM2Y3NWEiLCJqb2JfaWQiOiJidWlsZG1hbi9wcm9kdWN0aW9uL2J1aWxkaW5nL2YwZTlhNjZjLWIxNzItNGQ3Zi04MTBiLTE4OTlhNmYzZjc1YSIsImV4cGlyYXRpb24iOjYzMH19.CtfoAnbzqQDWkheMGauZ3_zVpnngHTz4vL6rTiRCzQpAfTsqlGUyQwbbhp3OI7smgYiWeGEGHdOyerkkFdTlBmejiCRhXVNF-A_UmwH8Ly72UyB7XgCb1Odn2Zi9PilSPlHrCKY-LYWEpKb6RVogNhJ5Ws3tBMcicPfV8hJJjd1tCPEb91jSbqX6BN-9MYvMZlomSHFUHDkumucX4-g2uQXX5c6SzP8kF99YuJKgD3n1kSPn66cr5fyluR3fu63UeeLuDLpQnliPi3zU1yiC4ZOPTTFLkrIhImUSAMbtUklfqzRZO1FVbVWf4mydplPgwPxs-OdNFFIm-UeW50MyKg"
Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=info msg="starting heartbeat to buildmanager"
Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=info msg="starting build"
Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=info msg="connecting to docker host: unix:/var/run/podman/podman.sock"
Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a podman[1727]: time="2023-08-03T08:57:26Z" level=fatal msg="invalid endpoint"
Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a systemd[1]: quay-builder.service: Main process exited, code=exited, status=1/FAILURE
Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a systemd[1]: quay-builder.service: Failed with result 'exit-code'.
Aug 03 08:57:26 f0e9a66c-b172-4d7f-810b-1899a6f3f75a systemd[1]: Failed to start quay-builder.service.

Expected results:

The build process should finish successfully.

Additional info:

Don't change any configuration, just use brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.9.0-13 to replace the latest version v3.9.0-18, everything work well.

The latest builder worker image v3.9.0 is v3.9.0-18

$ skopeo inspect --raw --tls-verify=false --creds='....8 docker://brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.9.0  |jq .
{
  "manifests": [
    {
      "digest": "sha256:677137792803636377413ec8d058d9b8ece2b75c91bb2a30fc435e78a9ee335e",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 760
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}

$ skopeo inspect --raw --tls-verify=false --creds='....8 docker://brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.9.0-18  |jq .
{
  "manifests": [
    {
      "digest": "sha256:677137792803636377413ec8d058d9b8ece2b75c91bb2a30fc435e78a9ee335e",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 760
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}