Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-4740

quay-quay-builder-qemu-rhcos-rhel8:v3.8.0 pod can not start on OCP 4.10 cluster

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • quay-v3.8.0
    • quay-builder
    • False
    • None
    • False

      Description of problem:

      After deploying quay3.8.0 on OCP 4.10 cluster,  try to trigger build on OCP 4.10 bare metal cluster, build failed.
      The quay-builder-qemu-rhcos-rhel8:v3.8.0 pod can not start correctly on OCP 4.10.

      Version-Release number of selected component (if applicable):

      quay :v3.8.0
      WORKER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-qemu-rhcos-rhel8:v3.8.0
      BUILDER_VM_CONTAINER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.8.0

      How reproducible:

      always

      Steps to Reproduce:
      1. set up builder environment on openstack bare metal machine + ocp4.10
      2. deploy quay on any supported ocp cluster, such as ocp 4.10
      3. configure builder as

      FEATURE_BUILD_SUPPORT: true
FEATURE_GITLAB_BUILD: true
GITLAB_TRIGGER_CONFIG:
    CLIENT_ID: b......2 
    CLIENT_SECRET: bb.....9 
GITLAB_TRIGGER_KIND:
    GITLAB_ENDPOINT: https://gitlab.com/
FEATURE_BITBUCKET_BUILD: true
BITBUCKET_TRIGGER_CONFIG:
    CONSUMER_KEY: U.....d 
    CONSUMER_SECRET: f.....Z 
FEATURE_GITHUB_BUILD: true
GITHUB_TRIGGER_CONFIG:
  API_ENDPOINT: https://api.github.com/
  CLIENT_ID: 2....d
  CLIENT_SECRET:  7....8
  GITHUB_ENDPOINT: https://github.com/
BUILDMAN_HOSTNAME: quayregistry-quay-builder-quay-enterprise.apps.whu410osp8.1114-npl.qe.rhcloud.com:443
BUILD_MANAGER:
- ephemeral
- ALLOWED_WORKER_COUNT: 30 
  ORCHESTRATOR_PREFIX: buildman/production/
  JOB_REGISTRATION_TIMEOUT: 600
  ORCHESTRATOR:
    REDIS_HOST: quayregistry-quay-redis
    REDIS_PASSWORD: "" 
    REDIS_SSL: false
    REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
  EXECUTORS:
  - EXECUTOR: kubernetes
    DEBUG: true
    K8S_API_SERVER: api.whu410bm02.qe.devcluster.openshift.com:6443 
    K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build_cluster.crt
    VOLUME_SIZE: 8G
    KUBERNETES_DISTRIBUTION: openshift
    CONTAINER_MEMORY_LIMITS: 5120Mi
    CONTAINER_CPU_LIMITS: 1000m
    CONTAINER_MEMORY_REQUEST: 3968Mi
    CONTAINER_CPU_REQUEST: 500m
    NODE_SELECTOR_LABEL_KEY: type 
    NODE_SELECTOR_LABEL_VALUE: builder 
    CONTAINER_RUNTIME: podman
    BUILDER_NAMESPACE: builder 
    SERVICE_ACCOUNT_NAME:  quay-builder  
    SERVICE_ACCOUNT_TOKEN: ey......0zXNbKDbafA
    QUAY_USERNAME: '|....6'
    QUAY_PASSWORD: e.....c8 
    WORKER_IMAGE:  brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8
    WORKER_TAG: v3.8.0 
    BUILDER_VM_CONTAINER_IMAGE: brew.registry.redhat.io/rh-osbs/quay-quay-builder-qemu-rhcos-rhel8:v3.8.0 
    SETUP_TIME: 180
    MINIMUM_RETRY_THRESHOLD: 0
    SSH_AUTHORIZED_KEYS:
    - ssh-rsa A.....ne
USERFILES_LOCATION: default
USERFILES_PATH: userfiles
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: 
  - default
DISTRIBUTED_STORAGE_PREFERENCE:
  - default
DISTRIBUTED_STORAGE_CONFIG:
  default:
      - SwiftStorage
      - auth_url: https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13000/v3/
        auth_version: "3"
        os_options:
            region_name: regionOne
            tenant_id: 5.....0
            user_domain_name: redhat.com
        storage_path: /datastorage/registry
        swift_container: quay
        swift_password: e...e
        swift_user: p....s
SUPER_USERS:
  - whuquay
  - whutest
FEATURE_ANONYMOUS_ACCESS: true 
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_PROXY_CACHE: true
BROWSER_API_CALLS_XHR_ONLY: false
CREATE_PRIVATE_REPO_ON_PUSH: true
CREATE_NAMESPACE_ON_PUSH: true
FEATURE_USERNAME_CONFIRMATION: false
FEATURE_USER_INITIALIZE: true 
AUTHENTICATION_TYPE: Database
FEATURE_UI_V2: True
FEATURE_LISTEN_IP_VERSION: IPv4
FEATURE_SUPERUSERS_FULL_ACCESS: True
FEATURE_RESTRICTED_USERS: true
RESTRICTED_USERS_WHITELIST:
  - user1
  - user2

      3. trigger build

      Actual results:

      The build process failed. the quay-builder-qemu-rhcos-rhel8 pod can not start correctly. Got quay-builder-qemu-rhcos-rhel8 pod log

      .....
++ qemu-img info coreos_production_qemu_image.qcow2
++ grep 'virtual size'
++ cut -d ' ' -f 5
++ tr -d '()'
+ MIN_IMAGE_SIZE_BYTES=17179869184
++ numfmt --from=iec 8G
+ VM_VOLUME_SIZE_BYTES=8589934592
+ '[' 8589934592 -gt 17179869184 ']'
+ /usr/libexec/qemu-kvm -enable-kvm -cpu host -nographic -drive if=virtio,file=/userdata/coreos_production_qemu_image.qcow2 -fw_cfg name=opt/com.coreos/config,file=/userdata/user_data -m 4G -machine accel=kvm -net nic,model=virtio -net user,hostfwd=tcp::2222-:22 -smp 2
KVM: entry failed, hardware error 0x8
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00080660
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=04 66 41 eb f1 66 83 c9 ff 66 89 c8 66 5b 66 5e 66 5f 66 c3 <ea> 5b e0 00 f0 30 36 2f 32 33 2f 39 39 00 fc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      Expected results:

      The quay-builder-qemu-rhcos-rhel8 pod should start correctly. the build process should complete successfully.

      Additional info:

      quay-quay-builder-qemu-rhcos-rhel8:v3.7.6 can start correctly on  OCP 4.10 bare metal cluster.

      quay-quay-builder-qemu-rhcos-rhel8:v3.8.0 can start correctly on  OCP 4.11 bare metal cluster.

      $ oc get clusterversions.config.openshift.io 
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-11-11-083401   True        False         163m    Cluster version is 4.11.0-0.nightly-2022-11-11

$ oc get pod -n builder
NAME                                               READY   STATUS    RESTARTS   AGE
263cfef1-8d37-4382-a20c-556d2f6c370d-6kbkr-2bzqb   1/1     Running   0          21m

+ set -o nounset
+ echo '{"ignition": {"version": "3.0.0"}, "passwd": {"users": [{"sshAuthorizedKeys": ["ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC
0DXCqWbugQT5CIGKflvFT3RFocubrdVeZoH9iCMZOuYvUC1LfOxmmdO5OKqFKoKFb3ShnHJgBo9VzCYDst6nLUeZsIpDk0bEZ0O3rUOD59zIgXpkFyLQr3Su9sCWOOj
to0RRtCjcvgwRsjDOVUEZ8GSCQ0aZMLvaLEELDj
......
+ VM_VOLUME_SIZE_BYTES=8589934592
+ '[' 8589934592 -gt 17179869184 ']'
+ /usr/libexec/qemu-kvm -enable-kvm -cpu host -nographic -drive if=virtio,file=/userdata/coreos_production_qemu_image.qcow2 -fw
_cfg name=opt/com.coreos/config,file=/userdata/user_data -m 4G -machine accel=kvm -net nic,model=virtio -net user,hostfwd=tcp::
2222-:22 -smp 2
^[c^[[?7l^[[2J^[[0mSeaBIOS (version 1.16.0-3.module+el8.7.0+16689+53d59bc2)^M
iPXE (http://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+BFF8DFC0+BFECDFC0 CA00^M
Press Ctrl-B to configure iPXE (PCI 00:03.0)...^M                                                                              
 ^M
Booting from Hard Disk...^M
..^M
error: ../../grub-core/disk/i386/pc/biosdisk.c:543:fai^[[10;54H^[[10;55Hlure reading sector^M
0x1670 from `fd0'.^M
........
Red Hat Enterprise Linux CoreOS 47.84.202211030947-0 (Ootpa) 4.7^M
SSH host key: SHA256:j3ir81vOidOEpgBxZD3VV6t3W7xrOoP9QEZhf1TdbCg (ED25519)^M
SSH host key: SHA256:EbbDDIfjkh+0+3AZSEeBqrt7gBJdX+gIpQQRbITl53g (ECDSA)^M
SSH host key: SHA256:wxAbjc7Nl+hxWcVCzMBn29wBobUJOMBcAYvu+EKsLbM (RSA)^M
ens3: 10.0.2.15 fec0::a93f:22ed:31f:40ad^M
263cfef1-8d37-4382-a20c-556d2f6c370d login:  

      It seems the hardware version will affect quay-quay-builder-qemu-rhcos-rhel8 working. 

      Below information are workable OCP 4.11 Openstack bare metal node

      # cat /proc/cpuinfo 
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 85
model name    : Intel Xeon Processor (Skylake, IBRS)
stepping    : 4
microcode    : 0x1
cpu MHz        : 2693.670
cache size    : 16384 KB
physical id    : 0
siblings    : 1
core id        : 0
cpu cores    : 1
apicid        : 0
initial apicid    : 0
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscalls
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa mmio_stale_data retbleed
bogomips    : 5387.34
clflush size    : 64
cache_alignment    : 64
address sizes    : 46 bits physical, 48 bits virtual
power management:processor    : 1
vendor_id    : GenuineIntel
cpu family    : 6
model        : 85
model name    : Intel Xeon Processor (Skylake, IBRS)
stepping    : 4
microcode    : 0x1
cpu MHz        : 2693.670
cache size    : 16384 KB
physical id    : 1
siblings    : 1
core id        : 0
cpu cores    : 1
apicid        : 1
initial apicid    : 1
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscalls
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa mmio_stale_data retbleed
bogomips    : 5387.34
clflush size    : 64
cache_alignment    : 64
address sizes    : 46 bits physical, 48 bits virtual
power management:processor    : 2
vendor_id    : GenuineIntel
cpu family    : 6
model        : 85
model name    : Intel Xeon Processor (Skylake, IBRS)
stepping    : 4
microcode    : 0x1
cpu MHz        : 2693.670
cache size    : 16384 KB
physical id    : 2
siblings    : 1
core id        : 0
cpu cores    : 1
apicid        : 2
initial apicid    : 2
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscalls
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa mmio_stale_data retbleed
bogomips    : 5387.34
clflush size    : 64
cache_alignment    : 64
address sizes    : 46 bits physical, 48 bits virtual
power management:processor    : 3
vendor_id    : GenuineIntel
cpu family    : 6
model        : 85
model name    : Intel Xeon Processor (Skylake, IBRS)
stepping    : 4
microcode    : 0x1
cpu MHz        : 2693.670
cache size    : 16384 KB
physical id    : 3
siblings    : 1
core id        : 0
cpu cores    : 1
apicid        : 3
initial apicid    : 3
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscalls
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa mmio_stale_data retbleed
bogomips    : 5387.34
clflush size    : 64
cache_alignment    : 64
address sizes    : 46 bits physical, 48 bits virtual
power management:

      sh-4.4# fdisk -l   
Disk /dev/vda: 45 GiB, 48318382080 bytes, 94371840 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 9180F729-5FED-40DB-9297-142C48D32DC2Device       Start      End  Sectors  Size Type
/dev/vda1     2048     4095     2048    1M BIOS boot
/dev/vda2     4096   264191   260096  127M EFI System
/dev/vda3   264192  1050623   786432  384M Linux filesystem
/dev/vda4  1050624 94371806 93321183 44.5G Linux filesystem

              Unassigned Unassigned
              rhwhu Weihua Hu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: