$ openshift-install create install-config --dir work ? SSH Public Key /home/fedora/.ssh/openshift-qe.pub ? Platform gcp INFO Credentials loaded from file "/home/fedora/.gcp/osServiceAccount.json" ? Project ID OpenShift QE (openshift-qe) ? Region us-central1 ? Base Domain qe.gcp.devcluster.openshift.com ? Cluster Name jiwei-0808-02 ? Pull Secret [? for help] ****** INFO Install-Config created in: work $ $ vim work/install-config.yaml $ $ openshift-install create manifests --dir work INFO Credentials loaded from file "/home/fedora/.gcp/osServiceAccount.json" INFO Consuming Install Config from target directory INFO Manifests created in: work/manifests and work/openshift $ $ grep image work/openshift/*.yaml work/openshift/99_openshift-cluster-api_master-machines-0.yaml: image: projects/rhcos-cloud/global/images/rhcos-410-84-202207061638-0-gcp-x86-64 work/openshift/99_openshift-cluster-api_master-machines-1.yaml: image: projects/rhcos-cloud/global/images/rhcos-410-84-202207061638-0-gcp-x86-64 work/openshift/99_openshift-cluster-api_master-machines-2.yaml: image: projects/rhcos-cloud/global/images/rhcos-410-84-202207061638-0-gcp-x86-64 work/openshift/99_openshift-cluster-api_worker-machineset-0.yaml: image: projects/rhcos-cloud/global/images/rhcos-410-84-202207061638-0-gcp-x86-64 work/openshift/99_openshift-cluster-api_worker-machineset-1.yaml: image: projects/rhcos-cloud/global/images/rhcos-410-84-202207061638-0-gcp-x86-64 work/openshift/99_openshift-cluster-api_worker-machineset-2.yaml: image: projects/rhcos-cloud/global/images/rhcos-410-84-202207061638-0-gcp-x86-64 $ $ sed -i 's/image: projects\/rhcos-cloud\/global\/images\/rhcos-410-84-202207061638-0-gcp-x86-64/image: projects\/redhat-marketplace-dev\/global\/images\/redhat-coreos-48-x86-64-202206140145/' work/openshift/*.yaml $ $ grep image work/openshift/*.yaml work/openshift/99_openshift-cluster-api_master-machines-0.yaml: image: projects/redhat-marketplace-dev/global/images/redhat-coreos-48-x86-64-202206140145 work/openshift/99_openshift-cluster-api_master-machines-1.yaml: image: projects/redhat-marketplace-dev/global/images/redhat-coreos-48-x86-64-202206140145 work/openshift/99_openshift-cluster-api_master-machines-2.yaml: image: projects/redhat-marketplace-dev/global/images/redhat-coreos-48-x86-64-202206140145 work/openshift/99_openshift-cluster-api_worker-machineset-0.yaml: image: projects/redhat-marketplace-dev/global/images/redhat-coreos-48-x86-64-202206140145 work/openshift/99_openshift-cluster-api_worker-machineset-1.yaml: image: projects/redhat-marketplace-dev/global/images/redhat-coreos-48-x86-64-202206140145 work/openshift/99_openshift-cluster-api_worker-machineset-2.yaml: image: projects/redhat-marketplace-dev/global/images/redhat-coreos-48-x86-64-202206140145 $ $ openshift-install create cluster --dir work INFO Consuming Common Manifests from target directory INFO Consuming Master Machines from target directory INFO Consuming Openshift Manifests from target directory INFO Consuming OpenShift Install (Manifests) from target directory INFO Consuming Worker Machines from target directory INFO Credentials loaded from file "/home/fedora/.gcp/osServiceAccount.json" INFO Creating infrastructure resources... INFO Waiting up to 20m0s (until 4:12AM) for the Kubernetes API at https://api.jiwei-0808-02.qe.gcp.devcluster.openshift.com:6443... INFO API v1.23.5+012e945 up INFO Waiting up to 30m0s (until 4:24AM) for bootstrapping to complete... INFO Destroying the bootstrap resources... INFO Waiting up to 40m0s (until 4:47AM) for the cluster at https://api.jiwei-0808-02.qe.gcp.devcluster.openshift.com:6443 to initialize. .. INFO Waiting up to 10m0s (until 4:23AM) for the openshift-console route to be created... INFO Install complete! INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/fedora/work/auth/kubeconfig' INFO Access the OpenShift web-console here: https://console-openshift-console.apps.jiwei-0808-02.qe.gcp.devcluster.openshift.com INFO Login to the console with user: "kubeadmin", and password: "AF2tp-MfXXh-a5AK6-mvyHJ" INFO Time elapsed: 23m5s $ $ gcloud compute disks list --filter='name~jiwei' --format='table(name,type,sizeGb,sourceImageId,sourceImage)' NAME TYPE SIZE_GB SOURCE_IMAGE_ID SOURCE_IMAGE jiwei-0808-02-ctkzl-master-0 pd-ssd 128 7421349014111453722 https://www.googleapis.com/compute/v1/projects/redhat-marketplace-dev/global/images/redhat-coreos-48-x86-64-202206140145 jiwei-0808-02-ctkzl-worker-b-pjbts pd-ssd 128 7421349014111453722 https://www.googleapis.com/compute/v1/projects/redhat-marketplace-dev/global/images/redhat-coreos-48-x86-64-202206140145 jiwei-0808-02-ctkzl-master-1 pd-ssd 128 7421349014111453722 https://www.googleapis.com/compute/v1/projects/redhat-marketplace-dev/global/images/redhat-coreos-48-x86-64-202206140145 jiwei-0808-02-ctkzl-worker-c-lkqc4 pd-ssd 128 7421349014111453722 https://www.googleapis.com/compute/v1/projects/redhat-marketplace-dev/global/images/redhat-coreos-48-x86-64-202206140145 jiwei-0808-02-ctkzl-master-2 pd-ssd 128 7421349014111453722 https://www.googleapis.com/compute/v1/projects/redhat-marketplace-dev/global/images/redhat-coreos-48-x86-64-202206140145 $ $ export KUBECONFIG=/home/fedora/work/auth/kubeconfig $ automation2022/v4-installer-autotest/bin/check_cluster_health.sh 4.10.26-x86_64 quay.io/openshift-release-dev/ocp-release Step #0: kubeconfig is avaiable Step #1: check cluster and payload availability Step #2: Make sure all machines are applied with latest machineconfig Checking #0 Command: oc get machineconfig NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 00-worker dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 01-master-container-runtime dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 01-master-kubelet dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 01-worker-container-runtime dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 01-worker-kubelet dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 99-master-generated-crio-seccomp-use-default 3.2.0 22m 99-master-generated-registries dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 99-master-ssh 3.2.0 24m 99-worker-generated-crio-seccomp-use-default 3.2.0 22m 99-worker-generated-registries dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 99-worker-ssh 3.2.0 24m rendered-master-359c32b16daff959a56c6fc0e99107c6 dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m rendered-worker-a0662aa655e26ec9466af6b842722d92 dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m Checking master machines are applied with latest master machineconfig... latest master machineconfig: rendered-master-359c32b16daff959a56c6fc0e99107c6 latest machineconfig - rendered-master-359c32b16daff959a56c6fc0e99107c6 is already applied to jiwei-0808-02-ctkzl-master-0.c.openshift-qe.internal jiwei-0808-02-ctkzl-master-1.c.openshift-qe.internal jiwei-0808-02-ctkzl-master-2.c.openshift-qe.internal masters are already applied with latest machineconfig Checking #0 Command: oc get machineconfig NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 00-worker dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 01-master-container-runtime dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 01-master-kubelet dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 01-worker-container-runtime dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 01-worker-kubelet dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 99-master-generated-crio-seccomp-use-default 3.2.0 22m 99-master-generated-registries dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 99-master-ssh 3.2.0 24m 99-worker-generated-crio-seccomp-use-default 3.2.0 22m 99-worker-generated-registries dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m 99-worker-ssh 3.2.0 24m rendered-master-359c32b16daff959a56c6fc0e99107c6 dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m rendered-worker-a0662aa655e26ec9466af6b842722d92 dc29945da95a65f460ad50ad1bbc10e1918a9c61 3.2.0 22m Checking worker machines are applied with latest worker machineconfig... latest worker machineconfig: rendered-worker-a0662aa655e26ec9466af6b842722d92 latest machineconfig - rendered-worker-a0662aa655e26ec9466af6b842722d92 is already applied to jiwei-0808-02-ctkzl-worker-b-pjbts.c.openshift-qe.internal jiwei-0808-02-ctkzl-worker-c-lkqc4.c.openshift-qe.internal workers are already applied with latest machineconfig Step #3: check all cluster operators get stable and ready Checking #0 Make sure every operator do not reporty empty column Make sure every operator column reports version Make sure every operator's AVAILABLE column is True Passed #0 Checking #1 Make sure every operator do not reporty empty column Make sure every operator column reports version Make sure every operator's AVAILABLE column is True Passed #1 Checking #2 Make sure every operator do not reporty empty column Make sure every operator column reports version Make sure every operator's AVAILABLE column is True Passed #2 All cluster operators get stable and ready Step #4: Make sure every machine is in 'Ready' status All nodes are ready Step #5: check all pods are in status running or complate Step #6: Get cluster version, make sure it is matched with your selected target payload version 4.10.26 True False 9m44s Cluster version is 4.10.26 Step #7: Make sure all operator report correct payload version All operators report correct payload version Step #8: Make sure your cluster's web console is accessible Skip ... ... Step #9: CVO Health Check Step #9.1: Make sure CVO is using the image from your selected target payload CVO image is matched with your selected payload image Step #9.2 Make sure CVO imagePullPolicy is 'IfNotPresent' CVO imagePullPolicy is set to 'IfNotPresent' Step #10: MCO Health Check all pods in openshift-machine-config-operator is running Step #10.1: Check pods in openshift-machine-config-operator project Step #10.2: Check if the machineconfigs and machineconfigpools are cluster scoped (no namespace qualification) Step #10.3: Make sure MCO is using the image from your selected target payload MCO pod is using: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a67ad93f1075f1ed9c89860b41f702ef73d3153d1f4615a47b1b45660c94c189 Step #10.4: Make sure MCO imagePullPolicy is 'IfNotPresent' MCO imagePullPolicy is IfNotPresent 10.6 Make sure machines in cluster is using machine-os-content from your selected target payload OS version: 410.84.202208030316-0 Machine is using the following os content, refer to machine-config-daemon-7kfq2 pod log in openshift-machine-config-operator namespace * pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23d0609643c25efcd30a7a64483fdee2343ced26b1fd08c0cbf8d03a5d405939 CustomOrigin: Managed by machine-config-operator Version: 410.84.202208030316-0 (2022-08-03T03:19:21Z) ostree://8fd19b78118cf1ba87bbb38adad80a5f8e74630c969d8c7d908e412ce4fb0e47 Version: 48.84.202206140145-0 (2022-06-14T01:48:29Z) machine-os-content is from quay.io quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23d0609643c25efcd30a7a64483fdee2343ced26b1fd08c0cbf8d03a5d405939 There are the machineconfig rendered-master-359c32b16daff959a56c6fc0e99107c6 rendered-worker-a0662aa655e26ec9466af6b842722d92 master and worker machineconfig is detected, and number is 2 Step #11.1: Make sure cloud credential operator pod is using the image from your selected target payload cloud credential operator pod is using: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92c9b2f2ae162e18c8ea850a549a6ffcc655d141eece53181ccbcf47d71bce2e Step #11.3: Make sure cloud credential operator imagePullPolicy is 'IfNotPresent' cloud credential operator imagePullPolicy is IfNotPresent cloud credential operator is from quay.io $ $ openshift-install destroy cluster --dir work INFO Credentials loaded from file "/home/fedora/.gcp/osServiceAccount.json" INFO Stopped instance jiwei-0808-02-ctkzl-worker-b-pjbts INFO Stopped instance jiwei-0808-02-ctkzl-worker-c-lkqc4 INFO Stopped instance jiwei-0808-02-ctkzl-master-1 INFO Stopped instance jiwei-0808-02-ctkzl-master-2 INFO Stopped instance jiwei-0808-02-ctkzl-master-0 INFO Deleted IAM project role bindings INFO Deleted service account projects/openshift-qe/serviceAccounts/jiwei-0808-02-ctkzl-m@openshift-qe.iam.gserviceaccount.com INFO Deleted service account projects/openshift-qe/serviceAccounts/jiwei-0808-0-openshift-m-p82pt@openshift-qe.iam.gserviceaccount.com INFO Deleted service account projects/openshift-qe/serviceAccounts/jiwei-0808-0-openshift-c-7m489@openshift-qe.iam.gserviceaccount.com INFO Deleted service account projects/openshift-qe/serviceAccounts/jiwei-0808-02-ctkzl-w@openshift-qe.iam.gserviceaccount.com INFO Deleted service account projects/openshift-qe/serviceAccounts/jiwei-0808-0-openshift-i-ctpv8@openshift-qe.iam.gserviceaccount.com INFO Deleted service account projects/openshift-qe/serviceAccounts/jiwei-0808-0-openshift-g-wgtdg@openshift-qe.iam.gserviceaccount.com INFO Deleted service account projects/openshift-qe/serviceAccounts/jiwei-0808-0-openshift-i-q8n8d@openshift-qe.iam.gserviceaccount.com INFO Deleted service account projects/openshift-qe/serviceAccounts/jiwei-0808-0-cloud-crede-vf528@openshift-qe.iam.gserviceaccount.com INFO Deleted service account projects/openshift-qe/serviceAccounts/jiwei-0808-0-openshift-g-2rsl7@openshift-qe.iam.gserviceaccount.com INFO Deleted 2 recordset(s) in zone qe INFO Deleted 3 recordset(s) in zone jiwei-0808-02-ctkzl-private-zone INFO Deleted DNS zone jiwei-0808-02-ctkzl-private-zone INFO Deleted bucket jiwei-0808-02-ctkzl-image-registry-us-east1-nixeqnbhsmkbyjvvtr INFO Deleted instance jiwei-0808-02-ctkzl-master-0 INFO Deleted instance jiwei-0808-02-ctkzl-worker-b-pjbts INFO Deleted instance jiwei-0808-02-ctkzl-master-1 INFO Deleted instance jiwei-0808-02-ctkzl-worker-c-lkqc4 INFO Deleted instance jiwei-0808-02-ctkzl-master-2 INFO Deleted disk jiwei-0808-02-ctkzl-master-0 INFO Deleted disk jiwei-0808-02-ctkzl-worker-b-pjbts INFO Deleted disk jiwei-0808-02-ctkzl-master-1 INFO Deleted disk jiwei-0808-02-ctkzl-worker-c-lkqc4 INFO Deleted disk jiwei-0808-02-ctkzl-master-2 INFO Deleted firewall rule jiwei-0808-02-ctkzl-api INFO Deleted firewall rule jiwei-0808-02-ctkzl-control-plane INFO Deleted firewall rule jiwei-0808-02-ctkzl-etcd INFO Deleted firewall rule jiwei-0808-02-ctkzl-health-checks INFO Deleted firewall rule jiwei-0808-02-ctkzl-internal-cluster INFO Deleted firewall rule jiwei-0808-02-ctkzl-internal-network INFO Deleted firewall rule k8s-fw-a6781b6b888e943bc93572dda9c91d08 INFO Deleted firewall rule k8s-a6781b6b888e943bc93572dda9c91d08-http-hc INFO Deleted address jiwei-0808-02-ctkzl-cluster-ip INFO Deleted address jiwei-0808-02-ctkzl-cluster-public-ip INFO Deleted forwarding rule jiwei-0808-02-ctkzl-api INFO Deleted forwarding rule jiwei-0808-02-ctkzl-api-internal INFO Deleted forwarding rule a6781b6b888e943bc93572dda9c91d08 INFO Deleted router jiwei-0808-02-ctkzl-router INFO Deleted subnetwork jiwei-0808-02-ctkzl-worker-subnet INFO Deleted target pool a6781b6b888e943bc93572dda9c91d08 INFO Deleted target pool jiwei-0808-02-ctkzl-api INFO Deleted backend service jiwei-0808-02-ctkzl-api-internal INFO Deleted subnetwork jiwei-0808-02-ctkzl-master-subnet INFO Deleted instance group jiwei-0808-02-ctkzl-master-us-east1-b INFO Deleted instance group jiwei-0808-02-ctkzl-master-us-east1-d INFO Deleted instance group jiwei-0808-02-ctkzl-master-us-east1-c INFO Deleted health check jiwei-0808-02-ctkzl-api-internal INFO Deleted HTTP health check a6781b6b888e943bc93572dda9c91d08 INFO Deleted HTTP health check jiwei-0808-02-ctkzl-api INFO Deleted network jiwei-0808-02-ctkzl-network INFO Time elapsed: 4m14s $