$ gcloud config get account jiwei@redhat.com $ gcloud config get project openshift-qe $ gcloud --project openshift-qe-shared-vpc services list --enabled NAME TITLE bigquery.googleapis.com BigQuery API bigquerystorage.googleapis.com BigQuery Storage API clouddebugger.googleapis.com Cloud Debugger API cloudresourcemanager.googleapis.com Cloud Resource Manager API cloudtrace.googleapis.com Cloud Trace API compute.googleapis.com Compute Engine API datastore.googleapis.com Cloud Datastore API deploymentmanager.googleapis.com Cloud Deployment Manager V2 API dns.googleapis.com Cloud DNS API iam.googleapis.com Identity and Access Management (IAM) API iamcredentials.googleapis.com IAM Service Account Credentials API logging.googleapis.com Cloud Logging API monitoring.googleapis.com Cloud Monitoring API networksecurity.googleapis.com Network Security API oslogin.googleapis.com Cloud OS Login API sql-component.googleapis.com Cloud SQL $ $ gcloud config get account jiwei-0523-sa@openshift-qe-shared-vpc.iam.gserviceaccount.com $ gcloud config get project openshift-qe-shared-vpc $ openshift-install version openshift-install 4.12.0-0.nightly-2022-10-23-204408 built from commit b1a8fbf247d48bf24afc48886f2f6dafc3eaf2bb release image registry.ci.openshift.org/ocp/release@sha256:5ca6ec23eba15f20e3c41050f2e35f92a9ae8d64c1e4866163defef1bf49352f release architecture amd64 $ $ openshift-install create install-config --dir work ? SSH Public Key /home/fedora/.ssh/openshift-qe.pub ? Platform gcp INFO Credentials loaded from file "/home/fedora/.gcp/osServiceAccount.json" ? Project ID OpenShift QE Shared VPC (openshift-qe-shared-vpc) ? Region us-central1 ? Base Domain qe-shared-vpc.qe.gcp.devcluster.openshift.com ? Cluster Name jiwei-1024c ? Pull Secret [? for help] ****** INFO Install-Config created in: work $ $ openshift-install create manifests --dir work INFO Credentials loaded from file "/home/fedora/.gcp/osServiceAccount.json" INFO Consuming Install Config from target directory INFO Manifests created in: work/manifests and work/openshift $ $ openshift-install create cluster --dir work INFO Consuming OpenShift Install (Manifests) from target directory INFO Consuming Worker Machines from target directory INFO Consuming Common Manifests from target directory INFO Consuming Openshift Manifests from target directory INFO Consuming Master Machines from target directory INFO Credentials loaded from file "/home/fedora/.gcp/osServiceAccount.json" WARNING Permission denied. Unable to fetch enabled services for project. WARNING Missing permissions to fetch Quotas and therefore will skip checking them: failed to load quota limits: googleapi: Error 403: Service Usage API has not been used in project 673766552981 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=673766552981 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry. WARNING Details: WARNING [ WARNING { WARNING "@type": "type.googleapis.com/google.rpc.Help", WARNING "links": [ WARNING { WARNING "description": "Google developers console API activation", WARNING "url": "https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=673766552981" WARNING } WARNING ] WARNING }, WARNING { WARNING "@type": "type.googleapis.com/google.rpc.ErrorInfo", WARNING "domain": "googleapis.com", WARNING "metadata": { WARNING "consumer": "projects/673766552981", WARNING "service": "serviceusage.googleapis.com" WARNING }, WARNING "reason": "SERVICE_DISABLED" WARNING } WARNING ] WARNING , accessNotConfigured, make sure you have `roles/servicemanagement.quotaViewer` assigned to the user. INFO Creating infrastructure resources... INFO Waiting up to 20m0s (until 3:38AM) for the Kubernetes API at https://api.jiwei-1024c.qe-shared-vpc.qe.gcp.devcluster.openshift.com:6443... INFO API v1.25.2+4bd0702 up INFO Waiting up to 30m0s (until 3:54AM) for bootstrapping to complete... INFO Destroying the bootstrap resources... INFO Waiting up to 40m0s (until 4:19AM) for the cluster at https://api.jiwei-1024c.qe-shared-vpc.qe.gcp.devcluster.openshift.com:6443 to initialize... ERROR Cluster operator authentication Degraded is True with IngressStateEndpoints_MissingSubsets::OAuthClientsController_SyncError::OAuthServerDeployment_PreconditionNotFulfilled::OAuthServerRouteEndpointAccessibleController_SyncError::OAuthServerServiceEndpointAccessibleController_SyncError::OAuthServerServiceEndpointsEndpointAccessibleController_SyncError::WellKnownReadyController_SyncError: IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server ERROR OAuthClientsControllerDegraded: no ingress for host oauth-openshift.apps.jiwei-1024c.qe-shared-vpc.qe.gcp.devcluster.openshift.com in route oauth-openshift in namespace openshift-authentication ERROR OAuthServerDeploymentDegraded: waiting for the oauth-openshift route to contain an admitted ingress: no admitted ingress for route oauth-openshift in namespace openshift-authentication ERROR OAuthServerDeploymentDegraded: ERROR OAuthServerRouteEndpointAccessibleControllerDegraded: route "openshift-authentication/oauth-openshift": status does not have a valid host address ERROR OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.176.14:443/healthz": dial tcp 172.30.176.14:443: connect: connection refused ERROR OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: oauth service endpoints are not ready ERROR WellKnownReadyControllerDegraded: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap "oauth-openshift" not found (check authentication operator, it is supposed to create this) ERROR Cluster operator authentication Available is False with OAuthServerDeployment_PreconditionNotFulfilled::OAuthServerRouteEndpointAccessibleController_ResourceNotFound::OAuthServerServiceEndpointAccessibleController_EndpointUnavailable::OAuthServerServiceEndpointsEndpointAccessibleController_ResourceNotFound::ReadyIngressNodes_NoReadyIngressNodes::WellKnown_NotReady: OAuthServerRouteEndpointAccessibleControllerAvailable: failed to retrieve route from cache: route.route.openshift.io "oauth-openshift" not found ERROR OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.176.14:443/healthz": dial tcp 172.30.176.14:443: connect: connection refused ERROR OAuthServerServiceEndpointsEndpointAccessibleControllerAvailable: endpoints "oauth-openshift" not found ERROR ReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes, 0 custom target nodes (none are schedulable or ready for ingress pods). ERROR WellKnownAvailable: The well-known endpoint is not yet available: failed to get oauth metadata from openshift-config-managed/oauth-openshift ConfigMap: configmap "oauth-openshift" not found (check authentication operator, it is supposed to create this) INFO Cluster operator baremetal Disabled is True with UnsupportedPlatform: Nothing to do on this Platform INFO Cluster operator cloud-controller-manager TrustedCABundleControllerControllerAvailable is True with AsExpected: Trusted CA Bundle Controller works as expected INFO Cluster operator cloud-controller-manager TrustedCABundleControllerControllerDegraded is False with AsExpected: Trusted CA Bundle Controller works as expected INFO Cluster operator cloud-controller-manager CloudConfigControllerAvailable is True with AsExpected: Cloud Config Controller works as expected INFO Cluster operator cloud-controller-manager CloudConfigControllerDegraded is False with AsExpected: Cloud Config Controller works as expected ERROR Cluster operator cloud-credential Degraded is True with CredentialsFailing: 7 of 7 credentials requests are failing to sync. INFO Cluster operator cloud-credential Progressing is True with Reconciling: 0 of 7 credentials requests provisioned, 7 reporting errors. ERROR Cluster operator cluster-autoscaler Degraded is True with MissingDependency: machine-api not ready ERROR Cluster operator console Degraded is True with DefaultRouteSync_FailedAdmitDefaultRoute::RouteHealth_RouteNotAdmitted::SyncLoopRefresh_FailedIngress: DefaultRouteSyncDegraded: no ingress for host console-openshift-console.apps.jiwei-1024c.qe-shared-vpc.qe.gcp.devcluster.openshift.com in route console in namespace openshift-console ERROR RouteHealthDegraded: console route is not admitted ERROR SyncLoopRefreshDegraded: no ingress for host console-openshift-console.apps.jiwei-1024c.qe-shared-vpc.qe.gcp.devcluster.openshift.com in route console in namespace openshift-console ERROR Cluster operator console Available is False with RouteHealth_RouteNotAdmitted: RouteHealthAvailable: console route is not admitted INFO Cluster operator etcd RecentBackup is Unknown with ControllerStarted: The etcd backup controller is starting, and will decide if recent backups are available or if a backup is required ERROR Cluster operator image-registry Available is False with DeploymentNotFound: Available: The deployment does not exist ERROR NodeCADaemonAvailable: The daemon set node-ca has available replicas ERROR ImagePrunerAvailable: Pruner CronJob has been created INFO Cluster operator image-registry Progressing is True with Error: Progressing: Unable to apply resources: unable to sync storage configuration: unable to get cluster minted credentials "openshift-image-registry/installer-cloud-credentials": secret "installer-cloud-credentials" not found INFO NodeCADaemonProgressing: The daemon set node-ca is deployed ERROR Cluster operator image-registry Degraded is True with Unavailable: Degraded: The deployment does not exist ERROR Cluster operator ingress Available is False with IngressUnavailable: The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DNSReady=False (NoZones: The record isn't present in any zones.) INFO Cluster operator ingress Progressing is True with Reconciling: ingresscontroller "default" is progressing: IngressControllerProgressing: One or more status conditions indicate progressing: DeploymentRollingOut=True (DeploymentRollingOut: Waiting for router deployment rollout to finish: 0 of 2 updated replica(s) are available... INFO ). INFO Not all ingress controllers are available. ERROR Cluster operator ingress Degraded is True with IngressDegraded: The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: PodsScheduled=False (PodsNotScheduled: Some pods are not scheduled: Pod "router-default-6dd9569c5d-dp9nt" cannot be scheduled: 0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. Pod "router-default-6dd9569c5d-dnv2z" cannot be scheduled: 0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.), DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1), DNSReady=False (NoZones: The record isn't present in any zones.), CanaryChecksSucceeding=Unknown (CanaryRouteNotAdmitted: Canary route is not admitted by the default ingress controller) INFO Cluster operator ingress EvaluationConditionsDetected is False with AsExpected: INFO Cluster operator insights ClusterTransferAvailable is False with NoClusterTransfer: no available cluster transfer INFO Cluster operator insights Disabled is False with AsExpected: INFO Cluster operator insights SCAAvailable is True with Updated: SCA certs successfully updated in the etc-pki-entitlement secret ERROR Cluster operator kube-controller-manager Degraded is True with GarbageCollector_Error: GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": dial tcp: lookup thanos-querier.openshift-monitoring.svc on 172.30.0.10:53: no such host INFO Cluster operator machine-api Progressing is True with SyncingResources: Progressing towards operator: 4.12.0-0.nightly-2022-10-23-204408 ERROR Cluster operator machine-api Degraded is True with SyncingFailed: Failed when progressing towards operator: 4.12.0-0.nightly-2022-10-23-204408 because minimum worker replica count (2) not yet met: current running replicas 0, waiting for [jiwei-1024c-zmsz6-worker-a-pzk94 jiwei-1024c-zmsz6-worker-b-zvmcn] ERROR Cluster operator machine-api Available is False with Initializing: Operator is initializing ERROR Cluster operator monitoring Available is False with UpdatingPrometheusOperatorFailed: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: got 2 unavailable replicas ERROR Cluster operator monitoring Degraded is True with UpdatingPrometheusOperatorFailed: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: got 2 unavailable replicas INFO Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack. INFO Cluster operator network ManagementStateDegraded is False with : INFO Cluster operator network Progressing is True with Deploying: Deployment "/openshift-network-diagnostics/network-check-source" is waiting for other operators to become ready INFO Deployment "/openshift-cloud-network-config-controller/cloud-network-config-controller" is waiting for other operators to become ready INFO Cluster operator storage Progressing is True with GCPPDCSIDriverOperatorCR_GCPPDDriverControllerServiceController_Deploying: GCPPDCSIDriverOperatorCRProgressing: GCPPDDriverControllerServiceControllerProgressing: Waiting for Deployment to deploy pods ERROR Cluster operator storage Available is False with GCPPDCSIDriverOperatorCR_GCPPDDriverControllerServiceController_Deploying: GCPPDCSIDriverOperatorCRAvailable: GCPPDDriverControllerServiceControllerAvailable: Waiting for Deployment ERROR Cluster initialization failed because one or more operators are not functioning properly. ERROR The cluster should be accessible for troubleshooting as detailed in the documentation linked below, ERROR https://docs.openshift.com/container-platform/latest/support/troubleshooting/troubleshooting-installations.html ERROR The 'wait-for install-complete' subcommand can then be used to continue the installation ERROR failed to initialize the cluster: Cluster operators authentication, console, image-registry, ingress, machine-api, monitoring, storage are not available $ $ gcloud compute instances list --filter='name~jiwei-1024c' NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS jiwei-1024c-zmsz6-master-0 us-central1-a n2-standard-4 10.0.0.5 RUNNING jiwei-1024c-zmsz6-master-1 us-central1-b n2-standard-4 10.0.0.3 RUNNING jiwei-1024c-zmsz6-master-2 us-central1-c n2-standard-4 10.0.0.4 RUNNING $