-
Bug
-
Resolution: Done
-
Critical
-
None
-
4.14.0
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
No
-
None
-
Proposed
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
While working on epic for set/delete http headers https://issues.redhat.com/browse/NE-982 I had to create API fields in the following:
and openshift-apiserver in the following:
I also added API changes to library-go.
and also added api changes and library-go changes to openshift-apiserver
However the e2e tests for openshift-apiserver are failing with these errors:
2023-07-01T05:47:45.324842630Z 2023-07-01T05:47:45.324Z ERROR operator.init.controller-runtime.source.EventHandler wait/loop.go:62 if kind is a CRD, it should be installed before calling Start {"kind": "Route.route.openshift.io", "error": "no matches for kind \"Route\" in version \"route.openshift.io/v1\""}
logs are here
You can download them from here:
gsutil -m cp -r gs://origin-ci-test/pr-logs/pull/openshift_openshift-apiserver/380/pull-ci-openshift-openshift-apiserver-master-e2e-aws-ovn/1675012390023860224/ .
However in the current.log I don't see these errors.
vim artifacts/must-gather/registry-build01-ci-openshift-org-ci-op-8yf7dct6-stable-sha256-6423f2816489eee14ad4c0e5f42a22b5e26802b188ced0719f0aaa9bc2d063c9/namespaces/openshift-ingress-operator/pods/ingress-operator-6895b8d458-pb9ll/ingress-operator/ingress-operator/logs/current.log
If you check errors in the e2e logs https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_openshift-apiserver/380/pull-ci-openshift-openshift-apiserver-master-e2e-aws-ovn/1675012390023860224
level=error msg=Cluster operator console Degraded is True with ConsoleDefaultRouteSync_FailedDefaultRouteApply::DownloadsDefaultRouteSync_FailedDefaultRouteApply::RouteHealth_FailedRouteGet::SyncLoopRefresh_FailedGet: ConsoleDefaultRouteSyncDegraded: Route.route.openshift.io "console" is invalid: spec.to.name: Required value205
level=error msg=DownloadsDefaultRouteSyncDegraded: Route.route.openshift.io "downloads" is invalid: spec.to.name: Required value206
level=error msg=RouteHealthDegraded: routes.route.openshift.io "console" not found207level=error msg=SyncLoopRefreshDegraded: routes.route.openshift.io "console" not found208
the install is failing with these errors :
Copying kubeconfig to shared dir as kubeconfig-minimal184 level=info msg=Waiting up to 40m0s (until 6:30AM UTC) for the cluster at https://api.ci-op-8yf7dct6-089b8.origin-ci-int-aws.dev.rhcloud.com:6443 to initialize...185 level=error msg=Cluster operator authentication Degraded is True with CustomRouteController_SyncError::IngressStateEndpoints_MissingSubsets::OAuthClientsController_SyncError::OAuthConfigRoute_NotFound::OAuthServerDeployment_PreconditionNotFulfilled::OAuthServerServiceEndpointAccessibleController_SyncError::OAuthServerServiceEndpointsEndpointAccessibleController_SyncError::WellKnownReadyController_SyncError: CustomRouteControllerDegraded: Route.route.openshift.io "oauth-openshift" is invalid: spec.to.name: Required value186 level=error msg=IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server187level=error msg=OAuthClientsControllerDegraded: route.route.openshift.io "oauth-openshift" not found188 level=error msg=OAuthConfigRouteDegraded: The OAuth server route 'openshift-authentication/oauth-openshift' was not found189 level=error msg=OAuthServerDeploymentDegraded: waiting for the oauth-openshift route to appear: route.route.openshift.io "oauth-openshift" not found190level=error msg=OAuthServerDeploymentDegraded: 191 level=error msg=OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.150.171:443/healthz": dial tcp 172.30.150.171:443: connect: connection refused192level=error msg=OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: oauth service endpoints are not ready193 level=error msg=WellKnownReadyControllerDegraded: route.route.openshift.io "oauth-openshift" not found194 level=error msg=Cluster operator authentication Available is False with OAuthServerDeployment_PreconditionNotFulfilled::OAuthServerRouteEndpointAccessibleController_ResourceNotFound::OAuthServerServiceEndpointAccessibleController_EndpointUnavailable::OAuthServerServiceEndpointsEndpointAccessibleController_ResourceNotFound::WellKnown_PrereqsNotReady: OAuthServerRouteEndpointAccessibleControllerAvailable: failed to retrieve route from cache: route.route.openshift.io "oauth-openshift" not found195 level=error msg=OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.150.171:443/healthz": dial tcp 172.30.150.171:443: connect: connection refused196 level=error msg=OAuthServerServiceEndpointsEndpointAccessibleControllerAvailable: endpoints "oauth-openshift" not found197level=error msg=WellKnownAvailable: route.route.openshift.io "oauth-openshift" not found198 level=info msg=Cluster operator baremetal Disabled is False with : 199 level=info msg=Cluster operator cloud-controller-manager TrustedCABundleControllerControllerAvailable is True with AsExpected: Trusted CA Bundle Controller works as expected200 level=info msg=Cluster operator cloud-controller-manager TrustedCABundleControllerControllerDegraded is False with AsExpected: Trusted CA Bundle Controller works as expected201 level=info msg=Cluster operator cloud-controller-manager CloudConfigControllerAvailable is True with AsExpected: Cloud Config Controller works as expected202 level=info msg=Cluster operator cloud-controller-manager CloudConfigControllerDegraded is False with AsExpected: Cloud Config Controller works as expected203 level=info msg=Cluster operator cloud-controller-manager CloudControllerOwner is True with AsExpected: Cluster Cloud Controller Manager Operator owns cloud controllers at 4.14.0-0.ci.test-2023-07-01-052930-ci-op-8yf7dct6-latest204 level=error msg=Cluster operator console Degraded is True with ConsoleDefaultRouteSync_FailedDefaultRouteApply::DownloadsDefaultRouteSync_FailedDefaultRouteApply::RouteHealth_FailedRouteGet::SyncLoopRefresh_FailedGet: ConsoleDefaultRouteSyncDegraded: Route.route.openshift.io "console" is invalid: spec.to.name: Required value205 level=error msg=DownloadsDefaultRouteSyncDegraded: Route.route.openshift.io "downloads" is invalid: spec.to.name: Required value206 level=error msg=RouteHealthDegraded: routes.route.openshift.io "console" not found207 level=error msg=SyncLoopRefreshDegraded: routes.route.openshift.io "console" not found208 level=error msg=Cluster operator console Available is False with RouteHealth_RouteNotAdmitted: RouteHealthAvailable: console route is not admitted209 level=info msg=Cluster operator etcd RecentBackup is Unknown with ControllerStarted: The etcd backup controller is starting, and will decide if recent backups are available or if a backup is required210 level=info msg=Cluster operator ingress EvaluationConditionsDetected is False with AsExpected: 211 level=info msg=Cluster operator insights ClusterTransferAvailable is False with BadRequest: failed to pull cluster transfer: OCM API https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27e317bbe7-9071-40a4-ba8d-1b5b25886192%27+and+status+is+%27accepted%27 returned HTTP 400: {"code":"ACCT-MGMT-21","href":"/api/accounts_mgmt/v1/errors/21","id":"21","kind":"Error","operation_id":"3eddf5e4-94fd-4ffd-ab96-6122c232a4db","reason":"pq: got 130445 parameters but PostgreSQL only supports 65535 parameters"}212 level=info msg=Cluster operator insights Disabled is False with AsExpected: 213 level=info msg=Cluster operator insights SCAAvailable is False with Forbidden: Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/certificates: OCM API https://api.openshift.com/api/accounts_mgmt/v1/certificates returned HTTP 403: {"code":"ACCT-MGMT-11","href":"/api/accounts_mgmt/v1/errors/11","id":"11","kind":"Error","operation_id":"2a2927f4-c63b-485b-81c9-54dca6213cde","reason":"Account with ID 2DUeKzzTD9ngfsQ6YgkzdJn1jA4 denied access to perform create on Certificate with HTTP call POST /api/accounts_mgmt/v1/certificates"}214 level=error msg=Cluster operator monitoring Available is False with UpdatingOpenshiftStateMetricsFailed: reconciling openshift-state-metrics Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/openshift-state-metrics: context deadline exceeded215level=error msg=Cluster operator monitoring Degraded is True with UpdatingOpenshiftStateMetricsFailed: reconciling openshift-state-metrics Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/openshift-state-metrics: context deadline exceeded216level=info msg=Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack.217 level=info msg=Cluster operator network ManagementStateDegraded is False with : 218 level=error msg=Cluster initialization failed because one or more operators are not functioning properly.219level=error msg=The cluster should be accessible for troubleshooting as detailed in the documentation linked below,220 level=error msg=https://docs.openshift.com/container-platform/latest/support/troubleshooting/troubleshooting-installations.html221level=error msg=The 'wait-for install-complete' subcommand can then be used to continue the installation222 level=error msg=failed to initialize the cluster: Cluster operators authentication, console, monitoring are not available223Installer exit with code 6224Tear down the backgroup process of copying kube config225The process - 130 is not existing any more226Setup phase finished, prepare env for next steps227Copying log bundle...228Removing REDACTED info from log...229tar: Removing leading `/' from member names230/tmp/installer/terraform.txt231Unsupported cluster type 'aws' to collect machine IDs232Copying required artifacts to shared dir233{"component":"entrypoint","error":"wrapped process failed: exit status 6","file":"k8s.io/test-infra/prow/entrypoint/run.go:79","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2023-07-01T06:30:30Z"}234error: failed to execute wrapped command: exit status 6 235INFO[2023-07-01T06:30:31Z] Step e2e-aws-ovn-ipi-install-install failed after 57m37s. 236INFO[2023-07-01T06:30:31Z] Step phase pre failed after 59m41s. 237INFO[2023-07-01T06:30:31Z] Running multi-stage phase post 238INFO[2023-07-01T06:30:31Z] Running step e2e-aws-ovn-gather-core-dump. 239INFO[2023-07-01T06:30:51Z] Step e2e-aws-ovn-gather-core-dump succeeded after 19s. 240INFO[2023-07-01T06:30:51Z] Running step e2e-aws-ovn-gather-aws-console. 241INFO[2023-07-01T06:31:09Z] Step e2e-aws-ovn-gather-aws-console succeeded after 18s. unfold_more10 skipped lines...252INFO[2023-07-01T06:40:31Z] Ran for 1h15m54s 253ERRO[2023-07-01T06:40:31Z] Some steps failed: 254ERRO[2023-07-01T06:40:31Z] 255 * could not run steps: step e2e-aws-ovn failed: "e2e-aws-ovn" pre steps failed: "e2e-aws-ovn" pod "e2e-aws-ovn-ipi-install-install" failed: could not watch pod: the pod ci-op-8yf7dct6/e2e-aws-ovn-ipi-install-install failed after 57m37s (failed containers: test): ContainerFailed one or more containers exited256257Container test exited with code 1, reason Error258---259alse with UpdatingOpenshiftStateMetricsFailed: reconciling openshift-state-metrics Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/openshift-state-metrics: context deadline exceeded260 level=error msg=Cluster operator monitoring Degraded is True with UpdatingOpenshiftStateMetricsFailed: reconciling openshift-state-metrics Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/openshift-state-metrics: context deadline exceeded261level=info msg=Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack.262level=info msg=Cluster operator network ManagementStateDegraded is False with : 263level=error msg=Cluster initialization failed because one or more operators are not functioning properly.264 level=error msg=The cluster should be accessible for troubleshooting as detailed in the documentation linked below,265level=error msg=https://docs.openshift.com/container-platform/latest/support/troubleshooting/troubleshooting-installations.html266 level=error msg=The 'wait-for install-complete' subcommand can then be used to continue the installation267 level=error msg=failed to initialize the cluster: Cluster operators authentication, console, monitoring are not available268Installer exit with code 6269Tear down the backgroup process of copying kube config270The process - 130 is not existing any more271Setup phase finished, prepare env for next steps272Copying log bundle...273Removing REDACTED info from log...274tar: Removing leading `/' from member names275/tmp/installer/terraform.txt276Unsupported cluster type 'aws' to collect machine IDs277Copying required artifacts to shared dir278{"component":"entrypoint","error":"wrapped process failed: exit status 6","file":"k8s.io/test-infra/prow/entrypoint/run.go:79","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2023-07-01T06:30:30Z"}279error: failed to execute wrapped command: exit status 6280---281Link to step on registry info site: https://steps.ci.openshift.org/reference/ipi-install-install282Link to job on registry info site: https://steps.ci.openshift.org/job?org=openshift&repo=openshift-apiserver&branch=master&test=e2e-aws-ovn 283INFO[2023-07-01T06:40:31Z] Reporting job state 'failed' with reason 'executing_graph:step_failed:utilizing_lease:executing_test:executing_multi_stage_test'
Is it because the route api has not been merged yet into the master ?
However openshift-apiserver is having the route api changes then why is this issue happening ?
Am I missing something ?