Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-15642

Routes not working despite API being added to library-go, openshift-apiserver

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • 4.14.0
    • openshift-apiserver
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • No
    • None
    • Proposed
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      While working on epic for set/delete http headers https://issues.redhat.com/browse/NE-982 I had to create API fields in the following:

      https://github.com/openshift/api/pull/1364/commits/2aaab98e2bd0d8c29542ae2f3c611d19ccd5debd#diff-4dfca44f3f68bff8b6381ff3fac676c6be96da9ec1d15224bbd70f839744eb40R155

      and openshift-apiserver in the following:

      https://github.com/openshift/openshift-apiserver/pull/380/commits/a2804e2439c886253c8ae9854055bf0fd618de09#diff-5f71380ea2089d69677ac8e2d171cf909b21bd6463f0f2b67883abfc839fe00dR70

       

      I also added API changes to library-go.

      https://github.com/openshift/library-go/pull/1449/commits/8cc933fcf157a541bcf178e04b5c0bfd57b4d35c#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R138

       

      and also added api changes and library-go changes to openshift-apiserver

       https://github.com/openshift/openshift-apiserver/pull/380/commits/0c097f817fa9d2c1bd6214aabe7c13772f769e68#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R216-R218

       

      However the e2e tests for openshift-apiserver are failing with these errors:

       

      2023-07-01T05:47:45.324842630Z 2023-07-01T05:47:45.324Z    ERROR    operator.init.controller-runtime.source.EventHandler    wait/loop.go:62    if kind is a CRD, it should be installed before calling Start    {"kind": "Route.route.openshift.io", "error": "no matches for kind \"Route\" in version \"route.openshift.io/v1\""}

      logs are here

      https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_openshift-apiserver/380/pull-ci-openshift-openshift-apiserver-master-e2e-aws-ovn/1675012390023860224/artifacts/e2e-aws-ovn/gather-must-gather/artifacts/must-gather.tar

      You can download them from here:

       gsutil -m cp -r gs://origin-ci-test/pr-logs/pull/openshift_openshift-apiserver/380/pull-ci-openshift-openshift-apiserver-master-e2e-aws-ovn/1675012390023860224/ .

      However in the current.log I don't see these errors.

      vim artifacts/must-gather/registry-build01-ci-openshift-org-ci-op-8yf7dct6-stable-sha256-6423f2816489eee14ad4c0e5f42a22b5e26802b188ced0719f0aaa9bc2d063c9/namespaces/openshift-ingress-operator/pods/ingress-operator-6895b8d458-pb9ll/ingress-operator/ingress-operator/logs/current.log

       

      If you check errors in the e2e logs https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_openshift-apiserver/380/pull-ci-openshift-openshift-apiserver-master-e2e-aws-ovn/1675012390023860224

       

      level=error msg=Cluster operator console Degraded is True with ConsoleDefaultRouteSync_FailedDefaultRouteApply::DownloadsDefaultRouteSync_FailedDefaultRouteApply::RouteHealth_FailedRouteGet::SyncLoopRefresh_FailedGet: ConsoleDefaultRouteSyncDegraded: Route.route.openshift.io "console" is invalid: spec.to.name: Required value205

      level=error msg=DownloadsDefaultRouteSyncDegraded: Route.route.openshift.io "downloads" is invalid: spec.to.name: Required value206

      level=error msg=RouteHealthDegraded: routes.route.openshift.io "console" not found207level=error msg=SyncLoopRefreshDegraded: routes.route.openshift.io "console" not found208

       

      the install is failing with these errors :

      Copying kubeconfig to shared dir as kubeconfig-minimal184
      level=info msg=Waiting up to 40m0s (until 6:30AM UTC) for the cluster at https://api.ci-op-8yf7dct6-089b8.origin-ci-int-aws.dev.rhcloud.com:6443 to initialize...185
      level=error msg=Cluster operator authentication Degraded is True with CustomRouteController_SyncError::IngressStateEndpoints_MissingSubsets::OAuthClientsController_SyncError::OAuthConfigRoute_NotFound::OAuthServerDeployment_PreconditionNotFulfilled::OAuthServerServiceEndpointAccessibleController_SyncError::OAuthServerServiceEndpointsEndpointAccessibleController_SyncError::WellKnownReadyController_SyncError: CustomRouteControllerDegraded: Route.route.openshift.io "oauth-openshift" is invalid: spec.to.name: Required value186
      level=error msg=IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server187level=error msg=OAuthClientsControllerDegraded: route.route.openshift.io "oauth-openshift" not found188
      level=error msg=OAuthConfigRouteDegraded: The OAuth server route 'openshift-authentication/oauth-openshift' was not found189
      level=error msg=OAuthServerDeploymentDegraded: waiting for the oauth-openshift route to appear: route.route.openshift.io "oauth-openshift" not found190level=error msg=OAuthServerDeploymentDegraded: 191
      level=error msg=OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.150.171:443/healthz": dial tcp 172.30.150.171:443: connect: connection refused192level=error msg=OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: oauth service endpoints are not ready193
      level=error msg=WellKnownReadyControllerDegraded: route.route.openshift.io "oauth-openshift" not found194
      level=error msg=Cluster operator authentication Available is False with OAuthServerDeployment_PreconditionNotFulfilled::OAuthServerRouteEndpointAccessibleController_ResourceNotFound::OAuthServerServiceEndpointAccessibleController_EndpointUnavailable::OAuthServerServiceEndpointsEndpointAccessibleController_ResourceNotFound::WellKnown_PrereqsNotReady: OAuthServerRouteEndpointAccessibleControllerAvailable: failed to retrieve route from cache: route.route.openshift.io "oauth-openshift" not found195
      level=error msg=OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.150.171:443/healthz": dial tcp 172.30.150.171:443: connect: connection refused196
      level=error msg=OAuthServerServiceEndpointsEndpointAccessibleControllerAvailable: endpoints "oauth-openshift" not found197level=error msg=WellKnownAvailable: route.route.openshift.io "oauth-openshift" not found198
      level=info msg=Cluster operator baremetal Disabled is False with : 199
      level=info msg=Cluster operator cloud-controller-manager TrustedCABundleControllerControllerAvailable is True with AsExpected: Trusted CA Bundle Controller works as expected200
      level=info msg=Cluster operator cloud-controller-manager TrustedCABundleControllerControllerDegraded is False with AsExpected: Trusted CA Bundle Controller works as expected201
      level=info msg=Cluster operator cloud-controller-manager CloudConfigControllerAvailable is True with AsExpected: Cloud Config Controller works as expected202
      level=info msg=Cluster operator cloud-controller-manager CloudConfigControllerDegraded is False with AsExpected: Cloud Config Controller works as expected203
      level=info msg=Cluster operator cloud-controller-manager CloudControllerOwner is True with AsExpected: Cluster Cloud Controller Manager Operator owns cloud controllers at 4.14.0-0.ci.test-2023-07-01-052930-ci-op-8yf7dct6-latest204
      level=error msg=Cluster operator console Degraded is True with ConsoleDefaultRouteSync_FailedDefaultRouteApply::DownloadsDefaultRouteSync_FailedDefaultRouteApply::RouteHealth_FailedRouteGet::SyncLoopRefresh_FailedGet: ConsoleDefaultRouteSyncDegraded: Route.route.openshift.io "console" is invalid: spec.to.name: Required value205
      level=error msg=DownloadsDefaultRouteSyncDegraded: Route.route.openshift.io "downloads" is invalid: spec.to.name: Required value206
      level=error msg=RouteHealthDegraded: routes.route.openshift.io "console" not found207
      level=error msg=SyncLoopRefreshDegraded: routes.route.openshift.io "console" not found208
      level=error msg=Cluster operator console Available is False with RouteHealth_RouteNotAdmitted: RouteHealthAvailable: console route is not admitted209
      level=info msg=Cluster operator etcd RecentBackup is Unknown with ControllerStarted: The etcd backup controller is starting, and will decide if recent backups are available or if a backup is required210
      level=info msg=Cluster operator ingress EvaluationConditionsDetected is False with AsExpected: 211
      level=info msg=Cluster operator insights ClusterTransferAvailable is False with BadRequest: failed to pull cluster transfer: OCM API https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27e317bbe7-9071-40a4-ba8d-1b5b25886192%27+and+status+is+%27accepted%27 returned HTTP 400: {"code":"ACCT-MGMT-21","href":"/api/accounts_mgmt/v1/errors/21","id":"21","kind":"Error","operation_id":"3eddf5e4-94fd-4ffd-ab96-6122c232a4db","reason":"pq: got 130445 parameters but PostgreSQL only supports 65535 parameters"}212
      level=info msg=Cluster operator insights Disabled is False with AsExpected: 213
      level=info msg=Cluster operator insights SCAAvailable is False with Forbidden: Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/certificates: OCM API https://api.openshift.com/api/accounts_mgmt/v1/certificates returned HTTP 403: {"code":"ACCT-MGMT-11","href":"/api/accounts_mgmt/v1/errors/11","id":"11","kind":"Error","operation_id":"2a2927f4-c63b-485b-81c9-54dca6213cde","reason":"Account with ID 2DUeKzzTD9ngfsQ6YgkzdJn1jA4 denied access to perform create on Certificate with HTTP call POST /api/accounts_mgmt/v1/certificates"}214
      level=error msg=Cluster operator monitoring Available is False with UpdatingOpenshiftStateMetricsFailed: reconciling openshift-state-metrics Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/openshift-state-metrics: context deadline exceeded215level=error msg=Cluster operator monitoring Degraded is True with UpdatingOpenshiftStateMetricsFailed: reconciling openshift-state-metrics Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/openshift-state-metrics: context deadline exceeded216level=info msg=Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack.217
      level=info msg=Cluster operator network ManagementStateDegraded is False with : 218
      level=error msg=Cluster initialization failed because one or more operators are not functioning properly.219level=error msg=The cluster should be accessible for troubleshooting as detailed in the documentation linked below,220
      level=error msg=https://docs.openshift.com/container-platform/latest/support/troubleshooting/troubleshooting-installations.html221level=error msg=The 'wait-for install-complete' subcommand can then be used to continue the installation222
      level=error msg=failed to initialize the cluster: Cluster operators authentication, console, monitoring are not available223Installer exit with code 6224Tear down the backgroup process of copying kube config225The process - 130 is not existing any more226Setup phase finished, prepare env for next steps227Copying log bundle...228Removing REDACTED info from log...229tar: Removing leading `/' from member names230/tmp/installer/terraform.txt231Unsupported cluster type 'aws' to collect machine IDs232Copying required artifacts to shared dir233{"component":"entrypoint","error":"wrapped process failed: exit status 6","file":"k8s.io/test-infra/prow/entrypoint/run.go:79","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2023-07-01T06:30:30Z"}234error: failed to execute wrapped command: exit status 6 235INFO[2023-07-01T06:30:31Z] Step e2e-aws-ovn-ipi-install-install failed after 57m37s. 236INFO[2023-07-01T06:30:31Z] Step phase pre failed after 59m41s.          237INFO[2023-07-01T06:30:31Z] Running multi-stage phase post               238INFO[2023-07-01T06:30:31Z] Running step e2e-aws-ovn-gather-core-dump.   239INFO[2023-07-01T06:30:51Z] Step e2e-aws-ovn-gather-core-dump succeeded after 19s. 240INFO[2023-07-01T06:30:51Z] Running step e2e-aws-ovn-gather-aws-console. 241INFO[2023-07-01T06:31:09Z] Step e2e-aws-ovn-gather-aws-console succeeded after 18s. unfold_more10 skipped lines...252INFO[2023-07-01T06:40:31Z] Ran for 1h15m54s                             253ERRO[2023-07-01T06:40:31Z] Some steps failed:                           254ERRO[2023-07-01T06:40:31Z] 255  * could not run steps: step e2e-aws-ovn failed: "e2e-aws-ovn" pre steps failed: "e2e-aws-ovn" pod "e2e-aws-ovn-ipi-install-install" failed: could not watch pod: the pod ci-op-8yf7dct6/e2e-aws-ovn-ipi-install-install failed after 57m37s (failed containers: test): ContainerFailed one or more containers exited256257Container test exited with code 1, reason Error258---259alse with UpdatingOpenshiftStateMetricsFailed: reconciling openshift-state-metrics Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/openshift-state-metrics: context deadline exceeded260
      level=error msg=Cluster operator monitoring Degraded is True with UpdatingOpenshiftStateMetricsFailed: reconciling openshift-state-metrics Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/openshift-state-metrics: context deadline exceeded261level=info msg=Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack.262level=info msg=Cluster operator network ManagementStateDegraded is False with : 263level=error msg=Cluster initialization failed because one or more operators are not functioning properly.264
      level=error msg=The cluster should be accessible for troubleshooting as detailed in the documentation linked below,265level=error msg=https://docs.openshift.com/container-platform/latest/support/troubleshooting/troubleshooting-installations.html266
      level=error msg=The 'wait-for install-complete' subcommand can then be used to continue the installation267
      level=error msg=failed to initialize the cluster: Cluster operators authentication, console, monitoring are not available268Installer exit with code 6269Tear down the backgroup process of copying kube config270The process - 130 is not existing any more271Setup phase finished, prepare env for next steps272Copying log bundle...273Removing REDACTED info from log...274tar: Removing leading `/' from member names275/tmp/installer/terraform.txt276Unsupported cluster type 'aws' to collect machine IDs277Copying required artifacts to shared dir278{"component":"entrypoint","error":"wrapped process failed: exit status 6","file":"k8s.io/test-infra/prow/entrypoint/run.go:79","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2023-07-01T06:30:30Z"}279error: failed to execute wrapped command: exit status 6280---281Link to step on registry info site: https://steps.ci.openshift.org/reference/ipi-install-install282Link to job on registry info site: https://steps.ci.openshift.org/job?org=openshift&repo=openshift-apiserver&branch=master&test=e2e-aws-ovn 283INFO[2023-07-01T06:40:31Z] Reporting job state 'failed' with reason 'executing_graph:step_failed:utilizing_lease:executing_test:executing_multi_stage_test'  

      Is it because the route api has not been merged yet into the master ?

      However openshift-apiserver is having the route api changes then why is this issue happening ?

      Am I missing something ?

              Unassigned Unassigned
              rhn-support-misalunk Miheer Salunke
              None
              None
              Rahul Gangwar Rahul Gangwar
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: