OC_HOST=$(oc whoami --show-server) OCP_PASSWORD=$(cat "${KUBEADMIN_PASSWORD_FILE}") export OPENSHIFT_HOST=${OC_HOST} export OPENSHIFT_PASSWORD=${OCP_PASSWORD} export MULTIAZ="false" export DESTRUCTIVE="false" export NUMBER_OF_TENANTS="2" export TENANTS_CREATION_TIMEOUT="3" export OUTPUT_DIR=${ARTIFACT_DIR} # running RHOAM testsuite ./setup_external.sh Login successful. You have access to 119 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". time="2023-09-18T19:11:39Z" level=info msg="calling reset on all prometheus gauge vectors" custom_metrics=StartGaugeVector === RUN TestAPIs I0918 19:11:40.474246 71 request.go:690] Waited for 1.026184996s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/managed.openshift.io/v1alpha2?timeout=32s I0918 19:11:50.503367 71 request.go:690] Waited for 3.676081425s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/apps.openshift.io/v1?timeout=32s I0918 19:12:00.550938 71 request.go:690] Waited for 2.326246033s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/capabilities.3scale.net/v1beta1 Running Suite: Functional Test Suite - / ======================================== Random Seed: 1695064299 Will run 35 of 61 specs ------------------------------ [BeforeSuite] /go/src/github.com/integr8ly/integreatly-operator/test/functional/suite_test.go:79 STEP: bootstrapping test environment @ 09/18/23 19:12:02 [BeforeSuite] PASSED [0.010 seconds] ------------------------------ integreatly AWS Specific Tests A12 - Verify CRO BlobStorage CRs Successful /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" [FAILED] in [JustBeforeEach] - /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/suite.go:863 @ 09/18/23 19:22:20.838 • [FAILED] [618.827 seconds] integreatly [JustBeforeEach] AWS Specific Tests A12 - Verify CRO BlobStorage CRs Successful [JustBeforeEach] /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:28 [It] /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 [FAILED] error waiting for RHMI CR status.stage to be "complete" In [JustBeforeEach] at: /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/suite.go:863 @ 09/18/23 19:22:20.838 ------------------------------ SS ------------------------------ integreatly managed-api Functional A25 - Verify standalone RHMI VPC exists and is configured properly /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:22:21.891062 71 request.go:690] Waited for 1.015796965s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/migration.k8s.io/v1alpha1 RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" [FAILED] in [JustBeforeEach] - /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/suite.go:863 @ 09/18/23 19:32:39.696 • [FAILED] [618.858 seconds] integreatly [JustBeforeEach] managed-api Functional A25 - Verify standalone RHMI VPC exists and is configured properly [JustBeforeEach] /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:28 [It] /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 [FAILED] error waiting for RHMI CR status.stage to be "complete" In [JustBeforeEach] at: /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/suite.go:863 @ 09/18/23 19:32:39.696 ------------------------------ SS ------------------------------ integreatly managed-api Functional F04 - Verify AWS s3 blob storage resources exist /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:32:40.748001 71 request.go:690] Waited for 1.014494097s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/k8s.cni.cncf.io/v1 RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" RHMI CR status.stage is: "installation". Waiting for: "complete" I0918 19:35:19.586227 71 request.go:690] Waited for 1.026131998s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/image.openshift.io/v1 time="2023-09-18T19:35:22Z" level=info msg="STS mode" [FAILED] in [It] - /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/node.go:463 @ 09/18/23 19:35:22.405 • [FAILED] [162.709 seconds] integreatly managed-api Functional [It] F04 - Verify AWS s3 blob storage resources exist /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 [FAILED] failed to create aws session: failed to get STS credentials: ROLE_ARN key should not be empty In [It] at: /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/node.go:463 @ 09/18/23 19:35:22.405 ------------------------------ integreatly managed-api HAPPY PATH A01 - Verify that all stages in the integreatly-operator CR report completed /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:35:29.611175 71 request.go:690] Waited for 2.36531832s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/infrastructure.cluster.x-k8s.io/v1beta1 • [8.607 seconds] ------------------------------ integreatly managed-api HAPPY PATH A03 - Verify all namespaces have been created with the correct name /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 • [8.747 seconds] ------------------------------ integreatly managed-api HAPPY PATH A05 - Verify product operator version /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:35:40.812281 71 request.go:690] Waited for 1.026819567s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/cloudcredential.openshift.io/v1 • [8.632 seconds] ------------------------------ integreatly managed-api HAPPY PATH A07 - Verify product versions /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:35:50.844053 71 request.go:690] Waited for 2.425966915s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/image.openshift.io/v1 • [8.609 seconds] ------------------------------ integreatly managed-api HAPPY PATH A08 - Verify all products routes are created /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:36:02.884434 71 request.go:690] Waited for 1.024173198s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/tuned.openshift.io/v1 • [8.801 seconds] ------------------------------ integreatly managed-api HAPPY PATH A09 - Verify Subscription Install Plan Strategy /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:36:12.911387 71 request.go:690] Waited for 2.264427123s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/infrastructure.cluster.x-k8s.io/v1alpha5 • [8.827 seconds] ------------------------------ integreatly managed-api HAPPY PATH A10 - Verify CRO Postgres CRs Successful /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:36:22.926914 71 request.go:690] Waited for 3.464940763s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/addons.managed.openshift.io/v1alpha1 • [8.663 seconds] ------------------------------ integreatly managed-api HAPPY PATH A11 - Verify CRO Redis CRs Successful /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 • [8.651 seconds] ------------------------------ integreatly managed-api HAPPY PATH A13 - Verify Deployment resources have the expected replicas /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:36:32.995727 71 request.go:690] Waited for 1.026079133s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/monitoring.openshift.io/v1alpha1 • [9.047 seconds] ------------------------------ integreatly managed-api HAPPY PATH A14 - Verify Deployment Config resources have the expected replicas /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:36:43.040475 71 request.go:690] Waited for 2.022374751s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/opendatahub.io/v1alpha • [8.931 seconds] ------------------------------ integreatly managed-api HAPPY PATH A15 - Verify Stateful Set resources have the expected replicas /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:36:53.074368 71 request.go:690] Waited for 3.125279299s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/monitoring.coreos.com/v1beta1 • [8.674 seconds] ------------------------------ integreatly managed-api HAPPY PATH A26 - Verify Sendgrid Credentials Are Configured Properly /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:37:04.460248 71 request.go:690] Waited for 1.028446464s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/autoscaling/v2 • [8.720 seconds] ------------------------------ integreatly managed-api HAPPY PATH A27 + A28 - Verify pod priority class is created and set /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:37:14.474214 71 request.go:690] Waited for 2.323159537s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/storage.k8s.io/v1beta1 • [8.975 seconds] ------------------------------ integreatly managed-api HAPPY PATH A32 - Validate SSO config /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:37:24.497425 71 request.go:690] Waited for 3.371587835s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/serving.kserve.io/v1alpha1 Checking requested resources present in Keycloak pods Checking the link for admin is available Checking the link for dedicated admin is available A32 - Validate SSO config test succeeded • [8.783 seconds] ------------------------------ integreatly managed-api HAPPY PATH C01 - Verify Alerts are not pending or firing apart from DeadMansSwitch /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMUserSsoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Firing alert to be ignored: RHOAMRhssoAvailability2hto1dErrorBudgetBurn Pattern ignored: soAvailability2hto1dErrorBudgetBurn Waiting 1 minute for alerts to normalise before retrying Namespace: redhat-rhoam-operator Pod name: 805d1d7867c194ec46d850740ef8b23761894d7fdd7b21b844a46d38c8pg482, Status: Succeeded Pod name: addon-managed-api-service-catalog-pwdpx, Status: Running Pod name: rhmi-operator-cb5495956-2l84k, Status: Running Namespace: redhat-rhoam-operator-observability Pod name: alertmanager-rhoam-0, Status: Running Pod name: alertmanager-rhoam-1, Status: Running Pod name: blackbox-exporter-7f95b96f55-qlfgd, Status: Running Pod name: prometheus-rhoam-0, Status: Running Namespace: redhat-rhoam-cloud-resources-operator Pod name: a5681f7f82150f1a61ef7fa4c2848fce78d5eaf8689d0064649fa6534fdk9kf, Status: Succeeded Pod name: cloud-resource-operator-55fc5b5d5f-pdvb4, Status: Running Pod name: rhmi-registry-cs-7vh5z, Status: Running Namespace: redhat-rhoam-rhsso Pod name: keycloak-0, Status: Running Pod name: keycloak-1, Status: Running Namespace: redhat-rhoam-rhsso-operator Pod name: 650a3ac5610ff48f1c5d6a5e0652296fb127e7141b523b508c62877bd1p29jc, Status: Succeeded Pod name: rhmi-registry-cs-k7dnt, Status: Running Pod name: rhsso-operator-669f4c4fb6-nb9z5, Status: Running Namespace: redhat-rhoam-3scale Pod name: apicast-production-1-deploy, Status: Succeeded Pod name: apicast-production-3-deploy, Status: Succeeded Pod name: apicast-production-3-p7fzh, Status: Running Pod name: apicast-production-3-pckzm, Status: Running Pod name: apicast-staging-1-deploy, Status: Succeeded Pod name: apicast-staging-3-deploy, Status: Succeeded Pod name: apicast-staging-3-h9qw4, Status: Running Pod name: apicast-staging-3-kbtgk, Status: Running Pod name: apicast-staging-3-lzspt, Status: Running Pod name: backend-cron-1-deploy, Status: Succeeded Pod name: backend-cron-2-deploy, Status: Succeeded Pod name: backend-cron-2-wdbtd, Status: Running Pod name: backend-listener-1-deploy, Status: Succeeded Pod name: backend-listener-3-cstkx, Status: Running Pod name: backend-listener-3-deploy, Status: Succeeded Pod name: backend-listener-3-s7ntf, Status: Running Pod name: backend-worker-1-deploy, Status: Succeeded Pod name: backend-worker-2-deploy, Status: Succeeded Pod name: backend-worker-2-w2zpx, Status: Running Pod name: backend-worker-2-xhjtd, Status: Running Pod name: marin3r-instance-55ffbdbcb4-dqfz5, Status: Running Pod name: system-app-1-deploy, Status: Succeeded Pod name: system-app-1-hook-post, Status: Succeeded Pod name: system-app-1-hook-pre, Status: Succeeded Pod name: system-app-3-deploy, Status: Succeeded Pod name: system-app-3-hook-post, Status: Succeeded Pod name: system-app-3-hook-pre, Status: Succeeded Pod name: system-app-3-q88j2, Status: Running Pod name: system-app-3-rphw8, Status: Running Pod name: system-app-3-xv88q, Status: Running Pod name: system-memcache-1-deploy, Status: Succeeded Pod name: system-memcache-2-deploy, Status: Succeeded Pod name: system-memcache-2-hqq9j, Status: Running Pod name: system-searchd-1-deploy, Status: Succeeded Pod name: system-searchd-2-deploy, Status: Succeeded Pod name: system-searchd-2-q7pwg, Status: Running Pod name: system-sidekiq-1-deploy, Status: Succeeded Pod name: system-sidekiq-3-4qbr5, Status: Running Pod name: system-sidekiq-3-cq7nm, Status: Running Pod name: system-sidekiq-3-deploy, Status: Succeeded Pod name: system-sidekiq-3-zpsl5, Status: Running Pod name: zync-1-deploy, Status: Succeeded Pod name: zync-2-9mkx2, Status: Running Pod name: zync-2-deploy, Status: Succeeded Pod name: zync-2-p8rhv, Status: Running Pod name: zync-2-q2pfl, Status: Running Pod name: zync-database-1-deploy, Status: Succeeded Pod name: zync-database-2-2jfh9, Status: Running Pod name: zync-database-2-deploy, Status: Succeeded Pod name: zync-que-1-deploy, Status: Succeeded Pod name: zync-que-2-9dc9m, Status: Running Pod name: zync-que-2-deploy, Status: Succeeded Pod name: zync-que-2-dfgnn, Status: Running Pod name: zync-que-2-m2cvh, Status: Running Namespace: redhat-rhoam-3scale-operator Pod name: 87cdf0563593742f5d4db128d41cdc5c6f6c7c86ef6de3c1554caea462xfr9c, Status: Succeeded Pod name: rhmi-registry-cs-k64jq, Status: Running Pod name: threescale-operator-controller-manager-v2-6d4f4d8584-sg2gr, Status: Running Namespace: redhat-rhoam-marin3r-operator Pod name: marin3r-controller-manager-fbf5c484c-ktmdw, Status: Running Pod name: marin3r-controller-webhook-577ddd474d-5zdl2, Status: Running Pod name: marin3r-controller-webhook-577ddd474d-gbfr5, Status: Running Pod name: rhmi-registry-cs-f5jh7, Status: Running Namespace: redhat-rhoam-marin3r Pod name: ratelimit-7f95cd987c-bpglc, Status: Running Pod name: ratelimit-7f95cd987c-wr7ts, Status: Running Namespace: redhat-rhoam-customer-monitoring-operator Pod name: grafana-deployment-6dccc99f94-lpnvq, Status: Running Pod name: grafana-operator-controller-manager-7c549cd4d-4szlf, Status: Running Pod name: rhmi-registry-cs-q96v9, Status: Running Namespace: redhat-rhoam-user-sso Pod name: keycloak-0, Status: Running Pod name: keycloak-1, Status: Running Namespace: redhat-rhoam-user-sso-operator Pod name: 650a3ac5610ff48f1c5d6a5e0652296fb127e7141b523b508c62877bd1vsv8q, Status: Succeeded Pod name: rhmi-registry-cs-cxdwp, Status: Running Pod name: rhsso-operator-7c59b7d45d-9lg58, Status: Running [FAILED] in [It] - /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/node.go:463 @ 09/18/23 19:52:34.375 • [FAILED] [909.299 seconds] integreatly managed-api HAPPY PATH [It] C01 - Verify Alerts are not pending or firing apart from DeadMansSwitch /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 [FAILED] The following alerts were pending: alert: RHOAMUserSsoAvailability6hto3dErrorBudgetBurn alert: RHOAMRhssoAvailability6hto3dErrorBudgetBurn In [It] at: /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/node.go:463 @ 09/18/23 19:52:34.375 ------------------------------ integreatly managed-api HAPPY PATH E10 - Verify Customer Grafana Route is accessible /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:52:35.426962 71 request.go:690] Waited for 1.024777038s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/metal3.io/v1alpha1 • [8.755 seconds] ------------------------------ integreatly managed-api HAPPY PATH F02 - Verify PodDisruptionBudgets exist /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:52:45.436998 71 request.go:690] Waited for 2.274220625s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/metal3.io/v1alpha1 • [8.650 seconds] ------------------------------ SSSS ------------------------------ integreatly managed-api IDP BASED B01B - Verify users can login to products /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:52:55.481261 71 request.go:690] Waited for 3.674830156s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/discovery.k8s.io/v1 env var USER_NUMBERS was not set, defaulting to 1 Performing OpenShift HTTP client setup with URL https://console-openshift-console.apps.coexist-52ad.lhej.s1.devshift.org/auth/login Error clicking IDP link Error while checking IDP is setup, retrying: error occurred during oauth login: OpenshiftClientSubmitForm(): failed to click testing-idp identity provider in oauth proxy login, ensure the identity provider exists on the cluster: Element not found matching expr 'a:contains('testing-idp')'. Performing OpenShift HTTP client setup with URL https://console-openshift-console.apps.coexist-52ad.lhej.s1.devshift.org/auth/login Error clicking IDP link Error while checking IDP is setup, retrying: error occurred during oauth login: OpenshiftClientSubmitForm(): failed to click testing-idp identity provider in oauth proxy login, ensure the identity provider exists on the cluster: Element not found matching expr 'a:contains('testing-idp')'. Performing OpenShift HTTP client setup with URL https://console-openshift-console.apps.coexist-52ad.lhej.s1.devshift.org/auth/login Error clicking IDP link Error while checking IDP is setup, retrying: error occurred during oauth login: OpenshiftClientSubmitForm(): failed to click testing-idp identity provider in oauth proxy login, ensure the identity provider exists on the cluster: Element not found matching expr 'a:contains('testing-idp')'. Performing OpenShift HTTP client setup with URL https://console-openshift-console.apps.coexist-52ad.lhej.s1.devshift.org/auth/login Error clicking IDP link Error while checking IDP is setup, retrying: error occurred during oauth login: OpenshiftClientSubmitForm(): failed to click testing-idp identity provider in oauth proxy login, ensure the identity provider exists on the cluster: Element not found matching expr 'a:contains('testing-idp')'. Performing OpenShift HTTP client setup with URL https://console-openshift-console.apps.coexist-52ad.lhej.s1.devshift.org/auth/login Error clicking IDP link Error while checking IDP is setup, retrying: error occurred during oauth login: OpenshiftClientSubmitForm(): failed to click testing-idp identity provider in oauth proxy login, ensure the identity provider exists on the cluster: Element not found matching expr 'a:contains('testing-idp')'. Performing OpenShift HTTP client setup with URL https://console-openshift-console.apps.coexist-52ad.lhej.s1.devshift.org/auth/login Trying to log into Customer Grafana: https://grafana-route-redhat-rhoam-customer-monitoring-operator.apps.coexist-52ad.lhej.s1.devshift.org for user: test-user01 [FAILED] in [It] - /go/src/github.com/integr8ly/integreatly-operator/test/common/product_login.go:58 @ 09/18/23 19:54:04.875 • [FAILED] [73.095 seconds] integreatly managed-api IDP BASED [It] B01B - Verify users can login to products /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 [FAILED] ChromeDP test failed with error: chrome failed to start: The setuid sandbox is not running as root. Common causes: * An unprivileged process using ptrace on it, like a debugger. * A parent process set prctl(PR_SET_NO_NEW_PRIVS, ...) Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted [0918/195403.345657:FATAL:zygote_host_impl_linux.cc(201)] Check failed: . : No such file or directory (2) [0918/195403.349019:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2) [0918/195403.349077:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: No such file or directory (2) [0918/195403.350736:ERROR:directory_reader_posix.cc(42)] opendir /tmp/Crashpad/attachments/cd22f69e-a270-4b34-a11e-d5fc2b8f75c5: No such file or directory (2) In [It] at: /go/src/github.com/integr8ly/integreatly-operator/test/common/product_login.go:58 @ 09/18/23 19:54:04.875 ------------------------------ integreatly managed-api IDP BASED B03 - Verify RHMI Developer User Permissions are Correct /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:54:05.928267 71 request.go:690] Waited for 1.016341709s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/kubeflow.org/v1beta1 not creating IDP, it's already created retrieved console master URL console-openshift-console.apps.coexist-52ad.lhej.s1.devshift.org retrieved openshift-Oauth route Performing OpenShift HTTP client setup with URL https://console-openshift-console.apps.coexist-52ad.lhej.s1.devshift.org/auth/login retrieved rhmi developer user tokens testing managed api developer projects Verifying RHMI Developer permissions for RHMI Resource • [10.920 seconds] ------------------------------ integreatly managed-api IDP BASED B04 - Verify Dedicated Admin User Permissions are Correct /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:54:16.847294 71 request.go:690] Waited for 1.015283564s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/autoscaling/v2 not creating IDP, it's already created Performing OpenShift HTTP client setup with URL https://console-openshift-console.apps.coexist-52ad.lhej.s1.devshift.org/auth/login Verifying Dedicated admin permissions for Secrets Resource • [9.952 seconds] ------------------------------ integreatly managed-api IDP BASED B06 - Verify users with no email get default email /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:54:26.850108 71 request.go:690] Waited for 1.057935643s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/samples.operator.openshift.io/v1 not creating IDP, it's already created • [22.699 seconds] ------------------------------ SSS ------------------------------ integreatly managed-api IDP BASED H03 - Verify 3scale CRUDL permissions /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:54:49.499816 71 request.go:690] Waited for 1.026855054s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/metal3.io/v1alpha1 STEP: Ensure testing IDP is configured @ 09/18/23 19:54:57.122 not creating IDP, it's already created STEP: Get RHMI CR @ 09/18/23 19:54:57.137 STEP: Get 3Scale URL and 3Scale clint @ 09/18/23 19:54:57.151 STEP: Login to 3Scale and become admin @ 09/18/23 19:54:57.151 STEP: Login to 3Scale @ 09/18/23 19:54:57.151 Attempting to open threescale login page with url: https://3scale-admin.apps.coexist-52ad.lhej.s1.devshift.org/p/login STEP: Open 3Scale login page @ 09/18/23 19:54:57.151 STEP: Check if user authenticated @ 09/18/23 19:54:57.22 User is not authenticated yet, going to authenticate through rhsso url https://3scale-admin.apps.coexist-52ad.lhej.s1.devshift.org/p/login STEP: Authenticate through RHSSO @ 09/18/23 19:54:57.22 STEP: Check if user authenticated @ 09/18/23 19:54:57.719 User is not authenticated through rhsso yet, going to select the IDP https://oauth-openshift.apps.coexist-52ad.lhej.s1.devshift.org/oauth/authorize?scope=user%3Afull&state=y4Loz81XVu4PgczSrLhsv8S8UrxMRxzJmHyRuo0AsdI.-SM11oWQukM.3scale&response_type=code&client_id=redhat-rhoam-rhsso&redirect_uri=https%3A%2F%2Fkeycloak-edge-redhat-rhoam-rhsso.apps.coexist-52ad.lhej.s1.devshift.org%2Fauth%2Frealms%2Fopenshift%2Fbroker%2Fopenshift-v4%2Fendpoint STEP: Select the testing IDP to authenticate through RHSSO @ 09/18/23 19:54:57.719 STEP: Check if user authenticated @ 09/18/23 19:54:57.794 User is not authenticated after selecting the IDP yet, going to submit the form to authenticate the user through rhsso https://keycloak-edge-redhat-rhoam-rhsso.apps.coexist-52ad.lhej.s1.devshift.org/auth/realms/testing-idp/protocol/openid-connect/auth?client_id=openshift&redirect_uri=https%3A%2F%2Foauth-openshift.apps.coexist-52ad.lhej.s1.devshift.org%2Foauth2callback%2Ftesting-idp&response_type=code&scope=openid&state=Y3NyZj1uTVQ5dkRRckNaX3dERXo1cUp0YzRyNlVZNE1IWk5MMUNsSU5kaUk3YUFrJnRoZW49JTJGb2F1dGglMkZhdXRob3JpemUlM0ZjbGllbnRfaWQlM0RyZWRoYXQtcmhvYW0tcmhzc28lMjZpZHAlM0R0ZXN0aW5nLWlkcCUyNnJlZGlyZWN0X3VyaSUzRGh0dHBzJTI1M0ElMjUyRiUyNTJGa2V5Y2xvYWstZWRnZS1yZWRoYXQtcmhvYW0tcmhzc28uYXBwcy5jb2V4aXN0LTUyYWQubGhlai5zMS5kZXZzaGlmdC5vcmclMjUyRmF1dGglMjUyRnJlYWxtcyUyNTJGb3BlbnNoaWZ0JTI1MkZicm9rZXIlMjUyRm9wZW5zaGlmdC12NCUyNTJGZW5kcG9pbnQlMjZyZXNwb25zZV90eXBlJTNEY29kZSUyNnNjb3BlJTNEdXNlciUyNTNBZnVsbCUyNnN0YXRlJTNEeTRMb3o4MVhWdTRQZ2N6U3JMaHN2OFM4VXJ4TVJ4ekptSHlSdW8wQXNkSS4tU00xMW9XUXVrTS4zc2NhbGU%3D STEP: Submit login form for testing IDP @ 09/18/23 19:54:57.794 STEP: Check if user authenticated after submitting RHSSO login form @ 09/18/23 19:54:58.655 STEP: Ensure 3Scale is available @ 09/18/23 19:54:58.723 STEP: Create a product @ 09/18/23 19:54:58.866 STEP: Delete the product @ 09/18/23 19:54:59.572 • [11.688 seconds] ------------------------------ integreatly managed-api IDP BASED H07 - ThreeScale User Promotion /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 19:55:01.187527 71 request.go:690] Waited for 1.01298918s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/autoscaling.openshift.io/v1 env var USER_NUMBERS was not set, defaulting to 1 not creating IDP, it's already created STEP: Login to 3Scale @ 09/18/23 19:55:08.767 Attempting to open threescale login page with url: https://3scale-admin.apps.coexist-52ad.lhej.s1.devshift.org/p/login STEP: Open 3Scale login page @ 09/18/23 19:55:08.767 STEP: Check if user authenticated @ 09/18/23 19:55:08.833 User is not authenticated yet, going to authenticate through rhsso url https://3scale-admin.apps.coexist-52ad.lhej.s1.devshift.org/p/login STEP: Authenticate through RHSSO @ 09/18/23 19:55:08.833 STEP: Check if user authenticated @ 09/18/23 19:55:09.009 User is not authenticated through rhsso yet, going to select the IDP https://oauth-openshift.apps.coexist-52ad.lhej.s1.devshift.org/oauth/authorize?scope=user%3Afull&state=49mweoVrcACeVOGzSUNT7295OuKXUAfVN6ZuXbodups.dYB4JgLfkjs.3scale&response_type=code&client_id=redhat-rhoam-rhsso&redirect_uri=https%3A%2F%2Fkeycloak-edge-redhat-rhoam-rhsso.apps.coexist-52ad.lhej.s1.devshift.org%2Fauth%2Frealms%2Fopenshift%2Fbroker%2Fopenshift-v4%2Fendpoint STEP: Select the testing IDP to authenticate through RHSSO @ 09/18/23 19:55:09.009 STEP: Check if user authenticated @ 09/18/23 19:55:09.051 User is not authenticated after selecting the IDP yet, going to submit the form to authenticate the user through rhsso https://keycloak-edge-redhat-rhoam-rhsso.apps.coexist-52ad.lhej.s1.devshift.org/auth/realms/testing-idp/protocol/openid-connect/auth?client_id=openshift&redirect_uri=https%3A%2F%2Foauth-openshift.apps.coexist-52ad.lhej.s1.devshift.org%2Foauth2callback%2Ftesting-idp&response_type=code&scope=openid&state=Y3NyZj1HQXJLaGJhdFdWaGZUdVVOdXhueTNHSnc1aGhRRHl2bFBfZUZ3WFhxeEU4JnRoZW49JTJGb2F1dGglMkZhdXRob3JpemUlM0ZjbGllbnRfaWQlM0RyZWRoYXQtcmhvYW0tcmhzc28lMjZpZHAlM0R0ZXN0aW5nLWlkcCUyNnJlZGlyZWN0X3VyaSUzRGh0dHBzJTI1M0ElMjUyRiUyNTJGa2V5Y2xvYWstZWRnZS1yZWRoYXQtcmhvYW0tcmhzc28uYXBwcy5jb2V4aXN0LTUyYWQubGhlai5zMS5kZXZzaGlmdC5vcmclMjUyRmF1dGglMjUyRnJlYWxtcyUyNTJGb3BlbnNoaWZ0JTI1MkZicm9rZXIlMjUyRm9wZW5zaGlmdC12NCUyNTJGZW5kcG9pbnQlMjZyZXNwb25zZV90eXBlJTNEY29kZSUyNnNjb3BlJTNEdXNlciUyNTNBZnVsbCUyNnN0YXRlJTNENDltd2VvVnJjQUNlVk9HelNVTlQ3Mjk1T3VLWFVBZlZONlp1WGJvZHVwcy5kWUI0SmdMZmtqcy4zc2NhbGU%3D STEP: Submit login form for testing IDP @ 09/18/23 19:55:09.051 STEP: Check if user authenticated after submitting RHSSO login form @ 09/18/23 19:55:09.74 Trying to log into 3scale for user: test-user01 STEP: Login to 3Scale @ 09/18/23 19:55:09.85 Attempting to open threescale login page with url: https://3scale-admin.apps.coexist-52ad.lhej.s1.devshift.org/p/login STEP: Open 3Scale login page @ 09/18/23 19:55:09.85 STEP: Check if user authenticated @ 09/18/23 19:55:09.919 User is not authenticated yet, going to authenticate through rhsso url https://3scale-admin.apps.coexist-52ad.lhej.s1.devshift.org/p/login STEP: Authenticate through RHSSO @ 09/18/23 19:55:09.919 STEP: Check if user authenticated @ 09/18/23 19:55:10.074 User is not authenticated through rhsso yet, going to select the IDP https://oauth-openshift.apps.coexist-52ad.lhej.s1.devshift.org/oauth/authorize?scope=user%3Afull&state=JNzqP61Y8f_1EydgkxzODOfOdE7l7Wpuvg-qPxLVo1I.Rled6MMGwyE.3scale&response_type=code&client_id=redhat-rhoam-rhsso&redirect_uri=https%3A%2F%2Fkeycloak-edge-redhat-rhoam-rhsso.apps.coexist-52ad.lhej.s1.devshift.org%2Fauth%2Frealms%2Fopenshift%2Fbroker%2Fopenshift-v4%2Fendpoint STEP: Select the testing IDP to authenticate through RHSSO @ 09/18/23 19:55:10.074 STEP: Check if user authenticated @ 09/18/23 19:55:10.116 User is not authenticated after selecting the IDP yet, going to submit the form to authenticate the user through rhsso https://keycloak-edge-redhat-rhoam-rhsso.apps.coexist-52ad.lhej.s1.devshift.org/auth/realms/testing-idp/protocol/openid-connect/auth?client_id=openshift&redirect_uri=https%3A%2F%2Foauth-openshift.apps.coexist-52ad.lhej.s1.devshift.org%2Foauth2callback%2Ftesting-idp&response_type=code&scope=openid&state=Y3NyZj1mSHJGTmIzR0hCcURSOEQ3ZkJXV2lDcEQwdU9VRTh3bDg0ejFyQ1lSTmpnJnRoZW49JTJGb2F1dGglMkZhdXRob3JpemUlM0ZjbGllbnRfaWQlM0RyZWRoYXQtcmhvYW0tcmhzc28lMjZpZHAlM0R0ZXN0aW5nLWlkcCUyNnJlZGlyZWN0X3VyaSUzRGh0dHBzJTI1M0ElMjUyRiUyNTJGa2V5Y2xvYWstZWRnZS1yZWRoYXQtcmhvYW0tcmhzc28uYXBwcy5jb2V4aXN0LTUyYWQubGhlai5zMS5kZXZzaGlmdC5vcmclMjUyRmF1dGglMjUyRnJlYWxtcyUyNTJGb3BlbnNoaWZ0JTI1MkZicm9rZXIlMjUyRm9wZW5zaGlmdC12NCUyNTJGZW5kcG9pbnQlMjZyZXNwb25zZV90eXBlJTNEY29kZSUyNnNjb3BlJTNEdXNlciUyNTNBZnVsbCUyNnN0YXRlJTNESk56cVA2MVk4Zl8xRXlkZ2t4ek9ET2ZPZEU3bDdXcHV2Zy1xUHhMVm8xSS5SbGVkNk1NR3d5RS4zc2NhbGU%3D STEP: Submit login form for testing IDP @ 09/18/23 19:55:10.116 STEP: Check if user authenticated after submitting RHSSO login form @ 09/18/23 19:55:10.866 • [361.702 seconds] ------------------------------ SS ------------------------------ integreatly managed-api IDP BASED H34 - Verify 3scale custom SMTP full config /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 20:01:02.965519 71 request.go:690] Waited for 1.026657627s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/imageregistry.operator.openshift.io/v1 time="2023-09-18T20:01:10Z" level=info msg="operator is hive managed" [SKIPPED] in [It] - /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/node.go:463 @ 09/18/23 20:01:10.558 S [SKIPPED] [8.720 seconds] integreatly managed-api IDP BASED [It] H34 - Verify 3scale custom SMTP full config /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 [SKIPPED] Addon custom smtp values are not fully configured. This test is not ok to run. In [It] at: /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/node.go:463 @ 09/18/23 20:01:10.558 ------------------------------ integreatly managed-api IDP BASED H35 - Verify 3scale custom SMTP partial config /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 20:01:13.010230 71 request.go:690] Waited for 2.426037672s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/kubeflow.org/v1 time="2023-09-18T20:01:19Z" level=info msg="operator is hive managed" [SKIPPED] in [It] - /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/node.go:463 @ 09/18/23 20:01:19.207 S [SKIPPED] [8.649 seconds] integreatly managed-api IDP BASED [It] H35 - Verify 3scale custom SMTP partial config /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 [SKIPPED] Addon custom smtp values are not partial configured. This test is not ok to run. In [It] at: /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/node.go:463 @ 09/18/23 20:01:19.207 ------------------------------ S ------------------------------ integreatly managed-api OBSERVABILITY TESTS A06 - Verify PVC /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 20:01:25.061428 71 request.go:690] Waited for 1.026314746s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/capabilities.3scale.net/v1alpha1 • [8.652 seconds] ------------------------------ integreatly managed-api OBSERVABILITY TESTS C04 - Verify Alerts exist /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 20:01:35.109267 71 request.go:690] Waited for 2.42506918s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/batch/v1 • [8.861 seconds] ------------------------------ integreatly managed-api OBSERVABILITY TESTS C08B - Verify alert links to SOPs /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 20:01:45.119479 71 request.go:690] Waited for 3.576280366s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/controlplane.operator.openshift.io/v1alpha1 [SKIPPED] in [It] - /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/node.go:463 @ 09/18/23 20:01:45.308 S [SKIPPED] [8.588 seconds] integreatly managed-api OBSERVABILITY TESTS [It] C08B - Verify alert links to SOPs /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 [SKIPPED] Gitlab token not provided, use GITLAB_TOKEN environment variable to specify it In [It] at: /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/node.go:463 @ 09/18/23 20:01:45.308 ------------------------------ integreatly managed-api OBSERVABILITY TESTS C10B - Verify Prometheus blackbox targets /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 • [8.869 seconds] ------------------------------ integreatly managed-api OBSERVABILITY TESTS E09 - Verify customer dashboards exist /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 20:01:55.230121 71 request.go:690] Waited for 1.026154737s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/policy/v1 • [8.834 seconds] ------------------------------ integreatly managed-api OBSERVABILITY TESTS M02B - Verify RHOAM version metric is exposed in Prometheus /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 20:02:05.263059 71 request.go:690] Waited for 2.225856189s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/managed.openshift.io/v1alpha1 • [8.786 seconds] ------------------------------ SSSSSSS ------------------------------ integreatly managed-api SCALABILITY TESTS F05 - Verify Replicas Scale correctly in Threescale /go/src/github.com/integr8ly/integreatly-operator/test/functional/integreatly_test.go:148 I0918 20:02:15.300136 71 request.go:690] Waited for 3.47450271s due to client-side throttling, not priority and fairness, request: GET:https://api.coexist-52ad.lhej.s1.devshift.org:6443/apis/project.openshift.io/v1 Checking correct number of replicas are set Checking correct number of updated replicas are set Checking correct number of updated replicas are set Number of replicas for apim.Spec.System.AppSpec is not correct : Replicas - 2, Expected - 3 retrying in : 20s seconds Checking correct number of replicas has been reset zync replicas ready 2, expected 3, unavailable replicas 1 zync replicas ready 2, expected 3, unavailable replicas 1 zync replicas ready 2, expected 3, unavailable replicas 1 • [189.168 seconds] ------------------------------ SSSSS ------------------------------ [AfterSuite] /go/src/github.com/integr8ly/integreatly-operator/test/functional/suite_test.go:108 STEP: tearing down the test environment @ 09/18/23 20:05:20.966 [AfterSuite] PASSED [0.001 seconds] ------------------------------ [ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report autogenerated by Ginkgo [ReportAfterSuite] PASSED [0.011 seconds] ------------------------------ Summarizing 5 Failures: [FAIL] integreatly [JustBeforeEach] AWS Specific Tests A12 - Verify CRO BlobStorage CRs Successful /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/suite.go:863 [FAIL] integreatly [JustBeforeEach] managed-api Functional A25 - Verify standalone RHMI VPC exists and is configured properly /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/suite.go:863 [FAIL] integreatly managed-api Functional [It] F04 - Verify AWS s3 blob storage resources exist /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/node.go:463 [FAIL] integreatly managed-api HAPPY PATH [It] C01 - Verify Alerts are not pending or firing apart from DeadMansSwitch /go/pkg/mod/github.com/onsi/ginkgo/v2@v2.9.1/internal/node.go:463 [FAIL] integreatly managed-api IDP BASED [It] B01B - Verify users can login to products /go/src/github.com/integr8ly/integreatly-operator/test/common/product_login.go:58 Ran 32 of 61 Specs in 3198.966 seconds FAIL! -- 27 Passed | 5 Failed | 0 Pending | 29 Skipped You're using deprecated Ginkgo functionality: ============================================= --ginkgo.progress is deprecated . The functionality provided by --progress was confusing and is no longer needed. Use --show-node-events instead to see node entry and exit events included in the timeline of failed and verbose specs. Or you can run with -vv to always see all node events. Lastly, --poll-progress-after and the PollProgressAfter decorator now provide a better mechanism for debugging specs that tend to get stuck. To silence deprecations that can be silenced set the following environment variable: ACK_GINKGO_DEPRECATIONS=2.9.1 --- FAIL: TestAPIs (3221.65s) FAIL