OCM_ENV=$API_HOST SET_ENVIRONMENT="1" OC_HOST=$(oc whoami --show-server) CLUSTER_ID=$(cat "${SHARED_DIR}/cluster-id") CLUSTER_NAME=$(cat "${SHARED_DIR}/cluster-name") OCM_TOKEN=$(cat /var/run/secrets/ci.openshift.io/cluster-profile/ocm-token) ROBOT_EXTRA_ARGS="-i $TEST_MARKER -e AutomationBug -e Resources-GPU -e Resources-2GPUS" RUN_SCRIPT_ARGS="--skip-oclogin true --set-urls-variables true --test-artifact-dir ${ARTIFACT_DIR}/results" export OCM_ENV export SET_ENVIRONMENT export OC_HOST export CLUSTER_NAME export OCM_TOKEN export ROBOT_EXTRA_ARGS export RUN_SCRIPT_ARGS export CLUSTER_ID mkdir "$ARTIFACT_DIR/results" # running RHODS testsuite ./ods_ci/build/run.sh RUN SCRIPT ARGS: --skip-oclogin true --set-urls-variables true --test-artifact-dir /logs/artifacts/results ROBOT EXTRA ARGS: -i Smoke -e AutomationBug -e Resources-GPU -e Resources-2GPUS SET TEST ENVIRONMENT: 1 -- USE OCM to install IDPs: 1 -----| SET_ENVIRONMENT option is enabled. ODS-CI is going to configure the test environment for you..|----- ---> Performing log in OCM ocm envirnment: stage Stage) validating user_config.json user_config.json found! Starting json validation.. --> Going through requested users configuration validating prefix: htp-user- --> suffix type: incremental_with_rand_base validating prefix: htp-basic-user- --> suffix type: incremental_with_rand_base --> Going through requested users configuration validating prefix: ldap-op- --> suffix type: incremental_with_rand_base validating prefix: ldap-usr- --> suffix type: incremental_with_rand_base validating prefix: ldap-noaccess- --> suffix type: incremental_with_rand_base validating prefix: ldap-special --> suffix type: custom Stage) Looking for LDAP and HTPASSWD already present in the cluster... ---- | Installing the required IDPs | ---- host: https://api.aws-osd-rhods.e46p.s1.devshift.org:6443 Stage) Setting HTPASSWD Identity provider --> Generating users based on requested configuration elaborating prefix: htp-user- --> suffix type: incremental_with_rand_base elaborating prefix: htp-basic-user- --> suffix type: incremental_with_rand_base --> Configuring HTP IDP and users Cluster name is aws-osd-rhods Configuring IDP for cluster 'aws-osd-rhods' Error: Failed to add IDP to cluster 'aws-osd-rhods': status is 500, identifier is '500', code is 'CLUSTERS-MGMT-500' and operation identifier is '47b82d37-79f8-48f3-a51b-832d83965dd3' Added 'cluster-admins' user 'htp-user-aJSGBqEMVhfitkgLmDzr' to cluster 'aws-osd-rhods' ----> Performing log in with newly created HTP user ----> Performing log in the cluster using oc CLI command Login failed (401 Unauthorized) Verify you have provided the correct credentials. Login failed 1 times. Trying again in 30 seconds (max 7 times). Login failed (401 Unauthorized) Verify you have provided the correct credentials. Login failed 2 times. Trying again in 30 seconds (max 7 times). Login failed (401 Unauthorized) Verify you have provided the correct credentials. Login failed 3 times. Trying again in 30 seconds (max 7 times). Login failed (401 Unauthorized) Verify you have provided the correct credentials. Login failed 4 times. Trying again in 30 seconds (max 7 times). Login failed (401 Unauthorized) Verify you have provided the correct credentials. Login failed 5 times. Trying again in 30 seconds (max 7 times). Login failed (401 Unauthorized) Verify you have provided the correct credentials. Login failed 6 times. Trying again in 30 seconds (max 7 times). Login failed (401 Unauthorized) Verify you have provided the correct credentials. Login failed 7 times. Trying again in 30 seconds (max 7 times). ---> Adding additional HTP users, if needed per requested configuration jq: error (at :42): Cannot iterate over null (null) Adding password for user htp-basic-user-oxLXknaSEbCsglNHGZVr1 Adding password for user htp-basic-user-oxLXknaSEbCsglNHGZVr2 error: exactly one NAME is required, got 0 See 'oc create secret generic -h' for help and examples Stage) Setting LDAP Identity provider --> Generating users based on requested configuration elaborating prefix: ldap-op- --> suffix type: incremental_with_rand_base elaborating prefix: ldap-usr- --> suffix type: incremental_with_rand_base elaborating prefix: ldap-noaccess- --> suffix type: incremental_with_rand_base elaborating prefix: ldap-special --> suffix type: custom --> configuring LDAP server and users namespace/openldap created secret/openldap created service/openldap created deployment.apps/openldap created { "kind": "IdentityProvider", "type": "LDAPIdentityProvider", "href": "/api/clusters_mgmt/v1/clusters/269kn0cd750pu7so4lfhbu2e2dsstogr/identity_providers/269m502inlb2uq2t8ei308afsvfn93hu", "id": "269m502inlb2uq2t8ei308afsvfn93hu", "name": "ldap-provider-qe", "mapping_method": "claim", "ldap": { "attributes": { "id": [ "dn" ], "name": [ "cn" ], "preferred_username": [ "uid" ] }, "bind_dn": "cn=admin,dc=example,dc=org", "insecure": true, "url": "ldap://openldap.openldap.svc.cluster.local:1389/dc=example,dc=org?uid" } } Stage) Configure RHODS test user groups group.user.openshift.io/rhods-admins created group.user.openshift.io/rhods-users created group.user.openshift.io/rhods-noaccess created Error from server (AlreadyExists): groups.user.openshift.io "dedicated-admins" already exists rhods-admins dedicated-admins group.user.openshift.io/rhods-admins added: "ldap-op-rzbUHjghTSwFuKYlCtVN1" group.user.openshift.io/rhods-admins added: "ldap-op-rzbUHjghTSwFuKYlCtVN2" group.user.openshift.io/rhods-admins added: "ldap-op-rzbUHjghTSwFuKYlCtVN3" group.user.openshift.io/rhods-admins added: "ldap-op-rzbUHjghTSwFuKYlCtVN4" group.user.openshift.io/rhods-admins added: "ldap-op-rzbUHjghTSwFuKYlCtVN5" Added 'dedicated-admins' user 'ldap-op-rzbUHjghTSwFuKYlCtVN1' to cluster 'aws-osd-rhods' Added 'dedicated-admins' user 'ldap-op-rzbUHjghTSwFuKYlCtVN2' to cluster 'aws-osd-rhods' Added 'dedicated-admins' user 'ldap-op-rzbUHjghTSwFuKYlCtVN3' to cluster 'aws-osd-rhods' Added 'dedicated-admins' user 'ldap-op-rzbUHjghTSwFuKYlCtVN4' to cluster 'aws-osd-rhods' Added 'dedicated-admins' user 'ldap-op-rzbUHjghTSwFuKYlCtVN5' to cluster 'aws-osd-rhods' rhods-users group.user.openshift.io/rhods-users added: "ldap-usr-QUkXnNuPmrthiFpBTYed1" group.user.openshift.io/rhods-users added: "ldap-usr-QUkXnNuPmrthiFpBTYed2" group.user.openshift.io/rhods-users added: "ldap-usr-QUkXnNuPmrthiFpBTYed3" group.user.openshift.io/rhods-users added: "ldap-usr-QUkXnNuPmrthiFpBTYed4" group.user.openshift.io/rhods-users added: "ldap-usr-QUkXnNuPmrthiFpBTYed5" rhods-noaccess group.user.openshift.io/rhods-noaccess added: "ldap-noaccess-TrxBmILpjQyzoRXSNYak1" group.user.openshift.io/rhods-noaccess added: "ldap-noaccess-TrxBmILpjQyzoRXSNYak2" group.user.openshift.io/rhods-noaccess added: "ldap-noaccess-TrxBmILpjQyzoRXSNYak3" group.user.openshift.io/rhods-noaccess added: "ldap-noaccess-TrxBmILpjQyzoRXSNYak4" group.user.openshift.io/rhods-noaccess added: "ldap-noaccess-TrxBmILpjQyzoRXSNYak5" rhods-users group.user.openshift.io/rhods-users added: "ldap-special." group.user.openshift.io/rhods-users added: "ldap-special^" group.user.openshift.io/rhods-users added: "ldap-special$" group.user.openshift.io/rhods-users added: "ldap-special*" group.user.openshift.io/rhods-users added: "ldap-special?" group.user.openshift.io/rhods-users added: "ldap-special(" group.user.openshift.io/rhods-users added: "ldap-special)" group.user.openshift.io/rhods-users added: "ldap-special[" group.user.openshift.io/rhods-users added: "ldap-special]" group.user.openshift.io/rhods-users added: "ldap-special{" group.user.openshift.io/rhods-users added: "ldap-special}" group.user.openshift.io/rhods-users added: "ldap-special|" group.user.openshift.io/rhods-users added: "ldap-special@" group.user.openshift.io/rhods-users added: "ldap-special;" Stage) Sleeping 180sec to wait for IDPs to become available -----| ODS-CI is starting the tests run...|----- ods_ci/test-variables.yml INFO: getting RHODS URLs from the cluster as per --set-urls-variables OCP Console URL set to: https://console-openshift-console.apps.aws-osd-rhods.e46p.s1.devshift.org RHODS Dashboard URL set to: https://rhods-dashboard-redhat-ods-applications.apps.aws-osd-rhods.e46p.s1.devshift.org RHODS API Server URL set to: https://api.aws-osd-rhods.e46p.s1.devshift.org:6443 RHODS Prometheus URL set to: https://prometheus-redhat-ods-monitoring.apps.aws-osd-rhods.e46p.s1.devshift.org INFO: we found a yq executable skipping OC login as per parameter --skip-oclogin ============================================================================== Tests ============================================================================== Tests.Deploy ============================================================================== Tests.Deploy.Installation ============================================================================== Tests.Deploy.Installation.Post Install :: Post install test cases that main... ============================================================================== Verify That CUDA Build Chain Succeeds :: Check Cuda builds are com... [ WARN ] Executing keyword 'Wait Until Page Contains' failed: Text 'Log in with OpenShift' did not appear in 15 seconds. | FAIL | This command requires htp-user-aJSGBqEMVhfitkgLmDzr to be connected to the cluster (oc login ...) Also teardown failed: Several failures occurred: 1) This command requires htp-user-aJSGBqEMVhfitkgLmDzr to be connected to the cluster (oc login ...) 2) Element 'xpath://li[contains(concat(' ',normalize-space(@class),' '),' jp-DirListing-item ')]' did not disappear in 30 seconds. 3) '"Error from server (NotFound): pods "jupyter-nb-ldap-2dop-2drzbuhjghtswfukylctvn1-0" not found"' does not match '""' ------------------------------------------------------------------------------ Verify RHODS Display Name and Version :: Verify consistent rhods d... | PASS | ------------------------------------------------------------------------------ Verify RHODS Notebooks Network Policies :: Verifies that the netwo... [ WARN ] Executing keyword 'Wait Until Page Contains' failed: Text 'Log in with OpenShift' did not appear in 15 seconds. | FAIL | This command requires htp-user-aJSGBqEMVhfitkgLmDzr to be connected to the cluster (oc login ...) ------------------------------------------------------------------------------ Tests.Deploy.Installation.Post Install :: Post install test cases ... | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== Tests.Deploy.Installation | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== Tests.Deploy | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== Tests.Monitor And Manage ============================================================================== Tests.Monitor And Manage.Metrics ============================================================================== Tests.Monitor And Manage.Metrics.Metrics :: Test suite testing ODS Metrics ============================================================================== Test Existence of Prometheus Alerting Rules :: Verifies the promet... | PASS | ------------------------------------------------------------------------------ Test Existence of Prometheus Recording Rules :: Verifies the prome... | PASS | ------------------------------------------------------------------------------ Tests.Monitor And Manage.Metrics.Metrics :: Test suite testing ODS... | PASS | 2 tests, 2 passed, 0 failed ============================================================================== Tests.Monitor And Manage.Metrics.Billing Metrics :: Test suite testing ODS ... ============================================================================== Verify OpenShift Monitoring Results Are Correct When Running Undef... | FAIL | Element with locator 'link:htpasswd' not found. ------------------------------------------------------------------------------ Test Billing Metric (Notebook Cpu Usage) On OpenShift Monitoring :... [ WARN ] Executing keyword 'Wait Until Page Contains' failed: Text 'Log in with OpenShift' did not appear in 15 seconds. | FAIL | This command requires htp-user-aJSGBqEMVhfitkgLmDzr to be connected to the cluster (oc login ...) ------------------------------------------------------------------------------ Tests.Monitor And Manage.Metrics.Billing Metrics :: Test suite tes... | FAIL | 2 tests, 0 passed, 2 failed ============================================================================== Tests.Monitor And Manage.Metrics | FAIL | 4 tests, 2 passed, 2 failed ============================================================================== Tests.Monitor And Manage.Alerts ============================================================================== Tests.Monitor And Manage.Alerts.Alerts :: RHODS monitoring alerts test suite ============================================================================== Verify All Alerts Severity :: Verifies that all alerts have the ex... | PASS | ------------------------------------------------------------------------------ Verify No Alerts Are Firing Except For DeadManSnitch :: Verifies t... | PASS | ------------------------------------------------------------------------------ Tests.Monitor And Manage.Alerts.Alerts :: RHODS monitoring alerts ... | PASS | 2 tests, 2 passed, 0 failed ============================================================================== Tests.Monitor And Manage.Alerts | PASS | 2 tests, 2 passed, 0 failed ============================================================================== Tests.Monitor And Manage | FAIL | 6 tests, 4 passed, 2 failed ============================================================================== Tests.Ods Dashboard ============================================================================== Tests.Ods Dashboard.Ods Dashboard ============================================================================== Verify RHODS Explore Section Contains Only Expected ISVs :: It ver... | PASS | ------------------------------------------------------------------------------ Verify CSS Style Of Getting Started Descriptions :: Verifies the C... | PASS | ------------------------------------------------------------------------------ Tests.Ods Dashboard.Ods Dashboard | PASS | 2 tests, 2 passed, 0 failed ============================================================================== Tests.Ods Dashboard.Ods Dashboard Projects ============================================================================== Tests.Ods Dashboard.Ods Dashboard Projects.Ods Dashboard Projects :: Suite ... ============================================================================== Verify Data Science Projects Page Is Accessible :: Verifies "Data ... | PASS | ------------------------------------------------------------------------------ Verify User Can Access Jupyter Launcher From DS Project Page :: Ve... | PASS | ------------------------------------------------------------------------------ Verify Workbench Images Have Multiple Versions :: Verifies that wo... | PASS | ------------------------------------------------------------------------------ Tests.Ods Dashboard.Ods Dashboard Projects.Ods Dashboard Projects ... | PASS | 3 tests, 3 passed, 0 failed ============================================================================== Tests.Ods Dashboard.Ods Dashboard Projects.Ods Dashboard Projects Permissio... ============================================================================== Verify User Can Access Permission Tab In Their Owned DS Project ::... | PASS | The Test is executed as part of Suite Setup ------------------------------------------------------------------------------ Tests.Ods Dashboard.Ods Dashboard Projects.Ods Dashboard Projects ... | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ods Dashboard.Ods Dashboard Projects | PASS | 4 tests, 4 passed, 0 failed ============================================================================== Tests.Ods Dashboard.Model Serving ============================================================================== Tests.Ods Dashboard.Model Serving.Model Serving ============================================================================== Verify Model Serving Installation :: Verifies that the core compon... | PASS | ------------------------------------------------------------------------------ Model not deployed, skipping deletion step during teardown Tests.Ods Dashboard.Model Serving.Model Serving | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ods Dashboard.Model Serving | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ods Dashboard | PASS | 7 tests, 7 passed, 0 failed ============================================================================== Tests.Jupyterhub ============================================================================== Tests.Jupyterhub.Base-User-Shutdown-Test :: Test Case that verifies a base ... ============================================================================== Verify Base User Can Stop A Running Server :: Verifies that a base... [ ERROR ] Pods not found in search | PASS | ------------------------------------------------------------------------------ Tests.Jupyterhub.Base-User-Shutdown-Test :: Test Case that verifie... | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Jupyterhub.Multiple-Image-Tags :: Test Suite for the multiple noteboo... ============================================================================== Verify All OOTB Images Have Version Dropdowns :: Verifies all imag... | PASS | ------------------------------------------------------------------------------ Tests.Jupyterhub.Multiple-Image-Tags :: Test Suite for the multipl... | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Jupyterhub.Special-User-Testing ============================================================================== Test Special Usernames | PASS | ------------------------------------------------------------------------------ Tests.Jupyterhub.Special-User-Testing | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Jupyterhub.Test-Minimal-Image ============================================================================== Open RHODS Dashboard | PASS | ------------------------------------------------------------------------------ Can Launch Jupyterhub | PASS | ------------------------------------------------------------------------------ Can Login to Jupyterhub | PASS | ------------------------------------------------------------------------------ Can Spawn Notebook [ WARN ] Executing keyword 'Wait Until Page Contains' failed: Text 'Log in with OpenShift' did not appear in 15 seconds. | PASS | ------------------------------------------------------------------------------ Can Launch Python3 Smoke Test Notebook | PASS | ------------------------------------------------------------------------------ Verify Tensorflow Can Be Installed In The Minimal Python Image Via... | FAIL | Teardown failed: Several failures occurred: 1) This command requires htp-user-aJSGBqEMVhfitkgLmDzr to be connected to the cluster (oc login ...) 2) Element 'xpath://li[contains(concat(' ',normalize-space(@class),' '),' jp-DirListing-item ')]' did not disappear in 30 seconds. 3) '"lost+found minimal-nb-image-test ods-ci-notebooks-main Untitled1.ipynb Untitled2.ipynb Untitled.ipynb"' does not match '""' ------------------------------------------------------------------------------ Verify Jupyterlab Server Pods Are Spawned In A Custom Namespace ::... | PASS | ------------------------------------------------------------------------------ Tests.Jupyterhub.Test-Minimal-Image | FAIL | Suite teardown failed: Several failures occurred: 1) This command requires htp-user-aJSGBqEMVhfitkgLmDzr to be connected to the cluster (oc login ...) 2) Element 'xpath://li[contains(concat(' ',normalize-space(@class),' '),' jp-DirListing-item ')]' did not disappear in 30 seconds. 3) '"lost+found minimal-nb-image-test ods-ci-notebooks-main Untitled1.ipynb Untitled2.ipynb Untitled.ipynb"' does not match '""' 7 tests, 0 passed, 7 failed ============================================================================== Tests.Jupyterhub.Test ============================================================================== Logged Into OpenShift | PASS | ------------------------------------------------------------------------------ Can Launch Jupyterhub | PASS | ------------------------------------------------------------------------------ Can Login To Jupyterhub | PASS | ------------------------------------------------------------------------------ Can Spawn Notebook [ WARN ] Executing keyword 'Wait Until Page Contains' failed: Text 'Log in with OpenShift' did not appear in 15 seconds. | PASS | ------------------------------------------------------------------------------ Tests.Jupyterhub.Test | FAIL | Suite teardown failed: Several failures occurred: 1) This command requires htp-user-aJSGBqEMVhfitkgLmDzr to be connected to the cluster (oc login ...) 2) Element 'xpath://li[contains(concat(' ',normalize-space(@class),' '),' jp-DirListing-item ')]' did not disappear in 30 seconds. 3) '"lost+found minimal-nb-image-test ods-ci-notebooks-main Untitled1.ipynb Untitled2.ipynb Untitled3.ipynb Untitled.ipynb"' does not match '""' 4 tests, 0 passed, 4 failed ============================================================================== Tests.Jupyterhub | FAIL | 14 tests, 3 passed, 11 failed ============================================================================== Tests.Must Gather ============================================================================== Tests.Must Gather.Test-Must-Gather-Logs ============================================================================== Verify that the must-gather image provides RHODS logs and info | PASS | ------------------------------------------------------------------------------ Tests.Must Gather.Test-Must-Gather-Logs | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Must Gather | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ai Apps ============================================================================== Tests.Ai Apps.Anaconda Commercial Edition ============================================================================== Tests.Ai Apps.Anaconda Commercial Edition.Anaconda Commercial Edition :: Te... ============================================================================== Verify Anaconda Professional Is Available In RHODS Dashboard Explo... | PASS | ------------------------------------------------------------------------------ Tests.Ai Apps.Anaconda Commercial Edition.Anaconda Commercial Edit... | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ai Apps.Anaconda Commercial Edition | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ai Apps.Ibm Watson Studio ============================================================================== Tests.Ai Apps.Ibm Watson Studio.Ibm Watson Studio ============================================================================== Verify IBM Watson Studio Is Available In RHODS Dashboard Explore Page | PASS | ------------------------------------------------------------------------------ Tests.Ai Apps.Ibm Watson Studio.Ibm Watson Studio | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ai Apps.Ibm Watson Studio | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ai Apps.Intel Aikit ============================================================================== Tests.Ai Apps.Intel Aikit.Intel Aikit ============================================================================== Verify Intel AIKIT Is Available In RHODS Dashboard Explore Page | PASS | ------------------------------------------------------------------------------ Tests.Ai Apps.Intel Aikit.Intel Aikit | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ai Apps.Intel Aikit | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ai Apps.Rhoam ============================================================================== Tests.Ai Apps.Rhoam.Rhoam ============================================================================== Verify RHOAM Is Available In RHODS Dashboard Explore Page | PASS | ------------------------------------------------------------------------------ Tests.Ai Apps.Rhoam.Rhoam | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ai Apps.Rhoam | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ai Apps.Openvino ============================================================================== Tests.Ai Apps.Openvino.Openvino ============================================================================== Verify OpenVino Is Available In RHODS Dashboard Explore Page | PASS | ------------------------------------------------------------------------------ Tests.Ai Apps.Openvino.Openvino | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ai Apps.Openvino | PASS | 1 test, 1 passed, 0 failed ============================================================================== Tests.Ai Apps | PASS | 5 tests, 5 passed, 0 failed ============================================================================== Tests | FAIL | 36 tests, 21 passed, 15 failed ============================================================================== Output: /logs/artifacts/results/ods-ci-2023-09-16-13-44-NYBIQeqXkq/output.xml XUnit: /logs/artifacts/results/ods-ci-2023-09-16-13-44-NYBIQeqXkq/xunit_test_result.xml Log: /logs/artifacts/results/ods-ci-2023-09-16-13-44-NYBIQeqXkq/log.html Report: /logs/artifacts/results/ods-ci-2023-09-16-13-44-NYBIQeqXkq/test_report.html 15