Major CRW nightly tests took about 9 hours to finish execution.
We could shorten the test execution time if run the tests in a parallel, taking into account resource limits in QE OCP and on PSI OpenStack.
An example of running the tests in parallel:
- https://main-jenkins-csb-crwqe.apps.ocp4.prod.psi.redhat.com/blue/organizations/jenkins/CodereadyWorkspaces%2Fe2e%2Fcomplex%2Fcrwctl-flow%2Fcrw-crwctl-no-oauth/detail/crw-crwctl-no-oauth/95/pipeline
- https://gitlab.cee.redhat.com/codeready-workspaces/crw-jenkins/-/tree/CRW-1748
stage("Run E2E tests") { steps { script { def stepsToRun = [:] stepsToRun["Run E2E devfile tests"] = { basicCrwPipelineHelper.runAllDevfileTests( productNamespace: namespace, ocpVersion: ocpVersion, ocpInfra: ocpInfra, productUrl: productUrl, testRepoAddress: testRepoAddress, testRepoBranch: testRepoBranch, testRootDir: testRootDir, e2eTestImage: e2eTestImage, e2eTestParameters: "-e TS_SELENIUM_VALUE_OPENSHIFT_OAUTH=false -e TS_SELENIUM_OCP_USERNAME=$openshiftDto.clusterAdminName -e TS_SELENIUM_OCP_PASSWORD=$openshiftDto.clusterAdminPassword" ) } stepsToRun["Run E2E plugin tests"] = { basicCrwPipelineHelper.runAllPluginTests( productNamespace: namespace, ocpVersion: ocpVersion, ocpInfra: ocpInfra, productUrl: productUrl, testRepoAddress: testRepoAddress, testRepoBranch: testRepoBranch, testRootDir: testRootDir, e2eTestImage: e2eTestImage, e2eTestParameters: "-e TS_SELENIUM_VALUE_OPENSHIFT_OAUTH=false -e TS_TEST_OPENSHIFT_PLUGIN_USERNAME=$openshiftDto.clusterAdminName -e TS_TEST_OPENSHIFT_PLUGIN_PASSWORD=$openshiftDto.clusterAdminPassword " ) } stepsToRun["failFast"] = false parallel stepsToRun } } } }
Next problems of parallel test execution have occurred https://main-jenkins-csb-crwqe.apps.ocp4.prod.psi.redhat.com/job/CodereadyWorkspaces/job/e2e/job/basic/job/typescript-tests/4335/console
1. Tests failed on login step.
2. Test called method "TestWorkspaceUtil.cleanUpAllWorkspaces()" and failed.
Root cause: E2E tests are being run with "-e DELETE_WORKSPACE_ON_FAILED_TEST=true"
✓ Create workspace using API (69ms)
Open workspace
▼ DriverHelper.navigateToUrl https://codeready-crw-crwctl-no-oauth.apps.ocp46.crw-qe.com/dashboard/#/ide/admin/xml-plugin-test
✓ Open workspace (181ms)
Wait until created workspace is started
▼ Ide.waitAndSwitchToIdeFrame
‣ DriverHelper.waitAndSwitchToFrame By(css selector, iframe#ide-iframe)
[ERROR] Switching to IDE frame failed after 900000 timeout.
1) Wait until created workspace is started
[ERROR] Test is failed.
Additional information (logs, dumps, screenshots) you can find in the
"./report/The_"VscodeXmlPlugin"_userstory_Create_workspace_Wait_until_created_workspace_is_started" folder.
3 passing (15m)
1 failing
1) The "VscodeXmlPlugin" userstory
Create workspace
Wait until created workspace is started:
TimeoutError: Waiting to be able to switch to frame
Wait timed out after 900010ms
at /tmp/e2e-codeready/.tmp/che/tests/e2e/node_modules/selenium-webdriver/lib/promise.js:2201:17
at ManagedPromise.invokeCallback_ (/tmp/e2e-codeready/.tmp/che/tests/e2e/node_modules/selenium-webdriver/lib/promise.js:1376:14)
at TaskQueue.execute_ (/tmp/e2e-codeready/.tmp/che/tests/e2e/node_modules/selenium-webdriver/lib/promise.js:3084:14)
at TaskQueue.executeNext_ (/tmp/e2e-codeready/.tmp/che/tests/e2e/node_modules/selenium-webdriver/lib/promise.js:3067:27)
at /tmp/e2e-codeready/.tmp/che/tests/e2e/node_modules/selenium-webdriver/lib/promise.js:2927:27
at /tmp/e2e-codeready/.tmp/che/tests/e2e/node_modules/selenium-webdriver/lib/promise.js:668:7
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
From: Task: Waiting to be able to switch to frame
at scheduleWait (/tmp/e2e-codeready/.tmp/che/tests/e2e/node_modules/selenium-webdriver/lib/promise.js:2188:20)
at ControlFlow.wait (/tmp/e2e-codeready/.tmp/che/tests/e2e/node_modules/selenium-webdriver/lib/promise.js:2517:12)
at thenableWebDriverProxy.wait (/tmp/e2e-codeready/.tmp/che/tests/e2e/node_modules/selenium-webdriver/lib/webdriver.js:934:29)
at DriverHelper.waitAndSwitchToFrame (/tmp/e2e-codeready/.tmp/che/tests/e2e/dist/utils/DriverHelper.js:468:27)
at Ide.waitAndSwitchToIdeFrame (/tmp/e2e-codeready/.tmp/che/tests/e2e/dist/pageobjects/ide/Ide.js:53:37)
at /tmp/e2e-codeready/dist/tests/plugins/VscodeXmlPlugin.spec.js:46:23
at Generator.next (<anonymous>)
at /tmp/e2e-codeready/dist/tests/plugins/VscodeXmlPlugin.spec.js:17:71
at new Promise (<anonymous>)
at __awaiter (/tmp/e2e-codeready/dist/tests/plugins/VscodeXmlPlugin.spec.js:13:12)
at Context.<anonymous> (/tmp/e2e-codeready/dist/tests/plugins/VscodeXmlPlugin.spec.js:45:63)
at callFn (/tmp/e2e-codeready/node_modules/mocha/lib/runnable.js:387:21)
at Test.Runnable.run (/tmp/e2e-codeready/node_modules/mocha/lib/runnable.js:379:7)
at Runner.runTest (/tmp/e2e-codeready/node_modules/mocha/lib/runner.js:535:10)
at /tmp/e2e-codeready/node_modules/mocha/lib/runner.js:653:12
at next (/tmp/e2e-codeready/node_modules/mocha/lib/runner.js:447:14)
at /tmp/e2e-codeready/node_modules/mocha/lib/runner.js:457:7
at next (/tmp/e2e-codeready/node_modules/mocha/lib/runner.js:362:14)
at Immediate.<anonymous> (/tmp/e2e-codeready/node_modules/mocha/lib/runner.js:425:5)
at processImmediate (internal/timers.js:461:21)
(node:327) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
‣ DriverHelper.getDriver
Property DELETE_WORKSPACE_ON_FAILED_TEST se to true - trying to stop and delete running workspace.
▼ TestWorkspaceUtil.cleanUpAllWorkspaces
▼ TestWorkspaceUtil.getIdOfRunningWorkspaces
[WARN] AccessToken is no longer active, checking refreshToken validity...
▼ Received accessToken from openid-connect using RefreshToken.
▼ Issued token at: 1618946826, access token expires in 300 seconds, refresh token expires in 1800 seconds
RequestHandler request:
