Story

CI scripts refer to all the bash files located in .ibm/pipelines. These scripts handle setting up the cluster, deploying RHDH using Helm chart and Operator, running the tests, and cleaning up. These scripts differ for different cluster types, such as OCP, AKS, GKE, etc. The main difference is between OCP and all the K8s clusters.

As a maintainer of the CI jobs, I want to be able to:

Minimize failures caused by errors in deployment.
Easily investigate failures caused by the CI scripts.
Be able to easily extend our deployments and tests for new functionalities and cluster types.
Use advanced IDE-like tools for developing the scripts, such as Bash Language Server (extension for VS Code: Bash IDE), ShellCheck, shfmt throughout all the scripts.
Unify how tests are skipped and targeted only for specific clusters/install methods.

Background

Partial refactoring was done in RHIDP-2155 and ~~RHIDP-4550~~, but it should be done more throughout.

All bash files in .ibm/pipelines folder should:

use shebang
```
#!/bin/bash
```

use proper sourcing with ShellCheck directive, e.g.:

# shellcheck source=.ibm/pipelines/utils.sh
source "$DIR"/utils.sh

have code deduplicated
use functions rather than use repeatedly hardcoded parts of code
verify that required env variables are populated throughout the setup, to fail early (e.g. `set -u`)
...

Test files in e2e-tests folder

Tests skip should be unified. Currently, we use the definition in playwright.config.ts and tests.skip directives. Using tests.skip in combination with tests.fixme directive seems like a good idea. e.g.:
- Test is only meant to run on OCP and should be always skipped on K8s, e.g.
```
test.skip(() => process.env.IS_OPENSHIFT.includes("false"));
```

Test is currently broken on Operator and should be fixed, e.g.

// TODO: fix https://issues.redhat.com/browse/RHIDP-6492 
test.fixme(() =>process.env.JOB_NAME.includes("operator"));

test.fixme(() => process.env.JOB_NAME.includes("operator"), "TODO: fix https://issues.redhat.com/browse/RHIDP-6492");

Documentation should be updated with the best practices for developing CI scripts and skipping tests.

The usage of ShellCheck and shfmt (Prettier add-on sh) as a pre-commit and GitHub action should be considered to maintain a good state of the codebase.