Description of problem:

1-010_validate-ootb-manage-other-namespace Kuttl test sometimes fails with flaky behaviors Openshift CI.

In the last few weeks, the test has failed around 26 times across 10 PRs with re-runs. It has been seen on all three OCP versions

Test location: https://github.com/redhat-developer/gitops-operator/tree/master/test/openshift/e2e/sequential/1-010_validate-ootb-manage-other-namespace  

Flake board: https://github.com/anandrkskd/gitops-operator-flake-board/blob/main/output/flake-stats.md 

Prerequisites (if any, like setup, operators/versions):{}

Steps to Reproduce

Presubmit jobs run on each PR

Actual results:

Most of the times the failure occurred when resources that were expected to be deleted still existed in the cluster

case.go:361: failed in step 5-check-rbac
case.go:363: resource matched of kind: rbac.authorization.k8s.io/v1, Kind=Role
case.go:363: resource matched of kind: rbac.authorization.k8s.io/v1, Kind=Role
case.go:363: resource matched of kind: rbac.authorization.k8s.io/v1, Kind=RoleBinding 

Logs: https://storage.googleapis.com/origin-ci-test/pr-logs/pull/redhat-developer_gitops-operator/514/pull-ci-redhat-developer-gitops-operator-master-v4.11-kuttl-sequential/1668314389104562176/build-log.txt  

Other failures seem to be because of the previous test failures (1-006_validate_machine_config and 1-018_validate_disable_default_instance) :

test step failed 1-install
resource ArgoCD:openshift-gitops/openshift-gitops: .status.phase: key is missing from map 

Logs: https://storage.googleapis.com/origin-ci-test/pr-logs/pull/redhat-developer_gitops-operator/481/pull-ci-redhat-developer-gitops-operator-master-v4.10-kuttl-sequential/1664374771212619776/build-log.txt 

and

case.go:361: failed in step 1-install
case.go:363: argocds.argoproj.io "openshift-gitops" not found 

Logs: https://storage.googleapis.com/origin-ci-test/pr-logs/pull/redhat-developer_gitops-operator/525/pull-ci-redhat-developer-gitops-operator-master-v4.12-kuttl-sequential/1668135278113460224/build-log.txt 

Expected results:

The test should succeed

Reproducibility (Always/Intermittent/Only Once):

Intermittent 

Dependency

For the last failure (test step failed 1-install) it should probably be fixed with

https://issues.redhat.com/browse/GITOPS-2844 

https://issues.redhat.com/browse/GITOPS-2829 

Acceptance criteria: 

1. Above mentioned test should succeed in Openshift CI

Definition of Done:

• Acceptance criterias are met
• Openshift CI does not encounter any flaky tests

Build Details:

Additional info (Such as Logs, Screenshots, etc):

 *