Description of problem:

While installing RHODs via add-on observed that Installation is getting failed due to missing `redhat-rhods-smtp` secret in redhat-ods-monitoring namespace

There is an issue opened in Manged service project:https://issues.redhat.com/browse/MGDAPI-4136

Prerequisites (if any, like setup, operators/versions):

RHODS 1.11

Steps to Reproduce

1. Create a cluster
2. Install RODs via addon

Actual results:

RHODS installation failed due to missing `redhat-rhods-smtp` secret stage environment

Expected results:

`redhat-rhods-smtp` secret should automatically get created and installation should be successful

Reproducibility (Always/Intermittent/Only Once):

Intermittent/always

Build Details:

Workaround:

the created secret with the below code using oc command:

kind: Secret
apiVersion: v1
metadata:
  name: redhat-rhods-smtp
  namespace: redhat-ods-monitoring
stringData:
  host: "smtp.corp.redhat.com"
  username: "takumar"
  password: "password"
  port: "25"
  tls: "false"

Additional info:

Addon-status:

    "addon_version": {
     "kind": "AddOnVersion",
     "id": "0.0.0",
     "href": "/api/clusters_mgmt/v1/addons/managed-odh/versions/0.0.0"
   },
   "state": "failed",

Deployer log:

[pod/rhods-operator-7f8d8885d8-qq9ct/rhods-deployer] Error from server (NotFound): secrets "redhat-rhods-smtp" not found