Description of problem:

During the upgrade from RHODS v1.14 to RHODS v1.15. We observed that Alerts "Kubeflow notebook controller pod is not running " and "ODH notebook controller pod is not running " are fired.

I believe this is happening because during the upgrade Prometheus pod is getting restarted before the Kubeflow notebook controller and ODH notebook controller pods are created

Prerequisites (if any, like setup, operators/versions):

RHODS 1.14

Steps to Reproduce

1. Create cluster
2. Install RHODS
3. Upgrade RHODS

Actual results:

Alerts "Kubeflow notebook controller pod is not running " and "ODH notebook controller pod is not running " are are fired during the upgrade

Expected results:

Alerts "Kubeflow notebook controller pod is not running " and "ODH notebook controller pod is not running " are not fired

Reproducibility (Always/Intermittent/Only Once):

Always

Build Details:

Workaround:

Additional info:

logs in https://drive.google.com/drive/folders/1cvy4oM3bDMSYJWbfxEyEWeCpP0fAOXn2