Description of problem:

If I fill up the jupyterhub notebook PVC over 90% the alert "user PVC is 90% used" is correctly fired but the e-mail notifying to the user is not properly send

If you check the alertmanager logs you'll find this  error that could be the cause:

level=error ts=2021-09-08T13:08:22.198Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="user-notifications/email[0]: notify retry canceled after 16 attempts: establish connection to server: dial tcp: lookup smtp.corp.redhat.com on 172.30.0.10:53: no such host"

Actual results:

No e-mail alert sent to the address specified in redhat-ods-operator > Secrets > addon-managed-odh-parameters > notification-emai

Expected results:

E-mail with the alert sent to the address

Reproducibility (Always/Intermittent/Only Once):

Always

Build Details: 

RHODS 1.1.1-22

Additional info:

This happens in a OSD cluster where RHODS 1.1.1-22 has been installed with our install script (using jenkins)

I'm running this code in the notebook to fill the pvc:

import os

# user notebook volumes are 2046640128 bytes
sizeBelow90Percent = 1650000000
sizeOver90Percent  = 1841976115
size100Percent = 2046640128

# Fill the volume over 90% to check that the alert is fired
with open('large_file_over90', 'wb') as fout:
    fout.write(os.urandom(sizeOver90Percent))