WHAT
The current alerts for the 3scale portals do not support custom domains. The alert checks need to be exposed in a different fashion.

Two alerts are required for each portal.

Primary Alert:
This alert will be a critical alert. It is designed to side step the DNS configured for the domain by using the cluster router IP. This alert needs to allow for insecure connections.
Example curl command being reproduced is "curl -k https://<custom-domain> --resolve <custom-domain>:443:<ocp router ip>"

Secondary Alert:
This alert is a warning alert. A DNS look up is used when trying to access the portal. The existing alerts ThreeScaleDeveloperUIBBT and ThreeScaleSystemAdminUIBBT can be re-purposed for this alert.

HOW
The secondary alert is relative easy to create as it is the re-purposing of existing alerts.

The Primary alert requires a metric to be exposed with from the RHOAM operator with the state of the portals. In the operator the following operations will be needed to generate the metric.

• Getting the ingress router URL for the default load balancer. Command line example oc get service router-default -o jsonpath=' {.status.loadBalancer.ingress[*].hostname} {"\n"}' -n openshift-ingress
• Doing a DNS lookup for the ingress router URL. Command line example nslookup <ingress router URL>. The nslookup behaviour should be reproducible in GoLang using the net.LookUpIP function. The DNS lookup may return more than one IP address.
• The portal address can now be accessed by recreating the curl {}resolve{-} command in the operator. A 200 status result is expected back. This call also needs to allow connections to insecure domains. Command line example curl -k https://<custom-domain> --resolve <custom-domain>:443:<ingress router ip>. To reproduce the -resolve flag in Golang this example can be used a bases for the implementation.
• New metric is exposed with the result for the portal URL
• New alert is created which consumes the new metric

As bout the primary and secondary alerts are related the SOPs for the alerts should reference and explain the relationship between the alerts.

Concern
If the metric on the primary alert is not be exposed the alert may fire given a false positive or not fire in case of actual issue. This concern is being discussed in Absent Metrics

TESTS

• unit tests added

DONE

• Primary alert created
• Secondary alert created
• SOPs created