We want to configure alertmanager to use TLS in openstack. To do that, we should be able to successfully use server-side apply on the Alertmanager CR, but afterwards we will also need to configure prometheus to use the https scheme and correct CA certificates to access alertmanager's API. Unfortunately we currently can't use server-side apply because the spec.alerting.alertmanagers are managed by the observability-operator and every change to it is immediately overwritten.

The resulting related part of Prometheus CR needs to look something like this (mainly the scheme and tlsConfig fields):

apiVersion: monitoring.rhobs/v1
kind: Prometheus
metadata:
  name: metric-storage
  namespace: openstack
spec:
  alerting:
    alertmanagers:
    - apiVersion: v2
      name: metric-storage-alertmanager2
      namespace: openstack
      port: web
      scheme: http
      tlsConfig:
        caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
        serverName: 'alertmanager.svc' 

I think the scheme and tlsConfig fields could be exposed somewhere in the alertmanagerConfig or prometheusConfig in MonitoringStack.

Slack discussions about the issue: https://redhat-internal.slack.com/archives/C03FR4B5MS7/p1708595283537899?thread_ts=1705907694.853099&cid=C03FR4B5MS7
https://redhat-internal.slack.com/archives/C03FR4B5MS7/p1708606919664429