Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4361

[release-5.7] Setting custom options on the application tenant removes user-alertmanager configuration

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this update, an issue in the Loki Operator caused the alert-manager configuration for the "application" tenant to disappear if the LokiStack was configured with additional options for that tenant. With this update, the generated Loki configuration now contains both the custom and the auto-generated configuration.
      Show
      Before this update, an issue in the Loki Operator caused the alert-manager configuration for the "application" tenant to disappear if the LokiStack was configured with additional options for that tenant. With this update, the generated Loki configuration now contains both the custom and the auto-generated configuration.
    • Bug Fix
    • Log Storage - Sprint 239

    Description

      Description of problem:

      When a LokiStack is configured in openshift-logging tenant mode with a ruler and there is an alertmanager for user workloads present in the cluster, the Loki Operator automatically configures the application tenant to use that alertmanager by adding configuration to the runtime-config.yaml in the ConfigMap used by Loki.

      Example LokiStack:

      apiVersion: loki.grafana.com/v1
      kind: LokiStack
      metadata:
        name: lokistack-dev
        namespace: openshift-logging
      spec:
        size: 1x.extra-small
        storage:
          schemas:
          - version: v12
            effectiveDate: 2022-06-01
          secret:
            name: test
            type: s3
        storageClassName: gp3-csi
        tenants:
          mode: openshift-logging
        rules:
          enabled: true
          selector:
            matchLabels:
              openshift.io/cluster-monitoring: "true"
          namespaceSelector:
            matchLabels:
              openshift.io/cluster-monitoring: "true"
      

      Generated configuration:

      ---
      overrides:
        application:
          ruler_alertmanager_config:
            alertmanager_url: https://_web._tcp.alertmanager-operated.openshift-user-workload-monitoring.svc
            enable_alertmanager_v2: true
            enable_alertmanager_discovery: true
            alertmanager_refresh_interval: 1m
            alertmanager_client:
              tls_ca_path: /var/run/ca/alertmanager/service-ca.crt
              tls_server_name: alertmanager-user-workload.openshift-user-workload-monitoring.svc.cluster.local
              type: Bearer
              credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      When the LokiStack is modified to contain overrides for the application tenant, for example a custom query-timeout, the alertmanager configuration vanishes from the configuration file and is replaced by the timeout:

      ---
      overrides:
        application:
          query_timeout: 5m 

      The configuration should actually contain both configuration options, the timeout and the alertmanager configuration, unless a custom alertmanager configuration is added by the user.

      Version-Release number of selected component (if applicable):

      How reproducible:

      Steps to Reproduce:

      1. Create LokiStack on OpenShift with user-workload monitoring enabled on the cluster and a custom queryTimeout for the application tenant
      2. Observe the contents of the generated ConfigMap

      Actual results:

      Generated ConfigMap contains only part of the configuration.

      Expected results:

      Generated ConfigMap contains both the autogenerated alertmanager configuration and the custom query-timeout.

      Additional info:

      LokiStack with custom queryTimeout:

      apiVersion: loki.grafana.com/v1
      kind: LokiStack
      metadata:
        name: lokistack-dev
        namespace: openshift-logging
      spec:
        limits:
          tenants:
            application:
              queries:
                queryTimeout: 5m
        managementState: Managed
        rules:
          enabled: true
          namespaceSelector:
            matchLabels:
              openshift.io/cluster-monitoring: "true"
          selector:
            matchLabels:
              openshift.io/cluster-monitoring: "true"
        size: 1x.extra-small
        storage:
          schemas:
          - effectiveDate: "2022-06-01"
            version: v12
          secret:
            name: test
            type: s3
        storageClassName: gp3-csi
        tenants:
          mode: openshift-logging 

      Attachments

        Issue Links

          Activity

            People

              rojacob@redhat.com Robert Jacob
              rojacob@redhat.com Robert Jacob
              Kabir Bharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: