Uploaded image for project: 'OpenShift Node'
  1. OpenShift Node
  2. OCPNODE-1594

Configure Prometheus to monitor CMA

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • None
    • OCPNODE Sprint 234 (Green), OCPNODE Sprint 235 (Green), OCPNODE Sprint 236 (Green)

      As an OpenShift cluster admin, I would like Prometheus to gather metrics about Custom Metrics Autoscaler if it is installed. Please enhance the CMA operator so that it creates and manages PodMonitor objects so that Prometheus will gather metrics for all three CMA controllers (custom-metrics-autoscaler-operator, keda-operator, keda-metrics-apiserver).

      1. These PodMonitor objects should be added to https://github.com/openshift/custom-metrics-autoscaler-operator/blob/main/resources/keda.yaml

      ---
      apiVersion: monitoring.coreos.com/v1
      kind: PodMonitor
      metadata:
        name: custom-metrics-autoscaler-operator
        namespace: openshift-keda
        labels:
          name: custom-metrics-autoscaler-operator
      spec:
        selector:
          matchLabels:
            name: custom-metrics-autoscaler-operator
        podMetricsEndpoints:
        - targetPort: 8080
          path: /metrics
          interval: 60s
      ---
      apiVersion: monitoring.coreos.com/v1
      kind: PodMonitor
      metadata:
        name: keda-metrics-apiserver
        namespace: openshift-keda
        labels:
          app: keda-metrics-apiserver
      spec:
        selector:
          matchLabels:
            app: keda-metrics-apiserver
        podMetricsEndpoints:
        - targetPort: 8080
          path: /metrics
          interval: 60s
      ---
      apiVersion: monitoring.coreos.com/v1
      kind: PodMonitor
      metadata:
        name: keda-operator
        namespace: openshift-keda
        labels:
          name: keda-operator
      spec:
        selector:
          matchLabels:
            name: keda-operator
        podMetricsEndpoints:
        - targetPort: 8080
          path: /metrics
          interval: 60s

       

      2. The operator controller should be modified so that it ensures the above objects exist:

      https://github.com/openshift/custom-metrics-autoscaler-operator/blob/main/controllers/keda/kedacontroller_controller.go#L87

      3. The operator should be given permission to create/edit PodMonitor objects in its own namespace and the CMA operator should have a containerPort so that Prometheus will monitor it:

      diff --git a/hack/cma-generate-csv.sh b/hack/cma-generate-csv.sh
      index d1cf99a..9a03ade 100755
      --- a/hack/cma-generate-csv.sh
      +++ b/hack/cma-generate-csv.sh
      @@ -40,6 +40,17 @@ spec:
         icon:
         - base64data: PHN2ZyBpZD0iTGF5ZXJfMSIgZGF0YS1uYW1lPSJMYXllciAxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAxOTIgMTQ1Ij48ZGVmcz48c3R5bGU+LmNscy0xe2ZpbGw6I2UwMDt9PC9zdHlsZT48L2RlZnM+PHRpdGxlPlJlZEhhdC1Mb2dvLUhhdC1Db2xvcjwvdGl0bGU+PHBhdGggZD0iTTE1Ny43Nyw2Mi42MWExNCwxNCwwLDAsMSwuMzEsMy40MmMwLDE0Ljg4LTE4LjEsMTcuNDYtMzAuNjEsMTcuNDZDNzguODMsODMuNDksNDIuNTMsNTMuMjYsNDIuNTMsNDRhNi40Myw2LjQzLDAsMCwxLC4yMi0xLjk0bC0zLjY2LDkuMDZhMTguNDUsMTguNDUsMCwwLDAtMS41MSw3LjMzYzAsMTguMTEsNDEsNDUuNDgsODcuNzQsNDUuNDgsMjAuNjksMCwzNi40My03Ljc2LDM2LjQzLTIxLjc3LDAtMS4wOCwwLTEuOTQtMS43My0xMC4xM1oiLz48cGF0aCBjbGFzcz0iY2xzLTEiIGQ9Ik0xMjcuNDcsODMuNDljMTIuNTEsMCwzMC42MS0yLjU4LDMwLjYxLTE3LjQ2YTE0LDE0LDAsMCwwLS4zMS0zLjQybC03LjQ1LTMyLjM2Yy0xLjcyLTcuMTItMy4yMy0xMC4zNS0xNS43My0xNi42QzEyNC44OSw4LjY5LDEwMy43Ni41LDk3LjUxLjUsOTEuNjkuNSw5MCw4LDgzLjA2LDhjLTYuNjgsMC0xMS42NC01LjYtMTcuODktNS42LTYsMC05LjkxLDQuMDktMTIuOTMsMTIuNSwwLDAtOC40MSwyMy43Mi05LjQ5LDI3LjE2QTYuNDMsNi40MywwLDAsMCw0Mi41Myw0NGMwLDkuMjIsMzYuMywzOS40NSw4NC45NCwzOS40NU0xNjAsNzIuMDdjMS43Myw4LjE5LDEuNzMsOS4wNSwxLjczLDEwLjEzLDAsMTQtMTUuNzQsMjEuNzctMzYuNDMsMjEuNzdDNzguNTQsMTA0LDM3LjU4LDc2LjYsMzcuNTgsNTguNDlhMTguNDUsMTguNDUsMCwwLDEsMS41MS03LjMzQzIyLjI3LDUyLC41LDU1LC41LDc0LjIyYzAsMzEuNDgsNzQuNTksNzAuMjgsMTMzLjY1LDcwLjI4LDQ1LjI4LDAsNTYuNy0yMC40OCw1Ni43LTM2LjY1LDAtMTIuNzItMTEtMjcuMTYtMzAuODMtMzUuNzgiLz48L3N2Zz4=
           mediatype: image/svg+xml
      +  install:
      +    spec:
      +      permissions:
      +      - rules:
      +        - apiGroups:
      +          - monitoring.coreos.com
      +          resources:
      +          - podmonitors
      +          verbs:
      +          - '*'
      +        serviceAccountName: custom-metrics-autoscaler-operator
         links:
         - name: Custom Metrics Autoscaler Documentation
           url: https://docs.openshift.com/container-platform/latest/nodes/pods/nodes-pods-autoscaling-custom.html
      @@ -65,6 +76,8 @@ jq_filter="$jq_filter"'.metadata.annotations."alm-examples" |= sub("\"namespace
       jq_filter="$jq_filter"'.spec.install.spec.deployments[0].spec.template.spec.containers[0].command |= [ "/usr/bin/bash" ] |'
       # export the env vars and then exec /manager
       jq_filter="$jq_filter"'.spec.install.spec.deployments[0].spec.template.spec.containers[0].args |= ["-c", "export KEDA_OPERATOR_IMAGE=$RELATED_IMAGE_1; export KEDA_METRICS_SERVER_IMAGE=$RELATED_IMAGE_2; exec /manager \"$0\" \"$@\"" ] + .  |'
      +# add a port to the CMA operator so its metrics can be monitored
      jYsNDIuNTMsNDRhNi40Myw2LjQzLDAsMCwxLC4yMi0xLjk0bC0zLjY2LDkuMDZhMTguNDUsMTguNDUsMCwwLDAtMS41MSw3LjMzYzAsMTguMTEsNDEsNDUuNDgsODcuNzQsNDUuNDgsMjAuNjksMCwzNi40My03Ljc2LDM2LjQzLTIxLjc3LDAtMS4wOCwwLTEuOTQtMS43My0xMC4xM1oiLz48cGF0aCBjbGFzcz0iY2xzLTEiIGQ9Ik0xMjcuNDcsODMuNDljMTIuNTEsMCwzMC42MS0yLjU4LDMwLjYxLTE3LjQ2YTE0LDE0LDAsMCwwLS4zMS0zLjQybC03LjQ1LTMyLjM2Yy0xLjcyLTcuMTItMy4yMy0xMC4zNS0xNS43My0xNi42QzEyNC44OSw4LjY5LDEwMy43Ni41LDk3LjUxLjUsOTEuNjkuNSw5MCw4LDgzLjA2LDhjLTYuNjgsMC0xMS42NC01LjYtMTcuODktNS42LTYsMC05LjkxLDQuMDktMTIuOTMsMTIuNSwwLDAtOC40MSwyMy43Mi05LjQ5LDI3LjE2QTYuNDMsNi40MywwLDAsMCw0Mi41Myw0NGMwLDkuMjIsMzYuMywzOS40NSw4NC45NCwzOS40NU0xNjAsNzIuMDdjMS43Myw4LjE5LDEuNzMsOS4wNSwxLjczLDEwLjEzLDAsMTQtMTUuNzQsMjEuNzctMzYuNDMsMjEuNzdDNzguNTQsMTA0LDM3LjU4LDc2LjYsMzcuNTgsNTguNDlhMTguNDUsMTguNDUsMCwwLDEsMS41MS03LjMzQzIyLjI3LDUyLC41LDU1LC41LDc0LjIyYzAsMzEuNDgsNzQuNTksNzAuMjgsMTMzLjY1LDcwLjI4LDQ1LjI4LDAsNTYuNy0yMC40OCw1Ni43LTM2LjY1LDAtMTIuNzItMTEtMjcuMTYtMzAuODMtMzUuNzgiLz48L3N2Zz4=
           mediatype: image/svg+xml
      +  install:
      +    spec:
      +      permissions:
      +      - rules:
      +        - apiGroups:
      +          - monitoring.coreos.com
      +          resources:
      +          - podmonitors
      +          verbs:
      +          - '*'
      +        serviceAccountName: custom-metrics-autoscaler-operator
         links:
         - name: Custom Metrics Autoscaler Documentation
           url: https://docs.openshift.com/container-platform/latest/nodes/pods/nodes-pods-autoscaling-custom.html
      @@ -65,6 +76,8 @@ jq_filter="$jq_filter"'.metadata.annotations."alm-examples" |= sub("\"namespace
       jq_filter="$jq_filter"'.spec.install.spec.deployments[0].spec.template.spec.containers[0].command |= [ "/usr/bin/bash" ] |'
       # export the env vars and then exec /manager
       jq_filter="$jq_filter"'.spec.install.spec.deployments[0].spec.template.spec.containers[0].args |= ["-c", "export KEDA_OPERATOR_IMAGE=$RELATED_IMAGE_1; export KEDA_METRICS_SERVER_IMAGE=$RELATED_IMAGE_2; exec /manager \"$0\" \"$@\"" ] + .  |'
      +# add a port to the CMA operator so its metrics can be monitored
      +jq_filter="$jq_filter"'.spec.install.spec.deployments[0].spec.template.spec.containers[0].ports |= [\{"containerPort":8080,"name":"http","protocol":"TCP"}] |'
       # create a spot to pass in the operand image specs as env vars
       jq_filter="$jq_filter"'.spec.install.spec.deployments[0].spec.template.spec.containers[0].env += [\{"name":"RELATED_IMAGE_1","value":"CMA_OPERAND_PLACEHOLDER_1"},\{"name":"RELATED_IMAGE_2","value":"CMA_OPERAND_PLACEHOLDER_2"}]'
       

       

              joelsmith.redhat Joel Smith
              joelsmith.redhat Joel Smith
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: