Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-72543

kube_horizontalpodautoscaler_spec_target_metric is missing a containername label in support of container resource based autoscaling

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.18
    • Monitoring
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      The HPA now contains feature that allow for scaling on `containerResource`

      apiVersion: autoscaling/v2
      kind: HorizontalPodAutoscaler
      metadata:
        labels:
        name: hpa-hello
        namespace: test-metrics
      spec:
        minReplicas: 1
        maxReplicas: 2
        metrics:
        - containerResource:
            container: hello-1
            name: cpu
            target:
              averageUtilization: 90
              type: Utilization
          type: ContainerResource
        - containerResource:
            container: hello-2
            name: cpu
            target:
              averageUtilization: 90
              type: Utilization
          type: ContainerResource
        scaleTargetRef:
          apiVersion: apps/v1
          kind: Deployment
          name: hello
      
          

      When the above configuration is used, PrometheusDuplicateTimestamps alerts are generated.

       
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: hello
        namespace: test-metrics
        labels:
          app: hello
          version: v1
      spec:
        strategy:
          type: Recreate
        replicas: 1
        selector:
          matchLabels:
            app: hello
        template:
          metadata:
            labels:
              app: hello
              version: v1
          spec:
            automountServiceAccountToken: false
            containers:
            - name: hello-1
              image: registry.access.redhat.com/ubi9/ubi-minimal:latest
              command: ["/bin/bash", "-c", "echo 'Hello from container 1'; sleep infinity"]
              resources:
                requests:
                  cpu: 50m
                  memory: 128Mi
              securityContext:
                readOnlyRootFilesystem: true
                allowPrivilegeEscalation: false
            - name: hello-2
              image: registry.access.redhat.com/ubi9/ubi-minimal:latest
              command: ["/bin/bash", "-c", "echo 'Hello from container 2'; sleep infinity"]
              resources:
                requests:
                  cpu: 50m
                  memory: 128Mi
              securityContext:
                readOnlyRootFilesystem: true
                allowPrivilegeEscalation: false
      

      currently the metrics used by the HPA - `kube_horizontalpodautoscaler_spec_target_metric` does not have a `containername` label

       
      oc exec -n openshift-monitoring prometheus-k8s-0 -- curl -s -k --cert /etc/prometheus/secrets/metrics-client-certs/tls.crt --key /etc/prometheus/secrets/metrics-client-certs/tls.key https://kube-state-metrics.openshift-monitoring.svc:8443/metrics | uniq -D
      
      kube_horizontalpodautoscaler_spec_target_metric{namespace="test-metrics",horizontalpodautoscaler="hpa-hello",metric_name="cpu",metric_target_type="utilization"} 90
      kube_horizontalpodautoscaler_spec_target_metric{namespace="test-metrics",horizontalpodautoscaler="hpa-hello",metric_name="cpu",metric_target_type="utilization"} 90
      

      The following upstream issue has been logged - https://github.com/kubernetes/kube-state-metrics/issues/2403

              jfajersk@redhat.com Jan Fajerski
              rhn-support-nigsmith Nigel Smith
              None
              None
              Junqi Zhao Junqi Zhao
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: