Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-15274

CMA prometheus trigger fails with error "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • 4.13.0
    • Pod Autoscaler
    • Important
    • No
    • 2
    • OCPNODE Sprint 238 (Green)
    • 1
    • Proposed
    • False
    • Hide

      None

      Show
      None
    • Hide
      Cause:
      A new feature added to Custom Metrics Autoscaler 2.10.1 created a regression in prometheus triggers using authentication other than PodIdentity, and explicitly specifying a PodIdentity provider "none".
       
      Consequence:
      Any prometheus trigger using an authentication type other than PodIdentity and specifying a PodIdentity provider "none" would never become ready and would fail to scale.
      Fix:
      Custom Metrics Autoscaler for OpenShift now properly handles the "none" PodIdentity provider type.
      Result:
      A prometheus trigger which specifies a PodIdentity provider type of "none" together with another authentication type can now properly scale.
      Show
      Cause: A new feature added to Custom Metrics Autoscaler 2.10.1 created a regression in prometheus triggers using authentication other than PodIdentity, and explicitly specifying a PodIdentity provider "none".   Consequence: Any prometheus trigger using an authentication type other than PodIdentity and specifying a PodIdentity provider "none" would never become ready and would fail to scale. Fix: Custom Metrics Autoscaler for OpenShift now properly handles the "none" PodIdentity provider type. Result: A prometheus trigger which specifies a PodIdentity provider type of "none" together with another authentication type can now properly scale.
    • Bug Fix
    • Proposed

      From https://github.com/kedacore/keda/issues/4695

      Description of problem:

      After creating a ScaledObject with a prometheus trigger and a ClusterTriggerAuthentication, an error appears in the logs of the keda-operator container saying "error parsing prometheus metadata: pod identity cannot be enabled with other auth types", even though no pod identity is specified.

      Version-Release number of selected component (if applicable):

      CMA 2.10.1-253 on OpenShift 1.25

      How reproducible:

      100%

      Steps to Reproduce:

      1. Create a scaled object:
      apiVersion: keda.sh/v1alpha1
      kind: ScaledObject
      metadata:
        name: pilot-autoscaler
        namespace: camel-test
        labels:
          camel.apache.org/integration: pilot
          scaledobject.keda.sh/name: pilot-autoscaler
      spec:
        maxReplicaCount: 5
        minReplicaCount: 1
        scaleTargetRef:
          apiVersion: camel.apache.org/v1
          kind: Integration
          name: pilot
        triggers:
          - authenticationRef:
              kind: ClusterTriggerAuthentication
              name: keda-trigger-auth-prometheus
            metadata:
              authModes: bearer
              comparisonOperator: GreaterThan
              metricName: application_camel_context_exchanges_inflight_count
              namespace: camel-test
              query: 'application_camel_context_exchanges_inflight_count{job="pilot"}'
              serverAddress: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9092'
              threshold: '4'
              unsafeSsl: 'false'
            type: prometheus
      2. Create a ClusterTriggerAuthentication
      apiVersion: keda.sh/v1alpha1
      kind: ClusterTriggerAuthentication
      metadata:
        name: keda-trigger-auth-prometheus
      spec:
        podIdentity:
          provider: none
        secretTargetRef:
          - key: token
            name: thanos-keda-token
            parameter: bearerToken
          - key: ca.crt
            name: thanos-keda-token
            parameter: ca
      3. Create a scalable workload, such as an "Integration" from the Apache Camel K operator (as in the above example)
       

      Actual results:

      No HPA object created, ScaledObject status contains
          - message: Failed to ensure HPA is correctly created for ScaledObject
            reason: ScaledObjectCheckFailed
            status: 'False'
            type: Ready

      Expected results:

      HPA object created, scaling proceeds

      Additional info:

      Logs from the keda-operator pod:
      2023-06-16T08:15:07Z	INFO	Creating a new HPA	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "HPA.Namespace": "camel-test", "HPA.Name": "keda-hpa-pilot-autoscaler"}
      2023-06-16T08:15:07Z	ERROR	scale_handler	error resolving auth params	{"type": "ScaledObject", "namespace": "camel-test", "name": "pilot-autoscaler", "scalerIndex": 0, "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
      github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
      	/remote-source/keda/app/pkg/scaling/scalers_builder.go:81
      github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
      	/remote-source/keda/app/pkg/scaling/scale_handler.go:352
      github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
      	/remote-source/keda/app/pkg/scaling/scale_handler.go:273
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
      	/remote-source/keda/app/controllers/keda/hpa.go:200
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
      	/remote-source/keda/app/controllers/keda/hpa.go:74
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
      	/remote-source/keda/app/controllers/keda/hpa.go:47
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
      	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:408
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
      	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:229
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
      	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:175
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
      2023-06-16T08:15:07Z	ERROR	Error getting scalers	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
      	/remote-source/keda/app/controllers/keda/hpa.go:202
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
      	/remote-source/keda/app/controllers/keda/hpa.go:74
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
      	/remote-source/keda/app/controllers/keda/hpa.go:47
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
      	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:408
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
      	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:229
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
      	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:175
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
      2023-06-16T08:15:07Z	ERROR	Failed to create new HPA resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "HPA.Namespace": "camel-test", "HPA.Name": "keda-hpa-pilot-autoscaler", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
      	/remote-source/keda/app/controllers/keda/hpa.go:49
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
      	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:408
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
      	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:229
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
      	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:175
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
      2023-06-16T08:15:07Z	ERROR	Failed to ensure HPA is correctly created for ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
      github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
      	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:178
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
      2023-06-16T08:15:07Z	ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
      	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
      

       

              joelsmith.redhat Joel Smith
              joelsmith.redhat Joel Smith
              Weinan Liu Weinan Liu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: