Loading...

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: None
Affects Version/s: 4.13.0
Component/s: Pod Autoscaler
Labels:
- triaged

Activity Type:
Quality / Stability / Reliability
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Story Points:
2
Severity:
Important
Regression:
No

Target Backport Versions:
None
Target Version:

4.14.0
Release Blocker:
Proposed
Sprint:
OCPNODE Sprint 238 (Green)
sprint_count:
1

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Release Note Status:
Proposed
Release Note Type:
Bug Fix
Release Note Text:

Hide
Cause:
A new feature added to Custom Metrics Autoscaler 2.10.1 created a regression in prometheus triggers using authentication other than PodIdentity, and explicitly specifying a PodIdentity provider "none".

Consequence:
Any prometheus trigger using an authentication type other than PodIdentity and specifying a PodIdentity provider "none" would never become ready and would fail to scale.
Fix:
Custom Metrics Autoscaler for OpenShift now properly handles the "none" PodIdentity provider type.
Result:
A prometheus trigger which specifies a PodIdentity provider type of "none" together with another authentication type can now properly scale.

Show
Cause: A new feature added to Custom Metrics Autoscaler 2.10.1 created a regression in prometheus triggers using authentication other than PodIdentity, and explicitly specifying a PodIdentity provider "none". Consequence: Any prometheus trigger using an authentication type other than PodIdentity and specifying a PodIdentity provider "none" would never become ready and would fail to scale. Fix: Custom Metrics Autoscaler for OpenShift now properly handles the "none" PodIdentity provider type. Result: A prometheus trigger which specifies a PodIdentity provider type of "none" together with another authentication type can now properly scale.

Escape Reason:
None
Escape Impact:
None
Corrective Measures:
None
SDLC stage when should've been found:
None

From https://github.com/kedacore/keda/issues/4695

Description of problem:

After creating a ScaledObject with a prometheus trigger and a ClusterTriggerAuthentication, an error appears in the logs of the keda-operator container saying "error parsing prometheus metadata: pod identity cannot be enabled with other auth types", even though no pod identity is specified.

Version-Release number of selected component (if applicable):

CMA 2.10.1-253 on OpenShift 1.25

How reproducible:

100%

Steps to Reproduce:

1. Create a scaled object:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: pilot-autoscaler
  namespace: camel-test
  labels:
    camel.apache.org/integration: pilot
    scaledobject.keda.sh/name: pilot-autoscaler
spec:
  maxReplicaCount: 5
  minReplicaCount: 1
  scaleTargetRef:
    apiVersion: camel.apache.org/v1
    kind: Integration
    name: pilot
  triggers:
    - authenticationRef:
        kind: ClusterTriggerAuthentication
        name: keda-trigger-auth-prometheus
      metadata:
        authModes: bearer
        comparisonOperator: GreaterThan
        metricName: application_camel_context_exchanges_inflight_count
        namespace: camel-test
        query: 'application_camel_context_exchanges_inflight_count{job="pilot"}'
        serverAddress: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9092'
        threshold: '4'
        unsafeSsl: 'false'
      type: prometheus
2. Create a ClusterTriggerAuthentication
apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: keda-trigger-auth-prometheus
spec:
  podIdentity:
    provider: none
  secretTargetRef:
    - key: token
      name: thanos-keda-token
      parameter: bearerToken
    - key: ca.crt
      name: thanos-keda-token
      parameter: ca
3. Create a scalable workload, such as an "Integration" from the Apache Camel K operator (as in the above example)

Actual results:

No HPA object created, ScaledObject status contains
    - message: Failed to ensure HPA is correctly created for ScaledObject
      reason: ScaledObjectCheckFailed
      status: 'False'
      type: Ready

Expected results:

HPA object created, scaling proceeds

Additional info:

Logs from the keda-operator pod:
2023-06-16T08:15:07Z	INFO	Creating a new HPA	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "HPA.Namespace": "camel-test", "HPA.Name": "keda-hpa-pilot-autoscaler"}
2023-06-16T08:15:07Z	ERROR	scale_handler	error resolving auth params	{"type": "ScaledObject", "namespace": "camel-test", "name": "pilot-autoscaler", "scalerIndex": 0, "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/remote-source/keda/app/pkg/scaling/scalers_builder.go:81
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/remote-source/keda/app/pkg/scaling/scale_handler.go:352
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/remote-source/keda/app/pkg/scaling/scale_handler.go:273
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/remote-source/keda/app/controllers/keda/hpa.go:200
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/remote-source/keda/app/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
	/remote-source/keda/app/controllers/keda/hpa.go:47
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:408
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:229
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:175
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2023-06-16T08:15:07Z	ERROR	Error getting scalers	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/remote-source/keda/app/controllers/keda/hpa.go:202
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/remote-source/keda/app/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
	/remote-source/keda/app/controllers/keda/hpa.go:47
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:408
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:229
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:175
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2023-06-16T08:15:07Z	ERROR	Failed to create new HPA resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "HPA.Namespace": "camel-test", "HPA.Name": "keda-hpa-pilot-autoscaler", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
	/remote-source/keda/app/controllers/keda/hpa.go:49
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:408
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:229
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:175
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2023-06-16T08:15:07Z	ERROR	Failed to ensure HPA is correctly created for ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:178
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2023-06-16T08:15:07Z	ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235