Starting with custom-metrics-autoscaler.v2.8.2-143, it was found that the error handling of KEDA is rather cumbersome when ScaleObjects are not correctly created, causing HTTP 500 response from kube-apiserver and triggering a lot of stacktraces which may not be necessary and can be handled more gracefully.

2023-03-30T11:11:42.943788791Z 2023-03-30T11:11:42Z	ERROR	scalehandler	Error getting scale decision	{"scaledobject.Name": "foo-bar-extract", "scaledObject.Namespace": "foo-bar", "scaleTarget.Name": "foo-bar-extract", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: unexpected EOF"}
2023-03-30T11:11:42.943788791Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2023-03-30T11:11:42.943788791Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:277
2023-03-30T11:11:42.943788791Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2023-03-30T11:11:42.943788791Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:149
2023-03-30T11:11:42.979947457Z 2023-03-30T11:11:42Z	ERROR	scalehandler	Error getting triggerAuth	{"type": "ScaledObject", "namespace": "foo-bar", "name": "foo-bar-extract", "triggerAuthRef.Name": "foo-bar-auth", "error": "TriggerAuthentication.keda.sh \"foo-bar-auth\" not found"}
2023-03-30T11:11:42.979947457Z github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRefAndPodIdentity
2023-03-30T11:11:42.979947457Z 	/remote-source/keda/app/pkg/scaling/resolver/scale_resolvers.go:136
2023-03-30T11:11:42.979947457Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1
2023-03-30T11:11:42.979947457Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:319
2023-03-30T11:11:42.979947457Z github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).refreshScaler
2023-03-30T11:11:42.979947457Z 	/remote-source/keda/app/pkg/scaling/cache/scalers_cache.go:205
2023-03-30T11:11:42.979947457Z github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
2023-03-30T11:11:42.979947457Z 	/remote-source/keda/app/pkg/scaling/cache/scalers_cache.go:92
2023-03-30T11:11:42.979947457Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2023-03-30T11:11:42.979947457Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:277
2023-03-30T11:11:42.979947457Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2023-03-30T11:11:42.979947457Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:149
2023-03-30T11:11:43.826604626Z 2023-03-30T11:11:43Z	ERROR	scalehandler	Error getting scale decision	{"scaledobject.Name": "foo-bar-extract", "scaledObject.Namespace": "foo-bar", "scaleTarget.Name": "foo-bar-extract", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: unexpected EOF"}
2023-03-30T11:11:43.826604626Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2023-03-30T11:11:43.826604626Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:277
2023-03-30T11:11:43.826604626Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2023-03-30T11:11:43.826604626Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:149
2023-03-30T11:11:43.847805622Z 2023-03-30T11:11:43Z	ERROR	scalehandler	Error getting triggerAuth	{"type": "ScaledObject", "namespace": "foo-bar", "name": "foo-bar-extract", "triggerAuthRef.Name": "foo-bar-auth", "error": "TriggerAuthentication.keda.sh \"foo-bar-auth\" not found"}
2023-03-30T11:11:43.847805622Z github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRefAndPodIdentity
2023-03-30T11:11:43.847805622Z 	/remote-source/keda/app/pkg/scaling/resolver/scale_resolvers.go:136
2023-03-30T11:11:43.847805622Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1
2023-03-30T11:11:43.847805622Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:319
2023-03-30T11:11:43.847805622Z github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).refreshScaler
2023-03-30T11:11:43.847805622Z 	/remote-source/keda/app/pkg/scaling/cache/scalers_cache.go:205
2023-03-30T11:11:43.847805622Z github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
2023-03-30T11:11:43.847805622Z 	/remote-source/keda/app/pkg/scaling/cache/scalers_cache.go:92
2023-03-30T11:11:43.847805622Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2023-03-30T11:11:43.847805622Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:277
2023-03-30T11:11:43.847805622Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2023-03-30T11:11:43.847805622Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:149
2023-03-30T11:11:44.703296881Z 2023-03-30T11:11:44Z	ERROR	scalehandler	Error getting scale decision	{"scaledobject.Name": "foo-bar-extract", "scaledObject.Namespace": "foo-bar", "scaleTarget.Name": "foo-bar-extract", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: unexpected EOF"}
2023-03-30T11:11:44.703296881Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2023-03-30T11:11:44.703296881Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:277
2023-03-30T11:11:44.703296881Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2023-03-30T11:11:44.703296881Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:149
2023-03-30T11:11:46.167483201Z 2023-03-30T11:11:46Z	ERROR	scalehandler	Error getting triggerAuth	{"type": "ScaledObject", "namespace": "foo-bar", "name": "foo-bar-blob", "triggerAuthRef.Name": "foo-bar-auth", "error": "TriggerAuthentication.keda.sh \"foo-bar-auth\" not found"}
2023-03-30T11:11:46.167483201Z github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRefAndPodIdentity
2023-03-30T11:11:46.167483201Z 	/remote-source/keda/app/pkg/scaling/resolver/scale_resolvers.go:136
2023-03-30T11:11:46.167483201Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1
2023-03-30T11:11:46.167483201Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:319
2023-03-30T11:11:46.167483201Z github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).refreshScaler
2023-03-30T11:11:46.167483201Z 	/remote-source/keda/app/pkg/scaling/cache/scalers_cache.go:205
2023-03-30T11:11:46.167483201Z github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
2023-03-30T11:11:46.167483201Z 	/remote-source/keda/app/pkg/scaling/cache/scalers_cache.go:92
2023-03-30T11:11:46.167483201Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2023-03-30T11:11:46.167483201Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:277
2023-03-30T11:11:46.167483201Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2023-03-30T11:11:46.167483201Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:149
2023-03-30T11:11:47.025322911Z 2023-03-30T11:11:47Z	ERROR	scalehandler	Error getting scale decision	{"scaledobject.Name": "foo-bar-blob", "scaledObject.Namespace": "foo-bar", "scaleTarget.Name": "foo-bar-blob", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: unexpected EOF"}
2023-03-30T11:11:47.025322911Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2023-03-30T11:11:47.025322911Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:277
2023-03-30T11:11:47.025322911Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2023-03-30T11:11:47.025322911Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:149
2023-03-30T11:11:47.097442661Z 2023-03-30T11:11:47Z	ERROR	scalehandler	Error getting triggerAuth	{"type": "ScaledObject", "namespace": "foo-bar", "name": "foo-bar-extract", "triggerAuthRef.Name": "foo-bar-auth", "error": "TriggerAuthentication.keda.sh \"foo-bar-auth\" not found"}
2023-03-30T11:11:47.097442661Z github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRefAndPodIdentity
2023-03-30T11:11:47.097442661Z 	/remote-source/keda/app/pkg/scaling/resolver/scale_resolvers.go:136
2023-03-30T11:11:47.097442661Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1
2023-03-30T11:11:47.097442661Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:319
2023-03-30T11:11:47.097442661Z github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).refreshScaler
2023-03-30T11:11:47.097442661Z 	/remote-source/keda/app/pkg/scaling/cache/scalers_cache.go:205
2023-03-30T11:11:47.097442661Z github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
2023-03-30T11:11:47.097442661Z 	/remote-source/keda/app/pkg/scaling/cache/scalers_cache.go:92
2023-03-30T11:11:47.097442661Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2023-03-30T11:11:47.097442661Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:277
2023-03-30T11:11:47.097442661Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2023-03-30T11:11:47.097442661Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:149
2023-03-30T11:11:47.939491666Z 2023-03-30T11:11:47Z	ERROR	scalehandler	Error getting scale decision	{"scaledobject.Name": "foo-bar-extract", "scaledObject.Namespace": "foo-bar", "scaleTarget.Name": "foo-bar-extract", "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: unexpected EOF"}
2023-03-30T11:11:47.939491666Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2023-03-30T11:11:47.939491666Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:277
2023-03-30T11:11:47.939491666Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2023-03-30T11:11:47.939491666Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:149
2023-03-30T11:11:47.983000664Z 2023-03-30T11:11:47Z	ERROR	scalehandler	Error getting triggerAuth	{"type": "ScaledObject", "namespace": "foo-bar", "name": "foo-bar-extract", "triggerAuthRef.Name": "foo-bar-auth", "error": "TriggerAuthentication.keda.sh \"foo-bar-auth\" not found"}
2023-03-30T11:11:47.983000664Z github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRefAndPodIdentity
2023-03-30T11:11:47.983000664Z 	/remote-source/keda/app/pkg/scaling/resolver/scale_resolvers.go:136
2023-03-30T11:11:47.983000664Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1
2023-03-30T11:11:47.983000664Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:319
2023-03-30T11:11:47.983000664Z github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).refreshScaler
2023-03-30T11:11:47.983000664Z 	/remote-source/keda/app/pkg/scaling/cache/scalers_cache.go:205
2023-03-30T11:11:47.983000664Z github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
2023-03-30T11:11:47.983000664Z 	/remote-source/keda/app/pkg/scaling/cache/scalers_cache.go:92
2023-03-30T11:11:47.983000664Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2023-03-30T11:11:47.983000664Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:277
2023-03-30T11:11:47.983000664Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2023-03-30T11:11:47.983000664Z 	/remote-source/keda/app/pkg/scaling/scale_handler.go:149

Its understood that the ScaleObjects were missing proper configuration. But it's expected that either the creation of problematic objects is rejected or at least handled more graceful as currently this is causing not required noise due to the stack-trace generated as well as the HTTP 500 triggered at the kube-apiserver level (within the horizontal-pod-autoscaler controller)

4.11.26

Always

1. Install OpenShift Container Platform 4 With Custom Metrics Autoscaler (CMA)
2. Create CMA Controller
3. Create problematic ScaleObject

Triggering unnecessary HTTP 500 response from horizontal-pod-autoscaler controller which in turn can cause KubeClientErrors alert to fire. Also keda-operator logs are flooded with stack-traces even though the problem is shown.

Generally those information would be expected in the ScaleObject so the creator of the given object is aware and does not need to be contacted by the CMA Operator/Maintainer.

Error handling should be more graceful and the stack-trace is unexpected. Also most of the problems are expected to be reported in the ScaleObject status so they can be seen and handled by the respective author of the object