Description of problem:

application-manager pod logs error "secret namespace not matched, appAddonNS= open-cluster-management-agent-addon" excessively

Version-Release number of selected component (if applicable):

2.11

How reproducible:

n/a

Steps to Reproduce:

3. ...

Actual results:

Excessive logging of secret changes

Expected results:

No logging of this error.

Additional info:

Looking into https://github.com/stolostron/multicloud-operators-subscription/blob/main/pkg/controller/spoketoken/spoke_token_controller.go

// detect if there is any change to the secret associated to the App Addon application-manager SA.
var applicationManagerSecretPredicateFunctions = predicate.Funcs{
UpdateFunc: func(e event.UpdateEvent) bool {
newSecret, ok := e.ObjectNew.(*corev1.Secret)
if !ok

if newSecret.Namespace != appAddonNS

This would mean that the application-manager's Service Account secret is created in another namespace than open-cluster-management-agent-addon and this is recognized by this watch request being called here

// watch for changes to the secrets associated to the App Addon application-manager SA
saSecretMapper := &applicationManagerSecretMapper

err = c.Watch(
source.Kind(mgr.GetCache(), &corev1.Secret{}),
handler.EnqueueRequestsFromMapFunc(saSecretMapper.Map),
applicationManagerSecretPredicateFunctions)

There is a pull request from customer in the upstream based on this assumpion

I think, the problem is that the logging is happening in the predicate function and not in the handler of this Watch() -> https://github.com/open-cluster-management-io/multicloud-operators-subscription/blob/dfcbf0c7d3347ace201c71211d75dbea36ec2beb/pkg/controller/spoketoken/spoke_token_controller.go#L123
Meaning that every update to a Secret triggers the update predicate function (https://github.com/open-cluster-management-io/multicloud-operators-subscription/blob/dfcbf0c7d3347ace201c71211d75dbea36ec2beb/pkg/controller/spoketoken/spoke_token_controller.go#L432) which filters the events (only when predicate returns true the handler is called) before passing them to the handler function.
That's why we see a lot of 'secret namespace not matched' but no 'app addon SA secret changed' (https://github.com/open-cluster-management-io/multicloud-operators-subscription/blob/dfcbf0c7d3347ace201c71211d75dbea36ec2beb/pkg/controller/spoketoken/spoke_token_controller.go#L101) from the handler map function.

In my opinion a patch would be to change the logging of the predicate functions (https://github.com/open-cluster-management-io/multicloud-operators-subscription/blob/dfcbf0c7d3347ace201c71211d75dbea36ec2beb/pkg/controller/spoketoken/spoke_token_controller.go#L431) to the style of the ServiceAccountPredicateFunctions which only logs on true and not on false.

https://github.com/open-cluster-management-io/multicloud-operators-subscription/pull/413