Description of problem:

The metrics collector backoff retries on permanent errors leading to unnecessary requests and error logs amplification.
See code: https://github.com/stolostron/multicluster-observability-operator/blob/9de48e182d32a33982cc2538c59b293e0617eed7/collectors/metrics/pkg/metricsclient/metricsclient.go#L553

We should not retry permanent error codes like 4** (except 429).

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.  
2.  
3. ...

Actual results:

Expected results:

Additional info: