-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
4.16.z
-
None
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
MON Sprint 284
-
1
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
The kube-rbac-proxy-web container within the prometheus-k8s pods (namespace: openshift-monitoring) is flooded with authentication errors (approx. 30k events per day). Despite following the official Red Hat remediation steps, the issue persists.
Version-Release number of selected component (if applicable):
OCP v4.16.44
How reproducible:
Not able to reproduce, partially reproduce via attempt to access the prometheus UI.
Steps to Reproduce:
n/a
Actual results:
[must-gather.local.9209806967749005987]$ omc logs -n openshift-monitoring prometheus-k8s-1 -c kube-rbac-proxy-web ... 2026-02-05T10:01:16.121304125Z I0205 10:01:16.121232 1 log.go:245] http: TLS handshake error from 10.131.6.14:52540: write tcp 10.131.6.15:9091->10.131.6.14:52540: write: connection reset by peer 2026-02-05T10:01:16.151928376Z I0205 10:01:16.151862 1 log.go:245] http: TLS handshake error from 10.130.6.18:36284: write tcp 10.131.6.15:9091->10.130.6.18:36284: write: connection reset by peer 2026-02-05T10:01:16.180290955Z I0205 10:01:16.180236 1 log.go:245] http: TLS handshake error from 10.130.8.10:36246: write tcp 10.131.6.15:9091->10.130.8.10:36246: write: connection reset by peer 2026-02-05T10:01:16.549417956Z E0205 10:01:16.549357 1 auth.go:47] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed] 2026-02-05T10:01:16.549417956Z E0205 10:01:16.549373 1 auth.go:47] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed] 2026-02-05T10:01:17.550300106Z E0205 10:01:17.550235 1 auth.go:47] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
Expected results:
No logs about: 'invalid bearer token, square/go-jose' if any then regarding unauthorized access to the Prometheus UI with an incorrect token; for example via: $ curl -s -k -H "Authorization: Bearer $BAD_TOKEN" "https://prometheus-k8s-federate-openshift-monitoring.apps.example.com/"
Additional info:
These issues comes via openshift-ingress router pods, but when we try to switch the router pods into trace mode and check potential external access to the system, there is not any. Source of these errors are mostly from openshift-ingress router pods.
We have a similar BZ for this issue which is already fixed in:
https://bugzilla.redhat.com/show_bug.cgi?id=1956879
- duplicates
-
-
- Closed
-
