Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-56148

TargetDown Alert and TOKEN_REVIEW: 'GET /metrics' unauthorized, invalid user token in console pod logs

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • None
    • None
    • In Progress
    • Bug Fix
    • Hide
      The "/metrics" endpoint was not correctly parsing a bearer token from the authorization header on internal prometheus scrape requests, which caused TokenReviews to fail and and all of these requests to be denied with a 401 response. This triggered a TargetDown alert for the console metrics endpoint. The metrics endpoint handler was updated to correctly parse a bearer token from the authorization header for TokenReview. This made the TokenReview step behave as expected, and resolved the TargetDown alert.
      Show
      The "/metrics" endpoint was not correctly parsing a bearer token from the authorization header on internal prometheus scrape requests, which caused TokenReviews to fail and and all of these requests to be denied with a 401 response. This triggered a TargetDown alert for the console metrics endpoint. The metrics endpoint handler was updated to correctly parse a bearer token from the authorization header for TokenReview. This made the TokenReview step behave as expected, and resolved the TargetDown alert.
    • None
    • None
    • None
    • None

      Description of problem:

         After installing 4.18.12 Customers see a TargetDown alert with the message 100% of the console/console targets in openshift-console namespace have been unreachable for more than 15 minutes.

In the console pod we are the seeing the following error: 

E0513 03:56:37.281535       1 middleware.go:51] TOKEN_REVIEW: 'GET /metrics' unauthorized, invalid user token, [invalid bearer token, token lookup failed]

      Version-Release number of selected component (if applicable):

      4.19.0
4.18.12 / 4.18.13 / 4.18.14 / 4.18.15 / 4.18.16 / 4.18.17
4.17.29 / 4.17.30 / 4.17.31 / 4.17.32 / 4.17.33 / 4.17.34

      How reproducible:

          Everytime.

      Steps to Reproduce:

          1. Install or upgrade to any of the listed versions above.
    2. Review active Alerts and Console Pod Logs

       

      Potentially related to:

      https://issues.redhat.com/browse/OCPBUGS-45369

      https://github.com/openshift/console/pull/14664

       

              rh-ee-jonjacks Jon Jackson
              rhn-support-mpeterma Morgan Peterman
              None
              None
              Sanket Pathak Sanket Pathak
              None
              Votes:
              32 Vote for this issue
              Watchers:
              71 Start watching this issue

                Created:
                Updated: