Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-57182

TargetDown Alert and TOKEN_REVIEW: 'GET /metrics' unauthorized, invalid user token in console pod logs

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Major Major
    • 4.17.z
    • 4.17.z, 4.18.z, 4.19.z, 4.20.0
    • Management Console
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • None
    • None
    • Customer Escalated
    • In Progress
    • Bug Fix
    • Hide
      Previously, the `/metrics` endpoint failed to correctly parse a bearer token from the authorization header on internal Prometheus scrape requests. This caused the `TokenReviews` to fail and a `TargetDown` alert was triggered for the console metrics endpoint. With this release, the `/metrics` endpoint correctly parses the bearer token from the authorization header, the `TokenReview` step works as intended, and the `TargetDown` alert no longer displays. (link:https://issues.redhat.com/browse/OCPBUGS-57182[OCPBUGS-57182])
      Show
      Previously, the `/metrics` endpoint failed to correctly parse a bearer token from the authorization header on internal Prometheus scrape requests. This caused the `TokenReviews` to fail and a `TargetDown` alert was triggered for the console metrics endpoint. With this release, the `/metrics` endpoint correctly parses the bearer token from the authorization header, the `TokenReview` step works as intended, and the `TargetDown` alert no longer displays. (link: https://issues.redhat.com/browse/OCPBUGS-57182 [ OCPBUGS-57182 ])
    • None
    • None
    • None
    • None

      This is a clone of issue OCPBUGS-57181. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-57180. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-56148. The following is the description of the original issue:
      —
      Description of problem:

         After installing 4.18.12 Customers see a TargetDown alert with the message 100% of the console/console targets in openshift-console namespace have been unreachable for more than 15 minutes.

In the console pod we are the seeing the following error: 

E0513 03:56:37.281535       1 middleware.go:51] TOKEN_REVIEW: 'GET /metrics' unauthorized, invalid user token, [invalid bearer token, token lookup failed]

      Version-Release number of selected component (if applicable):

      4.18.12 / 4.18.13
4.17.29 / 4.17.30

      How reproducible:

          Everytime.

      Steps to Reproduce:

          1. Install 4.18.12 or 4.17.29
    2. Review active Alerts and Console Pod Logs

       

      Potentially related to:

      https://issues.redhat.com/browse/OCPBUGS-45369

      https://github.com/openshift/console/pull/14664

       

              rh-ee-jonjacks Jon Jackson
              openshift-crt-jira-prow OpenShift Prow Bot
              None
              None
              YaDan Pei YaDan Pei
              None
              Votes:
              2 Vote for this issue
              Watchers:
              15 Start watching this issue

                Created:
                Updated:
                Resolved: