Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-8727

LokiStack gateway OIDC authentication fails when issuerCA contains multiple certificates

XMLWordPrintable

    • Incidents & Support
    • False
    • Hide

      None

      Show
      None
    • False
    • NEW
    • NEW
    • Bug Fix

      Description of problem:

        The LokiStack gateway (Observatorium API) fails OIDC token validation with x509: certificate signed by unknown authority when the issuerCA ConfigMap contains multiple CA certificates in a single PEM bundle.

        In static tenant mode with Kubernetes as the OIDC provider, the gateway makes two TLS calls during token validation:
        1. OIDC discovery to https://kubernetes.default.svc (signed by service-network-signer CA)
        2. JWKS fetch to https://api.<cluster>:6443 (signed by a different enterprise/external CA)

        Both CAs are correctly provided in the issuerCA ConfigMap as a concatenated PEM bundle. However, the gateway code uses pem.Decode() which reads only the first PEM block, silently discarding all subsequent certificates. The second TLS call fails because its CA was never loaded.

        The bug is in observatorium/api/authentication/oidc.go:

        block, _ := pem.Decode(config.IssuerRawCA)   // reads ONLY first PEM block
        cert, _ := x509.ParseCertificate(block.Bytes) // parses ONE cert
        t.TLSClientConfig.RootCAs.AddCert(config.issuerCA) // adds ONE cert to pool

       

       Version-Release number of selected component (if applicable):

        Loki Operator (all versions using Observatorium API gateway). Tested on OpenShift 4.x with LokiStack in static tenant mode. The bug exists in the current main branch of https://github.com/observatorium/api (authentication/oidc.go).

      How reproducible:

      1. %. Occurs on any cluster where kubernetes.default.svc and the external API URL use different CAs (common in enterprise environments with custom PKI)

      Steps to Reproduce:

      1. Deploy OpenShift cluster where:
          - kubernetes.default.svc is signed by kube-apiserver-service-network-signer (internal CA)
          - External API URL api.<cluster>:6443 is signed by a different CA (enterprise PKI)
      2. Deploy LokiStack with mode: static and OIDC authentication using issuerURL: https://kubernetes.default.svc
      3. Create a ConfigMap containing both CAs concatenated in one PEM file:
        ----BEGIN CERTIFICATE----
        <service-network-signer CA>
        ----END CERTIFICATE----
        ----BEGIN CERTIFICATE----
        <enterprise root CA>
        ----END CERTIFICATE----
      4. Reference this ConfigMap as issuerCA in the LokiStack CR authentication section
      5. Send a request with a valid ServiceAccount Bearer token to the Loki gateway

      Actual results:

        Gateway logs show x509: certificate signed by unknown authority. OIDC discovery succeeds (first CA loaded) but JWKS fetch fails (second CA silently discarded). All authenticated requests return 401/403.

      Expected results:

        All certificates in the PEM bundle should be loaded into the TLS certificate pool. The gateway should successfully validate tokens when the required CAs are provided in the issuerCA ConfigMap, regardless of how many certificates the PEM bundle contains.

      Upstream code reference: https://github.com/observatorium/api/blob/main/authentication/oidc.go

              Unassigned Unassigned
              ayesha.arshad.ccsp Ayesha Arshad
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: