Description

The Cost Management Operator consistently fails to authenticate with Keycloak due to an invalid scope (api.console) configured in the cost-management-service-account KeycloakClient. The scope does not exist in the Keycloak kubernetes realm, causing all authentication attempts to fail with a 400 error.

This is occurring while running through jwt auth scripts here.

Error Message

INFO  metricsconfig_controller.validateCredentials  failed to obtain service-account token: status: 400, error: invalid_scope, description: Invalid scopes: api.console
ERROR Reconciler error  {"controller": "costmanagementmetricsconfig", "error": "status: 400, error: invalid_scope, description: Invalid scopes: api.console"}

Root Cause

The scripts/deploy-rhsso.sh script creates the Cost Management Keycloak client with api.console in defaultClientScopes:

File: scripts/deploy-rhsso.sh (lines 344-348, 374)

spec:   client:     defaultClientScopes:     * openid
    * profile
    * email
    * api.console    # ← This scope does not exist in Keycloak!

Additionally, a protocol mapper attempts to "mock" this scope:

protocolMappers: * config:     claim.name: scope
    claim.value: api.console
  name: api-console-mock
  protocolMapper: oidc-hardcoded-claim-mapper

Why this fails:
1. Keycloak validates defaultClientScopes BEFORE issuing tokens
2. The api.console scope doesn't exist as a ClientScope in the realm
3. Keycloak returns 400 error during token request
4. The protocol mapper never executes (it only runs AFTER token generation)

Impact

• Cost Management Operator cannot obtain JWT tokens from Keycloak
• Authentication completely blocked for service account flow
• No metrics can be uploaded to ROS
• Operator continuously errors in reconciliation loop

Why This Scope Was Added

The api.console scope is used by Red Hat's console.redhat.com for upstream SaaS.

However:

• Local ROS deployment does NOT validate this scope
• Envoy JWT authentication does NOT check scopes
• ROS ingress backend does NOT validate scopes
• The scope serves NO purpose in the on-prem deployment

Attempted Workaround

We attempted to fix this by patching the KeycloakClient CR to remove api.console:

oc get keycloakclient cost-management-service-account -n rhsso -o json | \
  jq '.spec.client.defaultClientScopes = ["openid","profile","email"]' | \
  oc apply -f -

Result: The KeycloakClient CR was updated successfully, but the Cost Management Operator continues to receive the same 400 error. This suggests Keycloak's internal database has not synchronized with the CR change, or the RHSSO operator is not properly propagating the update.

Proper Fix Required (maybe)

The scripts/deploy-rhsso.sh script must be updated to NOT include api.console in defaultClientScopes.

Edit lines 344-348:

defaultClientScopes: * openid
* profile
* email
h1. Remove: - api.console

The hardcoded claim mapper can remain (it's harmless and will add the claim to successfully-issued tokens).

Steps to Reproduce

1. 1. Deploy RHSSO with Cost Management client:

         ./scripts/deploy-rhsso.sh
         

2. Deploy Cost Management Operator:

   ./scripts/setup-cost-mgmt-tls.sh
   

3. Check operator logs:

   oc logs -n costmanagement-metrics-operator deployment/costmanagement-metrics-operator | grep -i "invalid_scope"
   

4. Observe continuous 400 authentication errors

Verification After Fix

After removing api.console from the script:

1. 1. Redeploy Keycloak client
      2. Check KeycloakClient scopes:

         oc get keycloakclient cost-management-service-account -n rhsso -o jsonpath='{.spec.client.defaultClientScopes}'
         # Should return: ["openid","profile","email"]
         

3. Recreate CostManagementMetricsConfig to force re-authentication
4. Verify no authentication errors in operator logs

Additional Notes

• This issue was introduced in commit c697bd0 on Oct 13, 2025
• The scope appears to have been added speculatively for console.redhat.com compatibility
• The local ROS deployment does not require or validate this scope
• Other Keycloak clients (openshift-oidc-client) use only standard scopes and work correctly

Environment

• OpenShift Version: 4.18
• Cost Management Operator Version: 4.1.0
• RHSSO Operator: stable channel
• Keycloak Realm: kubernetes
• Script: scripts/deploy-rhsso.sh
• Namespace: rhsso
• Keycloak URL: https://keycloak-rhsso.apps.insights.qe.lab.redhat.com
• Self-signed certificates in use