Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-60078

ccoctl create-all incorrectly overwrites openid-configuration with S3 URL when CloudFront is in use

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

       

      Description of problem:

      When using ccoctl aws create-all to manage an OpenShift cluster's OIDC provider, re-running the command incorrectly overwrites the openid-configuration file in the OIDC S3 bucket. This replaces the intended CloudFront URL with the private S3 endpoint URL for the issuer and jwks_uri fields, leading to InvalidIdentityToken errors and widespread authentication failures across the cluster.

      ***

      Version-Release number of selected component (if applicable):

      ccoctl version corresponding to OpenShift Container Platform 4.14+

      ***

      How reproducible:

      Consistently reproducible.

      ***

      Steps to Reproduce:

      1.  Set up an OpenShift cluster on AWS using manual STS credentials with ccoctl aws create-all. Ensure the OIDC S3 bucket is fronted by a CloudFront distribution.
      2.  Manually edit the .well-known/openid-configuration file in the S3 bucket to point the issuer and jwks_uri to the CloudFront URL.
      3.  Re-run the same ccoctl aws create-all command from the same directory where the initial ccoctl run was performed.
      4.  Inspect the .well-known/openid-configuration file in the S3 bucket again (e.g., by accessing it via the CloudFront URL).

      ***

      Actual results:

      The issuer and jwks_uri fields in the openid-configuration file are reverted from the CloudFront URL to the private S3 bucket URL.

      Example of incorrect openid-configuration JSON:

      {   "issuer": "https://<EXAMPLE_S3_BUCKET_NAME>.s3.<EXAMPLE_REGION>.amazonaws.com",   "jwks_uri": "https://<EXAMPLE_S3_BUCKET_NAME>.s3.<EXAMPLE_REGION>.amazonaws.com/keys.json",   "id_token_signing_alg_values_supported": [     "RS256"   ] }

      ***

      Expected results:

      The issuer and jwks_uri fields in the openid-configuration file should remain unchanged and continue to point to the CloudFront URL.

      Example of correct openid-configuration JSON:

      {   "issuer": "https://<EXAMPLE_CLOUDFRONT_DISTRIBUTION_ID>.cloudfront.net",   "jwks_uri": "https://<EXAMPLE_CLOUDFRONT_DISTRIBUTION_ID>.cloudfront.net/.well-known/keys.json",   "id_token_signing_alg_values_supported": [     "RS256"   ] }

      ***

      Additional info:

      This behavior is a significant problem for clusters using CloudFront to serve OIDC metadata, as it breaks the OIDC authentication chain. This issue is specific to the ccoctl aws create-all command. The recommended upgrade path using ccoctl aws create-iam-roles does not exhibit this behavior.

      A discussion about this issue is available on a public forum.

      Additional info:

       

      This behavior is a significant problem for clusters using CloudFront to serve OIDC metadata, as it breaks the OIDC authentication chain. This issue is specific to the ccoctl aws create-all command. The recommended upgrade path using ccoctl aws create-iam-roles does not exhibit this behavior.

      slack dicussion thread here: https://redhat-internal.slack.com/archives/C04TMSTHUHK/p1753884542376789

      Show
        Description of problem: When using ccoctl aws create-all to manage an OpenShift cluster's OIDC provider, re-running the command incorrectly overwrites the openid-configuration file in the OIDC S3 bucket. This replaces the intended CloudFront URL with the private S3 endpoint URL for the issuer and jwks_uri fields, leading to InvalidIdentityToken errors and widespread authentication failures across the cluster. *** Version-Release number of selected component (if applicable): ccoctl version corresponding to OpenShift Container Platform 4.14+ *** How reproducible: Consistently reproducible. *** Steps to Reproduce: 1.  Set up an OpenShift cluster on AWS using manual STS credentials with ccoctl aws create-all. Ensure the OIDC S3 bucket is fronted by a CloudFront distribution. 2.  Manually edit the .well-known/openid-configuration file in the S3 bucket to point the issuer and jwks_uri to the CloudFront URL. 3.  Re-run the same ccoctl aws create-all command from the same directory where the initial ccoctl run was performed. 4.  Inspect the .well-known/openid-configuration file in the S3 bucket again (e.g., by accessing it via the CloudFront URL). *** Actual results: The issuer and jwks_uri fields in the openid-configuration file are reverted from the CloudFront URL to the private S3 bucket URL. Example of incorrect openid-configuration JSON: {   "issuer": "https://<EXAMPLE_S3_BUCKET_NAME>.s3.<EXAMPLE_REGION>.amazonaws.com",   "jwks_uri": "https://<EXAMPLE_S3_BUCKET_NAME>.s3.<EXAMPLE_REGION>.amazonaws.com/keys.json",   "id_token_signing_alg_values_supported": [     "RS256"   ] } *** Expected results: The issuer and jwks_uri fields in the openid-configuration file should remain unchanged and continue to point to the CloudFront URL. Example of correct openid-configuration JSON: {   "issuer": "https://<EXAMPLE_CLOUDFRONT_DISTRIBUTION_ID>.cloudfront.net",   "jwks_uri": "https://<EXAMPLE_CLOUDFRONT_DISTRIBUTION_ID>.cloudfront.net/.well-known/keys.json",   "id_token_signing_alg_values_supported": [     "RS256"   ] } *** Additional info: This behavior is a significant problem for clusters using CloudFront to serve OIDC metadata, as it breaks the OIDC authentication chain. This issue is specific to the ccoctl aws create-all command. The recommended upgrade path using ccoctl aws create-iam-roles does not exhibit this behavior. A discussion about this issue is available on a public forum. Additional info:   This behavior is a significant problem for clusters using CloudFront to serve OIDC metadata, as it breaks the OIDC authentication chain. This issue is specific to the ccoctl aws create-all command. The recommended upgrade path using ccoctl aws create-iam-roles does not exhibit this behavior. slack dicussion thread here: https://redhat-internal.slack.com/archives/C04TMSTHUHK/p1753884542376789
    • None
    • Important
    • None
    • Production
    • In Progress
    • Release Note Not Required
    • Hide
      Before this update, executing ccoctl multiple times while using the AWS platform and the --create-private-s3-bucket parameter would cause the wrong URL to be configured for the OIDC issuer. As a consequence, some cluster operators would no longer be able to authenticate to the AWS API. With this release, ccoctl properly configures the correct URL for the OIDC issuer. As a result, the cluster operators continue to authenticate as expected.
      Show
      Before this update, executing ccoctl multiple times while using the AWS platform and the --create-private-s3-bucket parameter would cause the wrong URL to be configured for the OIDC issuer. As a consequence, some cluster operators would no longer be able to authenticate to the AWS API. With this release, ccoctl properly configures the correct URL for the OIDC issuer. As a result, the cluster operators continue to authenticate as expected.

      sds

              jstuever@redhat.com Jeremiah Stuever
              rhn-support-vmedina1 Victor Medina
              Victor Medina
              None
              Jianping Shu Jianping Shu
              None
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: