Loading...

Type: Bug
Resolution: Done
Priority: Normal
Fix Version/s: None
Affects Version/s: 4.16.0
Component/s: HyperShift
Labels:
- OAuth
- hypershift
- triaged

Activity Type:
Quality / Stability / Reliability
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Story Points:
None
Severity:
Moderate
Regression:
None

Target Backport Versions:
None
Target Version:
None
Release Blocker:
None
Sprint:
None

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

PX Impact Score:

Release Note Status:
None
Release Note Type:
None
Release Note Text:
None

Escape Reason:
None
Escape Impact:
None
Corrective Measures:
None
SDLC stage when should've been found:
None

The Hosted Cluster installation completes successfully for version 4.15.9 with OAuth configuration. Also, the same cluster can be upgraded to the 4.16 version without any issues. However, if we create a new Hosted Cluster for the 4.16.11 version, the installation gets stuck at the OAuth part where control-plane-operator fails to generate oauth config even though the required secrets for OAuth are there already, same configuration works for the 4.15.9 version.

The issue is reproducible every time. I am sharing the steps below for reproducing the issue.

// Create the required configmap and secrets for HCP OAuth in Hosted Cluster namespace.

$ oc new-project clusters

$ oc create secret generic entraid-auth-client-secret-hcp1 --from-literal=clientSecret=abcd1234

$ cat cm.yaml
apiVersion: v1
data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    MIIH1jCCBr6gAwIBAgIQBVfgNq0MljBx+Ba1Wvj3xDANBgkqhkiG9w0BAQsFADBN
    MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMScwJQYDVQQDEx5E
    aWdpQ2VydCBTSEEyIFNlY3VyZSBTZXJ2ZXIgQ0EwHhcNMjQwODMxMDAwMDAwWhcN
    MjUwMjI4MjM1OTU5WjB/MQswCQYDVQQGEwJVUzETMBEGA1UECBMKV2FzaGluZ3Rv
    bjEQMA4GA1UEBxMHUmVkbW9uZDEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0
    aW9uMSkwJwYDVQQDEyBzdGFtcDIubG9naW4ubWljcm9zb2Z0b25saW5lLmNvbTCC
    ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL/3nnemNHTU4q/vofTx/8Wx
    xUKElp4pG+X/L76eH43FGkm3AaX/Sd8MyA8qq6ac4sK7h6O1pnfrMT7wTy0VIjnu
    p9M25cwvhRHDFjSO4CNVho0CO5m1uoi+wxdJ3N6jEHyq8Gq0LtNYj1h3BUzrOUnx
    2im32DIv22RPbNML4KN4xmQfdkAm3EZdHCbyEXmGfAFxXag16FCz0Lz6qBp9QVHt
    FgaY9pH7KVMKnv2+JtrJ4YA/uwPAIU69DMRWIFcoKijtWzQDNlDdufOLKSx3ipeh
    o5FQ8RwErir1Q4825b2hg3ZcvZ1xqiy+d9thwF8xO5IjcVdhvtxjC7S4IGR1euEC
    AwEAAaOCBH4wggR6MB8GA1UdIwQYMBaAFA+AYRyCMWHVLyjnjUY4tCzhxtniMB0G
    A1UdDgQWBBSD+HXo3eMKSUoiDLuccKPSAp5mdzCCASYGA1UdEQSCAR0wggEZgiBz
    dGFtcDIubG9naW4ubWljcm9zb2Z0b25saW5lLmNvbYIdbG9naW4ubWljcm9zb2Z0
    b25saW5lLWludC5jb22CG2xvZ2luLm1pY3Jvc29mdG9ubGluZS1wLmNvbYIZbG9n
    aW4ubWljcm9zb2Z0b25saW5lLmNvbYIebG9naW4yLm1pY3Jvc29mdG9ubGluZS1p
    bnQuY29tghpsb2dpbjIubWljcm9zb2Z0b25saW5lLmNvbYIfbG9naW5leC5taWNy
    b3NvZnRvbmxpbmUtaW50LmNvbYIbbG9naW5leC5taWNyb3NvZnRvbmxpbmUuY29t
    giRzdGFtcDIubG9naW4ubWljcm9zb2Z0b25saW5lLWludC5jb20wPgYDVR0gBDcw
    NTAzBgZngQwBAgIwKTAnBggrBgEFBQcCARYbaHR0cDovL3d3dy5kaWdpY2VydC5j
    b20vQ1BTMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYB
    BQUHAwIwgY0GA1UdHwSBhTCBgjA/oD2gO4Y5aHR0cDovL2NybDMuZGlnaWNlcnQu
    Y29tL0RpZ2ljZXJ0U0hBMlNlY3VyZVNlcnZlckNBLTEuY3JsMD+gPaA7hjlodHRw
    Oi8vY3JsNC5kaWdpY2VydC5jb20vRGlnaWNlcnRTSEEyU2VjdXJlU2VydmVyQ0Et
    MS5jcmwwfgYIKwYBBQUHAQEEcjBwMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5k
    aWdpY2VydC5jb20wSAYIKwYBBQUHMAKGPGh0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0
    LmNvbS9EaWdpQ2VydFNIQTJTZWN1cmVTZXJ2ZXJDQS0yLmNydDAMBgNVHRMBAf8E
    AjAAMIIBfwYKKwYBBAHWeQIEAgSCAW8EggFrAWkAdwDPEVbu1S58r/OHW9lpLpvp
    GnFnSrAX7KwB0lt3zsw7CAAAAZGocl09AAAEAwBIMEYCIQCCuKCDKarmIGZhQN9X
    IFByEK90C0zLTcwOiZbFiLCiuwIhAKbI/r6Oqb1z5kw+ggrfkfUKwNeLkx2DrKaM
    JEb+XJLHAHYAfVkeEuF4KnscYWd8Xv340IdcFKBOlZ65Ay/ZDowuebgAAAGRqHJc
    9wAABAMARzBFAiBybBJsko8E5sG3U8Ck1CIaZbc/c2lqCmavuJmDt2iQPQIhANG+
    k1XZ7gkJA9+p+O3RgRt3Ni7fHR8aipQh16tkZtLjAHYA5tIxY0B3jMEQQQbXcbnO
    wdJA9paEhvu6hzId/R43jlAAAAGRqHJdIgAABAMARzBFAiBmd5oMBcrfc/HxhmZF
    1O85jIQXioHfAXF4lUIASeeBugIhAKvyPDEO2sxkdgg8MqPvuzLhmgt/9twUO271
    fQ3109ktMA0GCSqGSIb3DQEBCwUAA4IBAQAyoCYf4y7pHNb6OMuW3Uv0zjo+3hb/
    RIz7UJa6yNm2v+aFFG9SKDs/Raa+O6IXb0zpS5Eh12yOEVaow1Lzk4Xuq1uuRfIF
    1LYe0yCNoS/3GsgPNY/f8+kAK/j3OS+fcWsGUfRaibNNkLVuKbALATnvTASB8tmT
    WvmhxeaYO+dpJd9ZVYUA/TfU9jfe2JCD/zaV15TiM0SrS5vYkgnT797sqvWmnHwU
    w87UfBiCCkKCq+JOAxJcKIGbmbHh/GpDGk+TykWcGImmsZJUFD9ep22D/Q6Yflqm
    1iqv5CqX8X5Guq9IDgSDuDdocsLi34YG5nBxzYiRSa0YjQlGZHx4XxNT
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    MIIE6DCCA9CgAwIBAgIQAnQuqhfKjiHHF7sf/P0MoDANBgkqhkiG9w0BAQsFADBh
    MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
    d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD
    QTAeFw0yMDA5MjMwMDAwMDBaFw0zMDA5MjIyMzU5NTlaME0xCzAJBgNVBAYTAlVT
    MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxJzAlBgNVBAMTHkRpZ2lDZXJ0IFNIQTIg
    U2VjdXJlIFNlcnZlciBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
    ANyuWJBNwcQwFZA1W248ghX1LFy949v/cUP6ZCWA1O4Yok3wZtAKc24RmDYXZK83
    nf36QYSvx6+M/hpzTc8zl5CilodTgyu5pnVILR1WN3vaMTIa16yrBvSqXUu3R0bd
    KpPDkC55gIDvEwRqFDu1m5K+wgdlTvza/P96rtxcflUxDOg5B6TXvi/TC2rSsd9f
    /ld0Uzs1gN2ujkSYs58O09rg1/RrKatEp0tYhG2SS4HD2nOLEpdIkARFdRrdNzGX
    kujNVA075ME/OV4uuPNcfhCOhkEAjUVmR7ChZc6gqikJTvOX6+guqw9ypzAO+sf0
    /RR3w6RbKFfCs/mC/bdFWJsCAwEAAaOCAa4wggGqMB0GA1UdDgQWBBQPgGEcgjFh
    1S8o541GOLQs4cbZ4jAfBgNVHSMEGDAWgBQD3lA1VtFMu2bwo+IbG8OXsj3RVTAO
    BgNVHQ8BAf8EBAMCAYYwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMBIG
    A1UdEwEB/wQIMAYBAf8CAQAwdgYIKwYBBQUHAQEEajBoMCQGCCsGAQUFBzABhhho
    dHRwOi8vb2NzcC5kaWdpY2VydC5jb20wQAYIKwYBBQUHMAKGNGh0dHA6Ly9jYWNl
    cnRzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEdsb2JhbFJvb3RDQS5jcnQwewYDVR0f
    BHQwcjA3oDWgM4YxaHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0R2xv
    YmFsUm9vdENBLmNybDA3oDWgM4YxaHR0cDovL2NybDQuZGlnaWNlcnQuY29tL0Rp
    Z2lDZXJ0R2xvYmFsUm9vdENBLmNybDAwBgNVHSAEKTAnMAcGBWeBDAEBMAgGBmeB
    DAECATAIBgZngQwBAgIwCAYGZ4EMAQIDMA0GCSqGSIb3DQEBCwUAA4IBAQB3MR8I
    l9cSm2PSEWUIpvZlubj6kgPLoX7hyA2MPrQbkb4CCF6fWXF7Ef3gwOOPWdegUqHQ
    S1TSSJZI73fpKQbLQxCgLzwWji3+HlU87MOY7hgNI+gH9bMtxKtXc1r2G1O6+x/6
    vYzTUVEgR17vf5irF0LKhVyfIjc0RXbyQ14AniKDrN+v0ebHExfppGlkTIBn6rak
    f4994VH6npdn6mkus5CkHBXIrMtPKex6XF2firjUDLuU7tC8y7WlHgjPxEEDDb0G
    w6D0yDdVSvG/5XlCNatBmO/8EznDu1vr72N8gJzISUZwa6CCUD7QBLbKJcXBBVVf
    8nwvV9GvlW+sbXlr
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  name: azure-ca
  namespace: clusters

$ oc create -f cm.yaml

// Deploy the hosted cluster
// Make sure the pull secret and ssh public key secret are already there in clusters namespace

$ cat hosted-cluster.yaml
apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  name: 'hypershift-cluster'
  namespace: 'clusters'
  labels:
    "cluster.open-cluster-management.io/clusterset": 'default'
spec:
  configuration:
    oauth:
      identityProviders:
        - mappingMethod: claim
          name: EntraID
          openID:
            ca:
              name: azure-ca
            claims:
              email:
                - email
              name:
                - name
              preferredUsername:
                - email
                - upn
            clientID: 9c622143-56ccc-47ad0-1a1a-1asdsa18858saxxx
            clientSecret:
              name: entraid-auth-client-secret-hcp1
            extraAuthorizeParameters:
              include_granted_scopes: 'true'
            extraScopes:
              - email
              - profile
            issuer: >-
              https://login.microsoftonline.com/d44zxxc4-5d99-5a7a-8a33-0vd7854axcxxc
          type: OpenID
  etcd:
    managed:
      storage:
        persistentVolume:
          size: 8Gi
        type: PersistentVolume
    managementType: Managed
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.16.11-multi
  pullSecret:
    name: pullsecret-cluster-hypershift-cluster
  sshKey:
    name: sshkey-cluster-hypershift-cluster
  networking:
    clusterNetwork:
      - cidr: 10.132.0.0/14
    serviceNetwork:
      - cidr: 172.31.0.0/16
    networkType: OVNKubernetes
  controllerAvailabilityPolicy: HighlyAvailable
  platform:
    type: KubeVirt
    kubevirt:
      baseDomainPassthrough: true
  infraID: 'hypershift-cluster'
  services:
  - service: OAuthServer
    servicePublishingStrategy:
      type: Route
  - service: OIDC
    servicePublishingStrategy:
      type: Route
  - service: Konnectivity
    servicePublishingStrategy:
      type: Route
  - service: Ignition
    servicePublishingStrategy:
      type: Route

$ oc create -f hosted-cluster.yaml

The installation will stuck with only below pods.

oc get pod
NAME                                          READY   STATUS    RESTARTS   AGE
capi-provider-5b4c988f68-9krgq                1/1     Running   0          13m
cluster-api-685c6d645f-z4jkj                  1/1     Running   0          13m
control-plane-operator-5c78f979df-xlxnt       1/1     Running   0          13m
control-plane-pki-operator-64f45cd885-kq4b8   1/1     Running   0          13m
etcd-0                                        4/4     Running   0          13m
etcd-1                                        4/4     Running   0          13m
etcd-2                                        4/4     Running   0          13m
ignition-server-bcd449dc4-h4j9d               1/1     Running   0          11m
ignition-server-bcd449dc4-sgh9m               1/1     Running   0          11m
ignition-server-bcd449dc4-snw28               1/1     Running   0          11m
ignition-server-proxy-9666bb6cf-9vhzg         1/1     Running   0          11m
ignition-server-proxy-9666bb6cf-lh5fl         1/1     Running   0          11m
ignition-server-proxy-9666bb6cf-rflvq         1/1     Running   0          11m
konnectivity-agent-54b849747b-fz77r           1/1     Running   0          11m
konnectivity-agent-54b849747b-gr2bm           1/1     Running   0          11m
konnectivity-agent-54b849747b-hxxh8           1/1     Running   0          11m
kube-apiserver-b45d65848-j44k7                4/4     Running   0          12m
kube-apiserver-b45d65848-sp2cv                4/4     Running   0          12m
kube-apiserver-b45d65848-wskk5                4/4     Running   0          12m
kube-controller-manager-6476d47f67-4pjn5      1/1     Running   0          12m
kube-controller-manager-6476d47f67-hrv4r      1/1     Running   0          12m
kube-controller-manager-6476d47f67-wdwz7      1/1     Running   0          12m
kube-scheduler-f6ffb896b-54rxh                1/1     Running   0          12m
kube-scheduler-f6ffb896b-dm6h8                1/1     Running   0          12m
kube-scheduler-f6ffb896b-w7xqp                1/1     Running   0          12m
openshift-apiserver-fdd7449f6-48lwz           3/3     Running   0          12m
openshift-apiserver-fdd7449f6-r6r8l           3/3     Running   0          12m
openshift-apiserver-fdd7449f6-xz4pj           3/3     Running   0          12m
openshift-oauth-apiserver-7466694656-22kgt    2/2     Running   0          11m
openshift-oauth-apiserver-7466694656-mhtlh    2/2     Running   0          11m
openshift-oauth-apiserver-7466694656-tvx8w    2/2     Running   0          11m

OAuth related errors will be there in control-plane-operator pod logs.

$ oc logs control-plane-operator-5c78f979df-xlxnt
...{"level":"info","ts":"2024-10-19T07:50:23Z","msg":"Reconciling OpenShift OAuth API Server","controller":"hostedcontrolplane","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedControlPlane","HostedControlPlane":{"name":"hypershift-cluster","namespace":"clusters-hypershift-cluster"},"namespace":"clusters-hypershift-cluster","name":"hypershift-cluster","reconcileID":"604a69a1-47a7-4efd-b410-be6b602c77a4"}{"level":"info","ts":"2024-10-19T07:50:23Z","msg":"Reconciled openshift oauth apiserver pdb","controller":"hostedcontrolplane","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedControlPlane","HostedControlPlane":{"name":"hypershift-cluster","namespace":"clusters-hypershift-cluster"},"namespace":"clusters-hypershift-cluster","name":"hypershift-cluster","reconcileID":"604a69a1-47a7-4efd-b410-be6b602c77a4","result":"unchanged"}{"level":"info","ts":"2024-10-19T07:50:23Z","msg":"Reconciling OAuth Server","controller":"hostedcontrolplane","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedControlPlane","HostedControlPlane":{"name":"hypershift-cluster","namespace":"clusters-hypershift-cluster"},"namespace":"clusters-hypershift-cluster","name":"hypershift-cluster","reconcileID":"604a69a1-47a7-4efd-b410-be6b602c77a4"}{"level":"error","ts":"2024-10-19T07:50:23Z","msg":"Reconciler error","controller":"hostedcontrolplane","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedControlPlane","HostedControlPlane":{"name":"hypershift-cluster","namespace":"clusters-hypershift-cluster"},"namespace":"clusters-hypershift-cluster","name":"hypershift-cluster","reconcileID":"604a69a1-47a7-4efd-b410-be6b602c77a4","error":"failed to update control plane: failed to reconcile openshift oauth apiserver: failed to reconcile oauth server config: failed to generate oauth config: failed to apply IDP EntraID config: EOF","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates