-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
4.16.0
-
Moderate
-
None
-
False
-
The Hosted Cluster installation completes successfully for version 4.15.9 with OAuth configuration. Also, the same cluster can be upgraded to the 4.16 version without any issues. However, if we create a new Hosted Cluster for the 4.16.11 version, the installation gets stuck at the OAuth part where control-plane-operator fails to generate oauth config even though the required secrets for OAuth are there already, same configuration works for the 4.15.9 version.
The issue is reproducible every time. I am sharing the steps below for reproducing the issue.
// Create the required configmap and secrets for HCP OAuth in Hosted Cluster namespace. $ oc new-project clusters $ oc create secret generic entraid-auth-client-secret-hcp1 --from-literal=clientSecret=abcd1234 $ cat cm.yaml apiVersion: v1 data: ca.crt: | -----BEGIN CERTIFICATE----- MIIH1jCCBr6gAwIBAgIQBVfgNq0MljBx+Ba1Wvj3xDANBgkqhkiG9w0BAQsFADBN MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMScwJQYDVQQDEx5E aWdpQ2VydCBTSEEyIFNlY3VyZSBTZXJ2ZXIgQ0EwHhcNMjQwODMxMDAwMDAwWhcN MjUwMjI4MjM1OTU5WjB/MQswCQYDVQQGEwJVUzETMBEGA1UECBMKV2FzaGluZ3Rv bjEQMA4GA1UEBxMHUmVkbW9uZDEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0 aW9uMSkwJwYDVQQDEyBzdGFtcDIubG9naW4ubWljcm9zb2Z0b25saW5lLmNvbTCC ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL/3nnemNHTU4q/vofTx/8Wx xUKElp4pG+X/L76eH43FGkm3AaX/Sd8MyA8qq6ac4sK7h6O1pnfrMT7wTy0VIjnu p9M25cwvhRHDFjSO4CNVho0CO5m1uoi+wxdJ3N6jEHyq8Gq0LtNYj1h3BUzrOUnx 2im32DIv22RPbNML4KN4xmQfdkAm3EZdHCbyEXmGfAFxXag16FCz0Lz6qBp9QVHt FgaY9pH7KVMKnv2+JtrJ4YA/uwPAIU69DMRWIFcoKijtWzQDNlDdufOLKSx3ipeh o5FQ8RwErir1Q4825b2hg3ZcvZ1xqiy+d9thwF8xO5IjcVdhvtxjC7S4IGR1euEC AwEAAaOCBH4wggR6MB8GA1UdIwQYMBaAFA+AYRyCMWHVLyjnjUY4tCzhxtniMB0G A1UdDgQWBBSD+HXo3eMKSUoiDLuccKPSAp5mdzCCASYGA1UdEQSCAR0wggEZgiBz dGFtcDIubG9naW4ubWljcm9zb2Z0b25saW5lLmNvbYIdbG9naW4ubWljcm9zb2Z0 b25saW5lLWludC5jb22CG2xvZ2luLm1pY3Jvc29mdG9ubGluZS1wLmNvbYIZbG9n aW4ubWljcm9zb2Z0b25saW5lLmNvbYIebG9naW4yLm1pY3Jvc29mdG9ubGluZS1p bnQuY29tghpsb2dpbjIubWljcm9zb2Z0b25saW5lLmNvbYIfbG9naW5leC5taWNy b3NvZnRvbmxpbmUtaW50LmNvbYIbbG9naW5leC5taWNyb3NvZnRvbmxpbmUuY29t giRzdGFtcDIubG9naW4ubWljcm9zb2Z0b25saW5lLWludC5jb20wPgYDVR0gBDcw NTAzBgZngQwBAgIwKTAnBggrBgEFBQcCARYbaHR0cDovL3d3dy5kaWdpY2VydC5j b20vQ1BTMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYB BQUHAwIwgY0GA1UdHwSBhTCBgjA/oD2gO4Y5aHR0cDovL2NybDMuZGlnaWNlcnQu Y29tL0RpZ2ljZXJ0U0hBMlNlY3VyZVNlcnZlckNBLTEuY3JsMD+gPaA7hjlodHRw Oi8vY3JsNC5kaWdpY2VydC5jb20vRGlnaWNlcnRTSEEyU2VjdXJlU2VydmVyQ0Et MS5jcmwwfgYIKwYBBQUHAQEEcjBwMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5k aWdpY2VydC5jb20wSAYIKwYBBQUHMAKGPGh0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0 LmNvbS9EaWdpQ2VydFNIQTJTZWN1cmVTZXJ2ZXJDQS0yLmNydDAMBgNVHRMBAf8E AjAAMIIBfwYKKwYBBAHWeQIEAgSCAW8EggFrAWkAdwDPEVbu1S58r/OHW9lpLpvp GnFnSrAX7KwB0lt3zsw7CAAAAZGocl09AAAEAwBIMEYCIQCCuKCDKarmIGZhQN9X IFByEK90C0zLTcwOiZbFiLCiuwIhAKbI/r6Oqb1z5kw+ggrfkfUKwNeLkx2DrKaM JEb+XJLHAHYAfVkeEuF4KnscYWd8Xv340IdcFKBOlZ65Ay/ZDowuebgAAAGRqHJc 9wAABAMARzBFAiBybBJsko8E5sG3U8Ck1CIaZbc/c2lqCmavuJmDt2iQPQIhANG+ k1XZ7gkJA9+p+O3RgRt3Ni7fHR8aipQh16tkZtLjAHYA5tIxY0B3jMEQQQbXcbnO wdJA9paEhvu6hzId/R43jlAAAAGRqHJdIgAABAMARzBFAiBmd5oMBcrfc/HxhmZF 1O85jIQXioHfAXF4lUIASeeBugIhAKvyPDEO2sxkdgg8MqPvuzLhmgt/9twUO271 fQ3109ktMA0GCSqGSIb3DQEBCwUAA4IBAQAyoCYf4y7pHNb6OMuW3Uv0zjo+3hb/ RIz7UJa6yNm2v+aFFG9SKDs/Raa+O6IXb0zpS5Eh12yOEVaow1Lzk4Xuq1uuRfIF 1LYe0yCNoS/3GsgPNY/f8+kAK/j3OS+fcWsGUfRaibNNkLVuKbALATnvTASB8tmT WvmhxeaYO+dpJd9ZVYUA/TfU9jfe2JCD/zaV15TiM0SrS5vYkgnT797sqvWmnHwU w87UfBiCCkKCq+JOAxJcKIGbmbHh/GpDGk+TykWcGImmsZJUFD9ep22D/Q6Yflqm 1iqv5CqX8X5Guq9IDgSDuDdocsLi34YG5nBxzYiRSa0YjQlGZHx4XxNT -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIE6DCCA9CgAwIBAgIQAnQuqhfKjiHHF7sf/P0MoDANBgkqhkiG9w0BAQsFADBh MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3 d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD QTAeFw0yMDA5MjMwMDAwMDBaFw0zMDA5MjIyMzU5NTlaME0xCzAJBgNVBAYTAlVT MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxJzAlBgNVBAMTHkRpZ2lDZXJ0IFNIQTIg U2VjdXJlIFNlcnZlciBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB ANyuWJBNwcQwFZA1W248ghX1LFy949v/cUP6ZCWA1O4Yok3wZtAKc24RmDYXZK83 nf36QYSvx6+M/hpzTc8zl5CilodTgyu5pnVILR1WN3vaMTIa16yrBvSqXUu3R0bd KpPDkC55gIDvEwRqFDu1m5K+wgdlTvza/P96rtxcflUxDOg5B6TXvi/TC2rSsd9f /ld0Uzs1gN2ujkSYs58O09rg1/RrKatEp0tYhG2SS4HD2nOLEpdIkARFdRrdNzGX kujNVA075ME/OV4uuPNcfhCOhkEAjUVmR7ChZc6gqikJTvOX6+guqw9ypzAO+sf0 /RR3w6RbKFfCs/mC/bdFWJsCAwEAAaOCAa4wggGqMB0GA1UdDgQWBBQPgGEcgjFh 1S8o541GOLQs4cbZ4jAfBgNVHSMEGDAWgBQD3lA1VtFMu2bwo+IbG8OXsj3RVTAO BgNVHQ8BAf8EBAMCAYYwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMBIG A1UdEwEB/wQIMAYBAf8CAQAwdgYIKwYBBQUHAQEEajBoMCQGCCsGAQUFBzABhhho dHRwOi8vb2NzcC5kaWdpY2VydC5jb20wQAYIKwYBBQUHMAKGNGh0dHA6Ly9jYWNl cnRzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEdsb2JhbFJvb3RDQS5jcnQwewYDVR0f BHQwcjA3oDWgM4YxaHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0R2xv YmFsUm9vdENBLmNybDA3oDWgM4YxaHR0cDovL2NybDQuZGlnaWNlcnQuY29tL0Rp Z2lDZXJ0R2xvYmFsUm9vdENBLmNybDAwBgNVHSAEKTAnMAcGBWeBDAEBMAgGBmeB DAECATAIBgZngQwBAgIwCAYGZ4EMAQIDMA0GCSqGSIb3DQEBCwUAA4IBAQB3MR8I l9cSm2PSEWUIpvZlubj6kgPLoX7hyA2MPrQbkb4CCF6fWXF7Ef3gwOOPWdegUqHQ S1TSSJZI73fpKQbLQxCgLzwWji3+HlU87MOY7hgNI+gH9bMtxKtXc1r2G1O6+x/6 vYzTUVEgR17vf5irF0LKhVyfIjc0RXbyQ14AniKDrN+v0ebHExfppGlkTIBn6rak f4994VH6npdn6mkus5CkHBXIrMtPKex6XF2firjUDLuU7tC8y7WlHgjPxEEDDb0G w6D0yDdVSvG/5XlCNatBmO/8EznDu1vr72N8gJzISUZwa6CCUD7QBLbKJcXBBVVf 8nwvV9GvlW+sbXlr -----END CERTIFICATE----- kind: ConfigMap metadata: name: azure-ca namespace: clusters $ oc create -f cm.yaml
// Deploy the hosted cluster // Make sure the pull secret and ssh public key secret are already there in clusters namespace $ cat hosted-cluster.yaml apiVersion: hypershift.openshift.io/v1beta1 kind: HostedCluster metadata: name: 'hypershift-cluster' namespace: 'clusters' labels: "cluster.open-cluster-management.io/clusterset": 'default' spec: configuration: oauth: identityProviders: - mappingMethod: claim name: EntraID openID: ca: name: azure-ca claims: email: - email name: - name preferredUsername: - email - upn clientID: 9c622143-56ccc-47ad0-1a1a-1asdsa18858saxxx clientSecret: name: entraid-auth-client-secret-hcp1 extraAuthorizeParameters: include_granted_scopes: 'true' extraScopes: - email - profile issuer: >- https://login.microsoftonline.com/d44zxxc4-5d99-5a7a-8a33-0vd7854axcxxc type: OpenID etcd: managed: storage: persistentVolume: size: 8Gi type: PersistentVolume managementType: Managed release: image: quay.io/openshift-release-dev/ocp-release:4.16.11-multi pullSecret: name: pullsecret-cluster-hypershift-cluster sshKey: name: sshkey-cluster-hypershift-cluster networking: clusterNetwork: - cidr: 10.132.0.0/14 serviceNetwork: - cidr: 172.31.0.0/16 networkType: OVNKubernetes controllerAvailabilityPolicy: HighlyAvailable platform: type: KubeVirt kubevirt: baseDomainPassthrough: true infraID: 'hypershift-cluster' services: - service: OAuthServer servicePublishingStrategy: type: Route - service: OIDC servicePublishingStrategy: type: Route - service: Konnectivity servicePublishingStrategy: type: Route - service: Ignition servicePublishingStrategy: type: Route $ oc create -f hosted-cluster.yaml
The installation will stuck with only below pods.
oc get pod NAME READY STATUS RESTARTS AGE capi-provider-5b4c988f68-9krgq 1/1 Running 0 13m cluster-api-685c6d645f-z4jkj 1/1 Running 0 13m control-plane-operator-5c78f979df-xlxnt 1/1 Running 0 13m control-plane-pki-operator-64f45cd885-kq4b8 1/1 Running 0 13m etcd-0 4/4 Running 0 13m etcd-1 4/4 Running 0 13m etcd-2 4/4 Running 0 13m ignition-server-bcd449dc4-h4j9d 1/1 Running 0 11m ignition-server-bcd449dc4-sgh9m 1/1 Running 0 11m ignition-server-bcd449dc4-snw28 1/1 Running 0 11m ignition-server-proxy-9666bb6cf-9vhzg 1/1 Running 0 11m ignition-server-proxy-9666bb6cf-lh5fl 1/1 Running 0 11m ignition-server-proxy-9666bb6cf-rflvq 1/1 Running 0 11m konnectivity-agent-54b849747b-fz77r 1/1 Running 0 11m konnectivity-agent-54b849747b-gr2bm 1/1 Running 0 11m konnectivity-agent-54b849747b-hxxh8 1/1 Running 0 11m kube-apiserver-b45d65848-j44k7 4/4 Running 0 12m kube-apiserver-b45d65848-sp2cv 4/4 Running 0 12m kube-apiserver-b45d65848-wskk5 4/4 Running 0 12m kube-controller-manager-6476d47f67-4pjn5 1/1 Running 0 12m kube-controller-manager-6476d47f67-hrv4r 1/1 Running 0 12m kube-controller-manager-6476d47f67-wdwz7 1/1 Running 0 12m kube-scheduler-f6ffb896b-54rxh 1/1 Running 0 12m kube-scheduler-f6ffb896b-dm6h8 1/1 Running 0 12m kube-scheduler-f6ffb896b-w7xqp 1/1 Running 0 12m openshift-apiserver-fdd7449f6-48lwz 3/3 Running 0 12m openshift-apiserver-fdd7449f6-r6r8l 3/3 Running 0 12m openshift-apiserver-fdd7449f6-xz4pj 3/3 Running 0 12m openshift-oauth-apiserver-7466694656-22kgt 2/2 Running 0 11m openshift-oauth-apiserver-7466694656-mhtlh 2/2 Running 0 11m openshift-oauth-apiserver-7466694656-tvx8w 2/2 Running 0 11m
OAuth related errors will be there in control-plane-operator pod logs.
$ oc logs control-plane-operator-5c78f979df-xlxnt ...{"level":"info","ts":"2024-10-19T07:50:23Z","msg":"Reconciling OpenShift OAuth API Server","controller":"hostedcontrolplane","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedControlPlane","HostedControlPlane":{"name":"hypershift-cluster","namespace":"clusters-hypershift-cluster"},"namespace":"clusters-hypershift-cluster","name":"hypershift-cluster","reconcileID":"604a69a1-47a7-4efd-b410-be6b602c77a4"}{"level":"info","ts":"2024-10-19T07:50:23Z","msg":"Reconciled openshift oauth apiserver pdb","controller":"hostedcontrolplane","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedControlPlane","HostedControlPlane":{"name":"hypershift-cluster","namespace":"clusters-hypershift-cluster"},"namespace":"clusters-hypershift-cluster","name":"hypershift-cluster","reconcileID":"604a69a1-47a7-4efd-b410-be6b602c77a4","result":"unchanged"}{"level":"info","ts":"2024-10-19T07:50:23Z","msg":"Reconciling OAuth Server","controller":"hostedcontrolplane","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedControlPlane","HostedControlPlane":{"name":"hypershift-cluster","namespace":"clusters-hypershift-cluster"},"namespace":"clusters-hypershift-cluster","name":"hypershift-cluster","reconcileID":"604a69a1-47a7-4efd-b410-be6b602c77a4"}{"level":"error","ts":"2024-10-19T07:50:23Z","msg":"Reconciler error","controller":"hostedcontrolplane","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedControlPlane","HostedControlPlane":{"name":"hypershift-cluster","namespace":"clusters-hypershift-cluster"},"namespace":"clusters-hypershift-cluster","name":"hypershift-cluster","reconcileID":"604a69a1-47a7-4efd-b410-be6b602c77a4","error":"failed to update control plane: failed to reconcile openshift oauth apiserver: failed to reconcile oauth server config: failed to generate oauth config: failed to apply IDP EntraID config: EOF","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
