-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
4.21
Description of problem:
NodePool has skew with HostedControlPlane, the np is using 4.18 or 4.19 while hcp is in 4.21, hc could't be ready, since service-ca-operator fails due to UserNamespacesSupport feature gate mismatch. That caused many clusteroperator failed with error such as 'tls secret not found'.
Version-Release number of selected component (if applicable):
4.21
How reproducible:
always
Steps to Reproduce:
1.Render hc resources to yaml file when creating 4.21 hostedcluster
2.Update the np release image to 4.18 or 4.19 version, and apply all the resources.
3.Check np and hcp
Actual results:
Step3 : np is ready, but hc couldn't be ready since many clusteroperator failed
oc get hc -A NAMESPACE NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE clusters wxj-18-nodeskew-421 wxj-18-nodeskew-421-admin-kubeconfig Partial True False The hosted control plane is available oc get np -A NAMESPACE NAME CLUSTER DESIRED NODES CURRENT NODES AUTOSCALING AUTOREPAIR VERSION UPDATINGVERSION UPDATINGCONFIG MESSAGE clusters wxj-18-nodeskew-421-us-east-2a wxj-18-nodeskew-421 2 2 False False 4.18.0-0.nightly-2025-11-17-063304 False False
Many clusteroperators in guest cluster are degraded
oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console csi-snapshot-controller 4.21.0-0.nightly-2025-11-13-042845 True False False 4h36m dns 4.21.0-0.nightly-2025-11-13-042845 False True True 4h36m DNS "default" is unavailable. image-registry False True True 4h37m Available: The deployment does not have available replicas... ingress False True True 4h36m The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.) insights 4.21.0-0.nightly-2025-11-13-042845 False False True 165m Unable to report: unable to build request to connect to Insights server: Post "https://console.redhat.com/api/ingress/v1/upload": dial tcp: lookup console.redhat.com on 172.31.0.10:53: read udp 10.133.0.10:41898->172.31.0.10:53: read: connection refused kube-apiserver 4.21.0-0.nightly-2025-11-13-042845 True False False 4h37m kube-controller-manager 4.21.0-0.nightly-2025-11-13-042845 True False False 4h37m kube-scheduler 4.21.0-0.nightly-2025-11-13-042845 True False False 4h37m kube-storage-version-migrator 4.21.0-0.nightly-2025-11-13-042845 True False False 4h31m monitoring network 4.21.0-0.nightly-2025-11-13-042845 True True False 4h36m DaemonSet "/openshift-multus/network-metrics-daemon" is waiting for other operators to become ready... node-tuning 4.21.0-0.nightly-2025-11-13-042845 True False False 4h33m openshift-apiserver 4.21.0-0.nightly-2025-11-13-042845 True False False 4h37m openshift-controller-manager 4.21.0-0.nightly-2025-11-13-042845 True False False 4h37m openshift-samples operator-lifecycle-manager 4.21.0-0.nightly-2025-11-13-042845 True False False 4h36m operator-lifecycle-manager-catalog 4.21.0-0.nightly-2025-11-13-042845 True False False 4h37m operator-lifecycle-manager-packageserver 4.21.0-0.nightly-2025-11-13-042845 True False False 4h37m service-ca storage 4.21.0-0.nightly-2025-11-13-042845 True False False 4h33m
Service ca pod has error "the feature gate "UserNamespacesSupport" is disabled: can't set spec.HostUsers"
oc get pods -n openshift-service-ca-operator NAME READY STATUS RESTARTS AGE service-ca-operator-85cdc9949d-cmmrw 0/1 ContainerCreating 0 4h34m oc describe pods -n openshift-service-ca-operator Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 2m56s (x1246 over 4h33m) kubelet Failed to create pod sandbox: the feature gate "UserNamespacesSupport" is disabled: can't set spec.HostUsers
Image registry failed due the tls secret not created
oc get pods -n openshift-image-registry
NAME READY STATUS RESTARTS AGE
image-registry-5d5c766c68-mksmp 0/1 ContainerCreating 0 4h34m
image-registry-7d657df547-ccx5n 0/1 ContainerCreating 0 4h35m
node-ca-cd8f9 1/1 Running 0 4h35m
node-ca-w85x9 1/1 Running 0 4h35m
oc describe pods image-registry-5d5c766c68-mksmp -n openshift-image-registry
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 3m33s (x141 over 4h34m) kubelet MountVolume.SetUp failed for volume "registry-tls" : secret "image-registry-tls" not found
Expected results:
The hc should be ready when np has skew with hcp.
Additional info:
Always meet this issue when hcp is 4.21 while np is 4.18/4.19/4.17(np skew Y-4 is not support offically)
Can't meet this issue when hcp is 4.20 while np is 4.18/4.19
Can't meet this issue when hcp is 4.21 while np is 4.16(This test scope is not support offically, just show example here)
Dump log https://drive.google.com/file/d/1Vl6JjNTlmHnGVA-JlwPSVhtOFm7c2eQ_/view?usp=drive_link
