-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
4.17.z
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
Customer Escalated
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
While trying to upgrade a HostedCluster from 4.16.9 to 4.17.9 it is stuck while upgrading the kube-apiserver pods (kube-apiserver container fail to start, causing also apply-bootstrap container to fail because apiserver never starts) when processing FeatureGates : E1229 13:51:24.627103 1 run.go:74] "command failed" err="invalid argument \"ValidatingAdmissionPolicy=false\" for \"--feature-gates\" flag: cannot set feature gate ValidatingAdmissionPolicy to false, feature is locked to true"
1/3 kube-apiserver of hosted cluster are in CrashLoopBackOff with the feature is locked to true message, the two other are on a different ReplicaSet so I think they are at a different revision
Hub Cluster is updated to 4.17.9: RHACM : 2.12.1 MCE : 2.7.2 HostedCluster is currently at 4.16.9 and facing issue while switching to stable-4.17 for 4.17.9. Cluster is disconnected environment. - HCP must-gather in 04021207 supportshell
Steps to Reproduce:
1. Upgrading HCP hostedCluster from 4.16.9 to 4.17.9
Actual results:
Fhe feature gate of ValidatingAdmissionPolicy was set to false, but in 1.30 it has been GA to invalidate, but kube-apiserver of the hosted cluster is failing. There was a PR that bumps openshift/api dependency, and k8s.io/api to v1.30.1 for this specific issue. References can be found here, which are specific to the issue that we're facing: https://github.com/openshift/hypershift/pull/4099 https://github.com/openshift/hypershift/pull/4095 https://github.com/openshift/release/pull/52491 https://issues.redhat.com/browse/HOSTEDCP-1693
Expected results:
To be able to upgrade without Apiserver being blocked.
Additional info:
We already tried modifying the APIServer and featuregates(https://access.redhat.com/solutions/5685971) and a restart of the hosted cluster control plane. Changing the ConfigMap that is attached to kube-apiserver, change the resource KubeAPIServer from HostedCluster kubeconfig, or edit the FeatureGate resource from the HostedCluster kubeconfig, none of them works. To revert it, changing the HostedCluster to stable-4.16 and the 4.16.9 image and the kube-apiserver came back healthy so it seems that as soon as I switch to stable-4.17 and the 4.17.9 image (which is the latest available), the first thing that happens is creating a kube-apiserver revision but this first step is stuck forever because of the ValidatingAdmissionPolicy that is disabled in HostedCluster.