Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-47767

[RHACM/HCP] kube-apiserver feature gate ValidatingAdmissionPolicy issue

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • Customer Escalated
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          While trying to upgrade a HostedCluster from 4.16.9 to 4.17.9 it is stuck while upgrading the kube-apiserver pods                                            (kube-apiserver container fail to start, causing also apply-bootstrap container to fail because apiserver never starts) when processing FeatureGates :
      E1229 13:51:24.627103 1 run.go:74] "command failed" err="invalid argument \"ValidatingAdmissionPolicy=false\" for \"--feature-gates\" flag: cannot set feature gate ValidatingAdmissionPolicy to false, feature is locked to true"

      1/3 kube-apiserver of hosted cluster are in CrashLoopBackOff with the feature is locked to true message, the two other are on a different ReplicaSet so I think they are at a different revision

      Hub Cluster is updated to 4.17.9:
      RHACM : 2.12.1
      MCE : 2.7.2
      HostedCluster is currently at 4.16.9 and facing issue while switching to stable-4.17 for 4.17.9.
      Cluster is disconnected environment.
      - HCP must-gather in 04021207 supportshell

       

      Steps to Reproduce:

          1. Upgrading HCP hostedCluster from 4.16.9 to 4.17.9
          

      Actual results:

          Fhe feature gate of ValidatingAdmissionPolicy was set to false, but in 1.30 it has been GA to invalidate, but kube-apiserver of the hosted cluster is failing. There was a PR that bumps openshift/api dependency, and k8s.io/api to v1.30.1 for this specific issue.
      References can be found here, which are specific to the issue that we're facing:
      
      https://github.com/openshift/hypershift/pull/4099 
      https://github.com/openshift/hypershift/pull/4095 
      https://github.com/openshift/release/pull/52491
      https://issues.redhat.com/browse/HOSTEDCP-1693
      

      Expected results:

      To be able to upgrade without Apiserver being blocked.    

      Additional info:

          We already tried modifying the APIServer and featuregates(https://access.redhat.com/solutions/5685971) and a restart of the hosted cluster control plane. Changing the ConfigMap that is attached to kube-apiserver, change the resource KubeAPIServer from HostedCluster kubeconfig, or edit the FeatureGate resource from the HostedCluster kubeconfig, none of them works. To revert it, changing the HostedCluster to stable-4.16 and the 4.16.9 image and the kube-apiserver came back healthy so it seems that as soon as I switch to stable-4.17 and the 4.17.9 image (which is the latest available), the first thing that happens is creating a kube-apiserver revision but this first step is stuck forever because of the ValidatingAdmissionPolicy that is disabled in HostedCluster.
      
       

              jparrill@redhat.com Juan Manuel Parrilla Madrid
              rhn-support-dahernan David Hernandez Fernandez
              David Hernandez Fernandez
              None
              Liangquan Li Liangquan Li
              None
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: