Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-64589

[RHCOS 10] NodeSwap feature gate panics kubelet

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.21
    • Node / Kubelet
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      When creating an OpenShift 4.21 cluster with release image registry.ci.openshift.org/rhcos-devel/ocp-4.21-10.1:ocp-release-4.21.0-ec.1-x86_64

Kubelet is unable to start in the master node and i.e. install fails

      Version-Release number of selected component (if applicable):

      4.21.0-ec.0 with RHEL 10 pre-test images

       

      Following guidance from https://docs.google.com/document/d/1wuTqy5Vqz6mCQYWFOX-3qAFBwV-L9xVORfYg1Nc5qc0/edit?tab=t.0 

       

      How reproducible:

      Always

      Steps to Reproduce:

          1.  Create a OpenShift cluster on GCP with OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE="registry.ci.openshift.org/rhcos-devel/ocp-4.21-10.1:ocp-release-4.21.0-ec.1-x86_64"
    
    2. openshift-install will fail after a while

    3. ssh into the master node and check kubelet logs

[core@swghosh-20251103-a24e-jngk6-master-0 ~]$ sudo journalctl -xeu kubelet
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339738  181580 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339741  181580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339743  181580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339746  181580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339748  181580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339751  181580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339753  181580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339755  181580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339758  181580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339760  181580 feature_gate.go:328] unrecognized feature gate: NewOLM
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339763  181580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339765  181580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339767  181580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339770  181580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: W1103 11:20:44.339772  181580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: panic: failed to merge global and in-flight KubeletConfiguration while setting defaults, error: cannot set feature gate NodeSwap to false, feature is locked to true
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: goroutine 1 [running]:
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: k8s.io/kubernetes/pkg/kubelet/apis/config/v1beta1.SetDefaults_KubeletConfiguration(0xc000755808)
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         k8s.io/kubernetes/pkg/kubelet/apis/config/v1beta1/defaults.go:71 +0x1439
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: k8s.io/kubernetes/pkg/kubelet/apis/config/v1beta1.SetObjectDefaults_KubeletConfiguration(0xc000755808)
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         k8s.io/kubernetes/pkg/kubelet/apis/config/v1beta1/zz_generated.defaults.go:41 +0x2a
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: k8s.io/kubernetes/pkg/kubelet/apis/config/v1beta1.addDefaultingFuncs.RegisterDefaults.func1({0x254cd20?, 0xc000755808?})
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         k8s.io/kubernetes/pkg/kubelet/apis/config/v1beta1/zz_generated.defaults.go:35 +0x2d
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: k8s.io/apimachinery/pkg/runtime.(*Scheme).Default(0x3c2abf0?, {0x2a22ad0, 0xc000755808})
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         k8s.io/apimachinery/pkg/runtime/scheme.go:356 +0x9f
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).Decode(0xc000945900, {0xc000344a00?, 0x0?, 0xc00030d7a0?}, 0x2a180e0?, {0x0, 0x0})
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:187 +0x596
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: k8s.io/kubernetes/pkg/kubelet/kubeletconfig/util/codec.DecodeKubeletConfiguration({0x2a42238, 0x3d28780}, 0xc000907860?, {0xc000344a00, 0x139a, 0x139b})
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         k8s.io/kubernetes/pkg/kubelet/kubeletconfig/util/codec/codec.go:75 +0xef
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: k8s.io/kubernetes/pkg/kubelet/kubeletconfig/configfiles.(*fsLoader).Load(0xc0000d37a0, {0x2a42238, 0x3d28780})
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         k8s.io/kubernetes/pkg/kubelet/kubeletconfig/configfiles/configfiles.go:76 +0x89
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: k8s.io/kubernetes/cmd/kubelet/app.loadConfigFile({0x2a42238, 0x3d28780}, {0x7ffe31db6789, 0x1c})
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         k8s.io/kubernetes/cmd/kubelet/app/server.go:448 +0x1df
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: k8s.io/kubernetes/cmd/kubelet/app.NewKubeletCommand.func1(0xc000576908, {0xc000052150, 0x12, 0x13})
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         k8s.io/kubernetes/cmd/kubelet/app/server.go:219 +0x2be
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: github.com/spf13/cobra.(*Command).execute(0xc000576908, {0xc000052150, 0x12, 0x13})
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         github.com/spf13/cobra@v1.9.1/command.go:1015 +0xaaa
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: github.com/spf13/cobra.(*Command).ExecuteC(0xc000576908)
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         github.com/spf13/cobra@v1.9.1/command.go:1148 +0x46f
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: github.com/spf13/cobra.(*Command).Execute(...)
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         github.com/spf13/cobra@v1.9.1/command.go:1071
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: k8s.io/component-base/cli.run(0xc000576908)
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         k8s.io/component-base/cli/run.go:143 +0x245
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: k8s.io/component-base/cli.Run(0x2a42238?)
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         k8s.io/component-base/cli/run.go:44 +0x17
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]: main.main()
Nov 03 11:20:44 swghosh-20251103-a24e-jngk6-master-0 kubenswrapper[181580]:         k8s.io/kubernetes/cmd/kubelet/kubelet.go:37 +0x2a    

      Actual results:

      Kubelet panics at NodeSwap feature gate

      Expected results:

          OpenShift 4.21 cluster should be able to come up on RHCOS 10

      Additional info:

              rh-ee-ngopalak Neeraj Krishna Gopalakrishna
              swghosh@redhat.com Swarup Ghosh
              None
              None
              Min Li Min Li
              None
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: