Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-60953

Missing NodeLogQuery feature gate in kubelet configuration

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Major Major
    • None
    • 4.20.0
    • Windows Containers
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 3
    • Important
    • None
    • None
    • None
    • WINC - Sprint 277
    • 1
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      The NodeLogQuery feature gate in kubelet is not honoring its default false value when not explicitly specified in the featureGates configuration given the downstream Openshift fork toggles the default to `true`

       

      https://github.com/openshift/kubernetes/blob/a237dcbeee8bbe1440c94349399cc6eac8870910/pkg/features/kube_features.go#L1543
       
      Resulting on: NodeLogQuery is omitted from the kubeletConfiguration featureGates array, the log query endpoints remain accessible despite the expected default behavior of being disabled.

       

      Version-Release number of selected component (if applicable):

      kubelet v1.32.6 (confirmed also in v1.32.5 and v1.31.9)

      How reproducible:

      Always - occurs consistently across multiple kubelet versions

      Steps to Reproduce:

      1. Configure kubelet with enableSystemLogQuery: true in kubeletConfiguration
      2. Do NOT include NodeLogQuery in the featureGates array (expecting default false behavior)
      3. Start kubelet service
      4. Test log query endpoints on both Linux and Windows nodes:

      • Linux: kubectl get --raw "/api/v1/nodes/linux-node/proxy/logs/?query=sshd&tailLines=1"
      • Windows: kubectl get --raw "/api/v1/nodes/win-node/proxy/logs/?query=VSS&tailLines=1"

      Actual results:

      • Kubelet starts successfully without any validation errors
      • Log query endpoints are accessible and return results
      • No error message about NodeLogQuery feature gate requirement
      • Both Linux and Windows nodes allow log querying despite NodeLogQuery not being explicitly enabled

      Below the effective kubelet configuration in a Windows worker node obtained with `oc get --raw "/api/v1/nodes/<node-name>/proxy/configz" | jq . `

      The `enableSystemLogQuery` is set, but the `NodeLogQuery` is missing from the `featureGates`, which is inconsistent with upstream documentation and may lead to misunderstanding.
       

      {
        "kubeletconfig": {
          "enableServer": true,
          "podLogsDir": "/var/log/pods",
          "syncFrequency": "1m0s",
          "fileCheckFrequency": "20s",
          "httpCheckFrequency": "20s",
          "address": "0.0.0.0",
          "port": 10250,
          "rotateCertificates": true,
          "serverTLSBootstrap": true,
          "authentication": {
            "x509": {
              "clientCAFile": "C:\\k\\kubelet-ca.crt"
            },
            "webhook": {
              "enabled": true,
              "cacheTTL": "2m0s"
            },
            "anonymous": {
              "enabled": false
            }
          },
          "authorization": {
            "mode": "Webhook",
            "webhook": {
              "cacheAuthorizedTTL": "5m0s",
              "cacheUnauthorizedTTL": "30s"
            }
          },
          "registryPullQPS": 5,
          "registryBurst": 10,
          "eventRecordQPS": 50,
          "eventBurst": 100,
          "enableDebuggingHandlers": true,
          "healthzPort": 10248,
          "healthzBindAddress": "127.0.0.1",
          "oomScoreAdj": -999,
          "clusterDomain": "cluster.local",
          "clusterDNS": [
            "172.30.0.10"
          ],
          "streamingConnectionIdleTimeout": "4h0m0s",
          "nodeStatusUpdateFrequency": "10s",
          "nodeStatusReportFrequency": "5m0s",
          "nodeLeaseDurationSeconds": 40,
          "imageMinimumGCAge": "2m0s",
          "imageMaximumGCAge": "0s",
          "imageGCHighThresholdPercent": 85,
          "imageGCLowThresholdPercent": 80,
          "volumeStatsAggPeriod": "1m0s",
          "cgroupsPerQOS": false,
          "cgroupDriver": "cgroupfs",
          "cpuManagerPolicy": "none",
          "cpuManagerReconcilePeriod": "10s",
          "memoryManagerPolicy": "None",
          "topologyManagerPolicy": "none",
          "topologyManagerScope": "container",
          "runtimeRequestTimeout": "10m0s",
          "hairpinMode": "promiscuous-bridge",
          "maxPods": 250,
          "podPidsLimit": -1,
          "resolvConf": "",
          "cpuCFSQuota": true,
          "cpuCFSQuotaPeriod": "100ms",
          "nodeStatusMaxImages": 50,
          "maxOpenFiles": 1000000,
          "contentType": "application/vnd.kubernetes.protobuf",
          "kubeAPIQPS": 50,
          "kubeAPIBurst": 100,
          "serializeImagePulls": false,
          "evictionHard": {
            "imagefs.available": "15%",
            "nodefs.available": "10%"
          },
          "evictionPressureTransitionPeriod": "5m0s",
          "enableControllerAttachDetach": true,
          "makeIPTablesUtilChains": true,
          "iptablesMasqueradeBit": 14,
          "iptablesDropBit": 15,
          "featureGates": {
            "RotateKubeletServerCertificate": true
          },
          "failSwapOn": true,
          "memorySwap": {},
          "containerLogMaxSize": "50Mi",
          "containerLogMaxFiles": 5,
          "containerLogMaxWorkers": 1,
          "containerLogMonitorInterval": "10s",
          "configMapAndSecretChangeDetectionStrategy": "Watch",
          "systemReserved": {
            "cpu": "500m",
            "ephemeral-storage": "1Gi",
            "memory": "2Gi"
          },
          "enforceNodeAllocatable": [
            "none"
          ],
          "volumePluginDir": "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/",
          "logging": {
            "format": "text",
            "flushFrequency": "5s",
            "verbosity": 2,
            "options": {
              "text": {
                "infoBufferSize": "0"
              },
              "json": {
                "infoBufferSize": "0"
              }
            }
          },
          "enableSystemLogHandler": true,
          "enableSystemLogQuery": true,
          "shutdownGracePeriod": "0s",
          "shutdownGracePeriodCriticalPods": "0s",
          "crashLoopBackOff": {},
          "enableProfilingHandler": true,
          "enableDebugFlagsHandler": true,
          "seccompDefault": false,
          "memoryThrottlingFactor": 0.9,
          "registerWithTaints": [
            {
              "key": "os",
              "value": "Windows",
              "effect": "NoSchedule"
            }
          ],
          "registerNode": true,
          "localStorageCapacityIsolation": true,
          "containerRuntimeEndpoint": "npipe://./pipe/containerd-containerd",
          "failCgroupV1": false
        }
      }
      
      

       

      Expected results:

      • Kubelet should fail to start with validation error when NodeLogQuery is not explicitly set to true
      • Should display error: "invalid configuration: NodeLogQuery feature gate is required for enableSystemLogQuery"
      • Log query endpoints should be inaccessible
      • Behavior should be consistent with explicit NodeLogQuery: false configuration

      Additional info:

              jvaldes@redhat.com Jose Valdes
              jvaldes@redhat.com Jose Valdes
              None
              None
              Aharon Rasouli Aharon Rasouli
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: