Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-4445

MCP stuck in pasued state because of error "kubeletconfig custom-kubeletconfig is not subset of rendered"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • 4.10.z
    • Compliance Operator
    • None
    • ?
    • None
    • 2
    • CMP Sprint 56, CMP Sprint 57, CMP Sprint 58
    • 3
    • False
    • Hide

      None

      Show
      None
    • Hide
      Cause: System reserved parameters are not rendered in kubelet configuration files.

      Consequence: The Compliance Operator will wait to unpause the machine configuration pool until the configuration is fully applied, resulting in a deadlock.

      Fix: Upgrade to the latest version of the compliance operator.

      Result: The compliance operator omits system reserved parameters during machine configuration pool evaluation.
      Show
      Cause: System reserved parameters are not rendered in kubelet configuration files. Consequence: The Compliance Operator will wait to unpause the machine configuration pool until the configuration is fully applied, resulting in a deadlock. Fix: Upgrade to the latest version of the compliance operator. Result: The compliance operator omits system reserved parameters during machine configuration pool evaluation.
    • Bug Fix

      Description of problem: While setting systemReserved/memory option using KubeletConfig object, and this option didn’t present in the generated kubelet machine config object.

      The KubeletConfig they are using is:

      ~~~
      [kni@sai-provisioner-cp-2 ~]$ oc get kubeletconfig -o yaml
      apiVersion: v1
      items:

      • apiVersion: machineconfiguration.openshift.io/v1
        kind: KubeletConfig
        metadata:
        creationTimestamp: "2022-11-18T12:23:15Z"
        finalizers:
      • 99-worker-generated-kubelet
      • 99-master-generated-kubelet
        generation: 1
        name: custom-kubeletconfig
        resourceVersion: "563342"
        uid: 3a08ed33-79b3-4198-87ff-a835af75da07
        spec:
        kubeletConfig:
        cpuManagerPolicy: static
        cpuManagerReconcilePeriod: 5s
        systemReserved:
        memory: 2Gi
        machineConfigPoolSelector:
        matchLabels:
        machineconfiguration.openshift.io/mco-built-in: ""

      and then machine config is

      ~~~
      {
      "kind": "KubeletConfiguration",
      "apiVersion": "kubelet.config.k8s.io/v1beta1",
      "staticPodPath": "/etc/kubernetes/manifests",
      "syncFrequency": "0s",
      "fileCheckFrequency": "0s",
      "httpCheckFrequency": "0s",
      "tlsCipherSuites": [
      "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
      "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
      "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
      "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
      "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
      "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
      ],
      "tlsMinVersion": "VersionTLS12",
      "rotateCertificates": true,
      "serverTLSBootstrap": true,
      "authentication": {
      "x509":

      { "clientCAFile": "/etc/kubernetes/kubelet-ca.crt" }

      ,
      "webhook":

      { "cacheTTL": "0s" }

      ,
      "anonymous":

      { "enabled": false }

      },
      "authorization": {
      "webhook":

      { "cacheAuthorizedTTL": "0s", "cacheUnauthorizedTTL": "0s" }

      },
      "clusterDomain": "cluster.local",
      "clusterDNS": [
      "172.30.0.10"
      ],
      "streamingConnectionIdleTimeout": "0s",
      "nodeStatusUpdateFrequency": "0s",
      "nodeStatusReportFrequency": "0s",
      "imageMinimumGCAge": "0s",
      "volumeStatsAggPeriod": "0s",
      "systemCgroups": "/system.slice",
      "cgroupRoot": "/",
      "cgroupDriver": "systemd",
      "cpuManagerPolicy": "static",
      "cpuManagerReconcilePeriod": "5s",
      "runtimeRequestTimeout": "0s",
      "maxPods": 250,
      "kubeAPIQPS": 50,
      "kubeAPIBurst": 100,
      "serializeImagePulls": false,
      "evictionPressureTransitionPeriod": "0s",
      "featureGates":

      { "APIPriorityAndFairness": true, "CSIMigrationAWS": false, "CSIMigrationAzureDisk": false, "CSIMigrationAzureFile": false, "CSIMigrationGCE": false, "CSIMigrationOpenStack": false, "CSIMigrationvSphere": false, "DownwardAPIHugePages": true, "LegacyNodeRoleBehavior": false, "NodeDisruptionExclusion": true, "PodSecurity": true, "RotateKubeletServerCertificate": true, "ServiceNodeExclusion": true, "SupportPodPidsLimit": true }

      ,
      "memorySwap": {},
      "containerLogMaxSize": "50Mi",
      "systemReserved":

      { "ephemeral-storage": "1Gi" }

      ,
      "logging": {
      "flushFrequency": 0,
      "verbosity": 0,
      "options": {
      "json":

      { "infoBufferSize": "0" }

      }
      },
      "shutdownGracePeriod": "0s",
      "shutdownGracePeriodCriticalPods": "0s"
      }
      ~~~
      due to this all machine config pool are paused.

      Compliance operator scan logs :
      ~~~
      {"level":"info","ts":"2022-11-24T09:03:44.400Z","logger":"suitectrl","msg":"KubeletConfig render diff:","Request.Namespace":"openshift-compliance","Request.Name":"idirect-worker-rhcos4-auto-apply-scan","MachineConfigPool.Name":"worker","

      {"level":"info","ts":"2022-11-24T09:03:44.400Z","logger":"suitectrl","msg":"KubeletConfig render diff:","Request.Namespace":"openshift-compliance","Request.Name":"idirect-worker-rhcos4-auto-apply-scan","MachineConfigPool.Name":"worker","Diff":"kubeletconfig custom-kubeletconfig is not subset of rendered MC 99-worker-generated-kubelet, diff: [[Path: /systemReserved/memory Expected: 2Gi Got: NOT FOUND]]"}

      , diff: [[Path: /systemReserved/memory Expected: 2Gi Got: NOT FOUND]]"}
      ~~~

              wenshen@redhat.com Vincent Shen
              rhn-support-agawand Asmita Gawand
              Xiaojie Yuan Xiaojie Yuan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: