Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-18640

Cluster fails to install at day-0 with PerformanceProfile

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done-Errata
    • Major
    • 4.14.z
    • 4.14
    • Node Tuning Operator
    • None
    • No
    • CNF Compute Sprint 251
    • 1
    • False
    • Hide

      None

      Show
      None
    • * Currently, applying a performance profile at day-0 is not supported.
    • Known Issue
    • Done
    • Hide
      2024-03-20: all fixes merged in the 4.14 branch
      5/3 : https://github.com/openshift/cluster-node-tuning-operator/pull/963 should be the last fix
      Show
      2024-03-20: all fixes merged in the 4.14 branch 5/3 : https://github.com/openshift/cluster-node-tuning-operator/pull/963 should be the last fix

    Description

      Description of problem:

      Picked up 4.14-ec-4 (which uses cgroups v1 as default) and trying to create a cluster with following PerformanceProfile (and corresponding mcp) by placing them in the manifests folder, 

       
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: clusterbotpp
spec:
  cpu:
    isolated: "1-3"
    reserved: "0"
  realTimeKernel:
    enabled: false
  nodeSelector:
    node-role.kubernetes.io/worker: ""
  machineConfigPoolSelector:
    pools.operator.machineconfiguration.openshift.io/worker: ""

      and, 

      apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: worker 
spec:
  machineConfigSelector:
    matchLabels:
      machineconfiguration.openshift.io/role: worker
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker: ""

      The cluster often fails to install because bootkube spends a lot of time chasing this error, 

       
Sep 06 18:32:43 ip-10-0-145-107 bootkube.sh[4925]: Created "clusterbotpp_kubeletconfig.yaml" kubeletconfigs.v1.machineconfiguration.openshift.io/performance-clusterbotpp -n
Sep 06 18:32:43 ip-10-0-145-107 bootkube.sh[4925]: Failed to update status for the "clusterbotpp_kubeletconfig.yaml" kubeletconfigs.v1.machineconfiguration.openshift.io/performance-clusterbotpp -n : Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "performance-clusterbotpp": StorageError: invalid object, Code: 4, Key: /kubernetes.io/machineconfiguration.openshift.io/kubeletconfigs/performance-clusterbotpp, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 11f98d74-af1b-4a4c-9692-6dce56ee5cd9, UID in object meta:
Sep 06 18:32:43 ip-10-0-145-107 bootkube.sh[4925]: [#1717] failed to create some manifests:
Sep 06 18:32:43 ip-10-0-145-107 bootkube.sh[4925]: "clusterbotpp_kubeletconfig.yaml": failed to update status for kubeletconfigs.v1.machineconfiguration.openshift.io/performance-clusterbotpp -n : Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "performance-clusterbotpp": StorageError: invalid object, Code: 4, Key: /kubernetes.io/machineconfiguration.openshift.io/kubeletconfigs/performance-clusterbotpp, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 11f98d74-af1b-4a4c-9692-6dce56ee5cd9, UID in object meta:
Sep 06 18:32:43 ip-10-0-145-107 bootkube.sh[4925]: Created "clusterbotpp_kubeletconfig.yaml" kubeletconfigs.v1.machineconfiguration.openshift.io/performance-clusterbotpp -n
Sep 06 18:32:43 ip-10-0-145-107 bootkube.sh[4925]: Failed to update status for the "clusterbotpp_kubeletconfig.yaml" kubeletconfigs.v1.machineconfiguration.openshift.io/performance-clusterbotpp -n : Operation cannot be fulfilled on kubeletconfigs.machineconfiguration.openshift.io "performance-clusterbotpp": StorageError: invalid object, Code: 4, Key: /kubernetes.io/machineconfiguration.openshift.io/kubeletconfigs/performance-clusterbotpp, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 597dfcf3-012d-4730-912a-78efabb920ba, UID in object meta:

      This leads to worker nodes not getting ready in time, which leads to installer marking the cluster installation failed. Ironically, even after the cluster installer returns with failure, if you wait long enough (sometimes) I have observed the cluster eventually reconciles and the worker nodes get provisioned. 

      I am attaching the installation logs from one such run with this issue. 

       

       

      Version-Release number of selected component (if applicable):

      4.14

      How reproducible:

      Often

      Steps to Reproduce:

      1. Try to install new cluster by placing PeformanceProfile in the manifests folder
2.
3.

      Actual results:

      Cluster installation failed.

      Expected results:

      Cluster installation should succeed.

      Additional info:

      Also, I didn't observe this occurring in 4.13.9. 

      Attachments

        Issue Links

          blocks

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-17859 Avoid extra reboot with cgroups v2 at day-0 for PerformanceProfile

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed
          depends on

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-29752 day-0 with PerformanceProfile manifest renderer uses invalid uid

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-29751 day-0 with PerformanceProfile manifest renderer uses invalid uid

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Verified

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-25116 Cluster fails to install at day-0 with PerformanceProfile

          • Undefined - Priority not specified.
          • Closed
          is depended on by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-25115 [4.14] Cluster fails to install at day-0 with PerformanceProfile

          • Undefined - Priority not specified.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-25116 Cluster fails to install at day-0 with PerformanceProfile

          • Undefined - Priority not specified.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-19352 Node in NotReady state as unified_cgroup_hierarchy=1 are set

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          links to

          Web Link https://github.com/openshift/cluster-node-tuning-operator/pull/963

          GitHub openshift/cluster-node-tuning-operator#854: OCPBUGS-18640: Fix Racing Machine Configs and add Day 0 Support

          GitHub openshift/cluster-node-tuning-operator#871: OCPBUGS-18640: Fix Racing Machine Configs and add Day 0 Support (#854)

          GitHub openshift/cluster-node-tuning-operator#989: OCPBUGS-18640: [release-4.14][manual] backport performance profile owner reference ehnancements

          Web Link PR 935

          Web Link RHBA-2024:1458 OpenShift Container Platform 4.14.z bug fix update

          Activity

            People

              fromani@redhat.com Francesco Romani
              harpatil@redhat.com Harshal Patil
              Shereen Haj Shereen Haj
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: