Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-45111

Provisioning 4.17.3 OCP cluster on bare metal using Agent Based Installer is reporting MCO degraded just after the installation

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.17.z
    • Node / Kubelet
    • None
    • Moderate
    • None
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      We are provisioning 4.17.3 OCP cluster on bare metal using Agent Based Installer. After installation, MCO is getting degraded with content mismatch in master nodes.
      
      Here is the nodes information:
      > $ oc get nodes -owide
      NAME                     STATUS                     ROLES                         AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                                KERNEL-VERSION                 CONTAINER-RUNTIME
      c2-esx02.racka05.local   Ready                      control-plane,master,worker   5d5h   v1.30.5   20.13.172.102   <none>        Red Hat Enterprise Linux CoreOS 417.94.202410211619-0   5.14.0-427.42.1.el9_4.x86_64   cri-o://1.30.6-6.rhaos4.17.git6ac6e96.el9
      c2-esx03.racka05.local   Ready                      control-plane,master,worker   5d5h   v1.30.5   20.13.172.103   <none>        Red Hat Enterprise Linux CoreOS 417.94.202410211619-0   5.14.0-427.42.1.el9_4.x86_64   cri-o://1.30.6-6.rhaos4.17.git6ac6e96.el9
      c2-esx04.racka05.local   Ready,SchedulingDisabled   control-plane,master,worker   5d5h   v1.30.5   20.13.172.104   <none>        Red Hat Enterprise Linux CoreOS 417.94.202410211619-0   5.14.0-427.42.1.el9_4.x86_64   cri-o://1.30.6-6.rhaos4.17.git6ac6e96.el9
      
      
      For the node without errors(c2-esx02.racka05.local and c2-esx03.racka05.local):
          currentConfig: rendered-master-2a545301fdfe36a460760fc8734f7aaf
          desiredConfig: rendered-master-2a545301fdfe36a460760fc8734f7aaf
      
      For the node with errors(c2-esx04.racka05.local):
          currentConfig: rendered-master-d8f7478567e27665f3128a439d9c5b50
          desiredConfig: rendered-master-2a545301fdfe36a460760fc8734f7aaf
      
      
      
      > $ OLD_RENDERED=rendered-master-d8f7478567e27665f3128a439d9c5b50
      > $ NEW_RENDERED=rendered-master-2a545301fdfe36a460760fc8734f7aaf
      > $ diff -NrU 5 <(oc get mc ${OLD_RENDERED} -o json) <(oc get mc ${NEW_RENDERED} -o json)
      --- /proc/self/fd/11    2024-11-27 01:57:06.609339330 +0000
      +++ /proc/self/fd/18    2024-11-27 01:57:06.609339330 +0000
      @@ -4,25 +4,25 @@
           "metadata": {
               "annotations": {
                   "machineconfiguration.openshift.io/generated-by-controller-version": "6308e62a4edabb857def36e2ffdaad1f92cc4734",
                   "machineconfiguration.openshift.io/release-image-version": "4.17.3"
               },
      -        "creationTimestamp": "2024-11-21T19:58:07Z",
      +        "creationTimestamp": "2024-11-21T19:58:13Z",
               "generation": 1,
      -        "name": "rendered-master-d8f7478567e27665f3128a439d9c5b50",
      +        "name": "rendered-master-2a545301fdfe36a460760fc8734f7aaf",
               "ownerReferences": [
                   {
                       "apiVersion": "machineconfiguration.openshift.io/v1",
                       "blockOwnerDeletion": true,
                       "controller": true,
                       "kind": "MachineConfigPool",
                       "name": "master",
                       "uid": "27cd9183-3ee9-4014-8b35-967ebb7da815"
                   }
               ],
      -        "resourceVersion": "9330",
      -        "uid": "c7f2a699-644c-4237-80c1-e6da5a089b6e"
      +        "resourceVersion": "9466",
      +        "uid": "5b9a97f1-cc98-4f01-b769-06fcce327f75"
           },
           "spec": {
               "baseOSExtensionsContainerImage": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:669a24aafb6b665fc36549e3beab648b663ab7db7dce308ad0e0ef7affe94130",
               "config": {
                   "ignition": {
      @@ -281,11 +281,12 @@
                               "overwrite": true,
                               "path": "/etc/kubernetes/manifests/keepalived.yaml"
                           },
                           {
                               "contents": {
      -                            "source": "data:,NODE_SIZING_ENABLED%3Dfalse%0ASYSTEM_RESERVED_MEMORY%3D1Gi%0ASYSTEM_RESERVED_CPU%3D500m%0ASYSTEM_RESERVED_ES%3D1Gi"
      +                            "compression": "",
      +                            "source": "data:text/plain;charset=utf-8;base64,Tk9ERV9TSVpJTkdfRU5BQkxFRD10cnVlClNZU1RFTV9SRVNFUlZFRF9NRU1PUlk9MUdpClNZU1RFTV9SRVNFUlZFRF9DUFU9NTAwbQpTWVNURU1fUkVTRVJWRURfRVM9MUdpCg=="
                               },
                               "mode": 420,
                               "overwrite": true,
                               "path": "/etc/node-sizing-enabled.env"
                           },
      
      The oc client binary and openshift-install binary. oc: v4.17.2 and openshift-install: v4.17.3. 

      Additional info:

          

              svanka@redhat.com Sai Ramesh Vanka
              rhn-support-skohli Shubh Kohli
              Manoj Hans Manoj Hans
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated: