Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-56446

MCP is reported degraded in the upgrade from OCP 4.15.44 to 4.16.39 with error "Marking Degraded due to: "unexpected on-disk state validating against rendered-worker-XXXXX: content mismatch for file \"/etc/systemd/system/kubelet-dependencies.target\""

XMLWordPrintable

    • Quality / Stability / Reliability
    • True
    • Hide

      This will block upgrades of all clusters on 4.16, 4.17, 4.18 with RHEL worker nodes. 

      Show
      This will block upgrades of all clusters on 4.16, 4.17, 4.18 with RHEL worker nodes. 
    • None
    • None
    • Yes
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      Description of problem:-
      
      1. There is a issue during OCP 4.15.44 -> 4.16.39 upgrade.
      
      2. MCP is degraded on worker
      ---
      $ oc get mcp
      NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
      master   rendered-master-14d387d2bd42b09d5752b5c4248eb068   True      False      False      3              3                   3                     0                      1y
      worker   rendered-worker-a97985cbd6ac7ef1e229e760887c2782   False     True       True       3              0                   0                     1                      1y
      ---
      
      4. MCP worker status.conditions
      ---
      ...
      {
        "lastTransitionTime": "2025-05-14T16:51:29Z",
        "message": "Node ocp-worker3 is reporting: \"unexpected on-disk state validating against rendered-worker-a97985cbd6ac7ef1e229e760887c2782: content mismatch for file \\\"/etc/systemd/system/kubelet-dependencies.target\\\"\"",
        "reason": "1 nodes are reporting degraded status on sync",
        "status": "True",
        "type": "NodeDegraded"
      }
      ---
      
      5. logs machine-config-daemon node worker ocp-worker3
      ---
      2025-05-15T14:07:54.381041434-03:00 E0515 14:07:54.380906   58750 on_disk_validation.go:251] content mismatch for file "/etc/systemd/system/kubelet-dependencies.target" (-want +got):
      2025-05-15T14:07:54.381041434-03:00   []uint8(
      2025-05-15T14:07:54.381041434-03:00       """
      2025-05-15T14:07:54.381041434-03:00       ... // 2 identical lines
      2025-05-15T14:07:54.381041434-03:00       Documentation=https://github.com/openshift/machine-config-operator/
      2025-05-15T14:07:54.381041434-03:00       Requires=basic.target network-online.target
      2025-05-15T14:07:54.381041434-03:00 -     Wants=NetworkManager-wait-online.service crio-wipe.service
      2025-05-15T14:07:54.381041434-03:00 +     Wants=NetworkManager-wait-online.service crio-wipe.service disable-mglru.service
      2025-05-15T14:07:54.381041434-03:00       Wants=rpc-statd.service
      2025-05-15T14:07:54.381041434-03:00       """
      2025-05-15T14:07:54.381041434-03:00   )
      2025-05-15T14:07:54.381041434-03:00 E0515 14:07:54.380980   58750 writer.go:226] Marking Degraded due to: "unexpected on-disk state validating against rendered-worker-a97985cbd6ac7ef1e229e760887c2782: content mismatch for file \"/etc/systemd/system/kubelet-dependencies.target\"" 
      
      

      Version-Release number of selected component (if applicable):

          

      How reproducible:

          

      Steps to Reproduce:

          1.
          2.
          3.
          

      Actual results:

          

      Expected results:

          

      Additional info:

          

              jlebon1@redhat.com Jonathan Lebon
              rhn-support-macastil Mario Castillo
              None
              None
              Sergio Regidor de la Rosa Sergio Regidor de la Rosa
              None
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated: