Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-63310

NVMe disk by-id rename breaks LSO/ODF during EUS upgrade to 4.20

XMLWordPrintable

    • Quality / Stability / Reliability
    • True
    • Hide

      The issue blocks the EUS upgrade path. Added the Blocker to get OCP awareness 

      Show
      The issue blocks the EUS upgrade path. Added the Blocker to get OCP awareness 
    • None
    • Critical
    • Yes
    • x86_64
    • None
    • None
    • Rejected
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          NVMe id not persisted from previous version after EUS upgrade

      Version-Release number of selected component (if applicable):

      Bare-metal: SCALELAB-4482
      
      Starting environment:
      OCP 4.18.24
      LSO 4.18.0-202509240837
      ODF 4.18.11
      CNV 4.18.17
      
      Intermediate environment during upgrade:
      OCP 4.19.14
      LSO 4.19.0-202510071855
      ODF 4.19.6
      CNV 4.19.6 (control-plane only)
      
      Ending environment, upgrade incomplete:
      OCP 4.20.0-rc.3
      LSO 4.19.0-202510071855
      ODF 4.19.6
      CNV 4.20.0.rhel9-207

      How reproducible:

         unknown

      Steps to Reproduce:

          1. Install OCP/LSO/ODF/CNV 4.18
          2. Perform EUS upgrade
          3. Unpause worker MCP as last step of upgrade
          

      Actual results:

          NVMe links renamed upon worker upgrade from 4.18 to 4.20

      Expected results:

          NVMe links persist from 4.18 to 4.20

      Additional info:

      Node not fully draining, OSD is still running
      d41-h03-000-r660
          Red Hat Enterprise Linux CoreOS 418.94.202509100653-0
          Kernel Version: 5.14.0-427.87.1.el9_4.x86_64
          Container Runtime: cri-o://1.31.12-3.rhaos4.18.gitdc59c78.el9
          Kubelet Version: v1.31.12
      
      Namespace: openshift-storage
      Multiple OSD pods are not starting
      
      Alert:
      Pod: rook-ceph-osd-5-547576567f-27mg6
      Generated from kubelet on d40-h20-000-r660
      MapVolume.EvalHostSymlinks failed for volume "local-pv-6daced2" : lstat /dev/disk/by-id/nvme-eui.36435230548269370025384100000003: no such file or directory
      
      Node: d40-h20-000-r660 - amd64
          Red Hat Enterprise Linux CoreOS 9.6.20250925-0 (Plow)
          Kernel Version: 5.14.0-570.49.1.el9_6.x86_64
          Container Runtime: cri-o://1.33.4-4.rhaos4.20.gitc8e7df2.el9
          Kubelet Version: v1.33.5
      
      sh-5.1# nvme list
      Node                  Generic               SN                   Model                                    Namespace  Usage                      Format           FW Rev  
      --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
      /dev/nvme0n1          /dev/ng0n1            S6CRNA0T826937       Dell Ent NVMe v2 AGN MU U.2 3.2TB        0x1          3.20  TB /   3.20  TB    512   B +  0 B   2.3.0   
      
      sh-5.1# ls /dev/disk/by-path -l | grep nvme
      lrwxrwxrwx. 1 root root 13 Oct 17 16:46 pci-0000:62:00.0-nvme-1 -> ../../nvme0n1
      
      sh-5.1# ls /dev/disk/by-id -l | grep nvme
      lrwxrwxrwx. 1 root root 13 Oct 17 16:46 nvme-Dell_Ent_NVMe_v2_AGN_MU_U.2_3.2TB_S6CRNA0T826937 -> ../../nvme0n1
      lrwxrwxrwx. 1 root root 13 Oct 17 16:46 nvme-Dell_Ent_NVMe_v2_AGN_MU_U.2_3.2TB_S6CRNA0T826937_1 -> ../../nvme0n1
      lrwxrwxrwx. 1 root root 13 Oct 17 16:46 nvme-nvme.144d-533643524e413054383236393337-44656c6c20456e74204e564d652076322041474e204d5520552e3220332e325442-00000001 -> ../../nvme0n1
      
      
      
      
      Additional Note: this did not occur when upgrading from 4.19 to 4.20.

       

              Unassigned Unassigned
              rhn-support-sbennert Sarah Bennert
              None
              None
              Michael Nguyen Michael Nguyen
              None
              Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

                Created:
                Updated: