Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-44955

When using 'user_friendly_names' in multipath.conf, the ostree-prepare-root service fails

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Critical Critical
    • None
    • 4.12.z, 4.14.z
    • RHCOS
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 1
    • Important
    • None
    • None
    • None
    • None
    • None
    • Customer Escalated
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          When using 'user_friendly_names' in multipath.conf, the ostree-prepare-root service fails with `[    3.848167] ostree-prepare-root[1056]: ostree-prepare-root: Couldn't find specified OSTree root '/sysroot//ostree/boot.0/rhcos/935213ae673b3d922f7477bc290c3ced6d6e966d2d9b9e0b2a43d5e66034cb73/0 ': No such file or directory`

      Version-Release number of selected component (if applicable):

      o OpenShift 4.12
      o CoreOS 412.86
      AND
      o OpenShift 4.14
      o CoreOS 414.92

      How reproducible:

          Every time

      Steps to Reproduce:

      o Root multipath device (IBM,3303 NVDISK) has 2x sub paths and it is connected through 2 vio adapters
      o Non-root multipath devices (HITACHI OPEN-V) also have 2x paths and they are connected through IBM virtual FC adapters
      
      
      Isuse:
      ------
      o While booting up the system with above root, non-root LUNs, CoreOS fails to setup OSTree and logs following errors:
      
      
      [  OK  ] Finished dracut pre-mount hook.
      [    3.618321] systemd[1]: Starting File System Check on /dev/disk/by-label/dm-mpath-root...
      	 Starting File System Check…disk/by-label/dm-mpath-root...
      [    3.628471] systemd-fsck[1041]: /usr/sbin/fsck.xfs: XFS file system.
      [  OK  ] Finished File System Check…v/disk/by-label/dm-mpath-root.
      [    3.629044] systemd[1]: Finished File System Check on /dev/disk/by-label/dm-mpath-root.
      [    3.629899] systemd[1]: Mounting /sysroot...
      	 Mounting /sysroot...
      [    3.791507] SGI XFS with ACLs, security attributes, scrub, quota, no debug enabled
      [    3.793571] XFS (dm-10): Mounting V5 Filesystem
      [    3.822919] XFS (dm-10): Starting recovery (logdev: internal)
      [    3.838027] XFS (dm-10): Ending recovery (logdev: internal)
      [    3.844288] xfs filesystem being mounted at /sysroot supports timestamps until 2038 (0x7fffffff)
      [  OK  ] Mounted /sysroot.
      [    3.845241] systemd[1]: Mounted /sysroot.
      [    3.846177] systemd[1]: Starting OSTree Prepare OS/...
      	 Starting OSTree Prepare OS/...
      [    3.847337] ostree-prepare-root[1056]: preparing sysroot at /sysroot
      [    3.848167] ostree-prepare-root[1056]: ostree-prepare-root: Couldn't find specified OSTree root '/sysroot//ostree/boot.0/rhcos/935213ae673b3d922f7477bc290c3ced6d6e966d2d9b9e0b2a43d5e66034cb73/0
      ': No such file or directory
      [    3.838494] ostree-prepare-root[1056]: ostree-prepare-root: Couldn't find specified OSTree root '/sysroot//ostree/boot.0/rhcos/935213ae673b3d922f7477bc290c3ced6d6e966d2d9b9e0b2a43d5e66034cb73/0
      ': No such file or directory
      [    3.848345] systemd[1]: ostree-prepare-root.service: Main process exited, code=exited, status=1/FAILURE
      [    3.848473] systemd[1]: ostree-prepare-root.service: Failed with result 'exit-code'.
      [    3.848647] systemd[1]: Failed to start OSTree Prepare OS/.
      [FAILED] Failed to start OSTree Prepare OS/.
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      See 'systemctl status ostree-prepare-root.service' for details.
      [...]
      
      
      o As per the customer, if they disable the IBM virtual FC adapters and do not map HITACHI LUNs then above failure does not occur and system boots up fine.
      
      Console logs:
      
      
      0050-console_scelepar04804.txt			
      	^--- This file contains the logs when both IBM Virtual FC adapters were enabled
      	     HITACHI OPEN-V LUNs were assigned and boot process failed.
      
      0060-console_boot_full_scelepar04804.txt
      	^--- After above failed boot attempt, the FC adapters were disabled, there were
      	     no HITACHI OPEN-V LUNs and system came up fine.
      
      
      The boot from SAN configurations with identical setup are known to work fine with RHEL 8.x and 9.x systems. In this case, the steps to setup CoreOS specific OSTree root are failing.     

      Actual results:

          RHCOS node fails to boot and drops into emergency mode

      Expected results:

          RHCOS node boots properly from san

      Additional info:

          

              rhn-support-jmarrero Joseph Marrero Corchado
              rhn-support-emahoney Evan Mahoney
              None
              None
              Michael Nguyen Michael Nguyen
              None
              Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: