Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-100907

nm-initrd-generator fails to create connections in RHEL 10 kdump initramfs (NVMe/TCP)

Linking RHIVOS CVEs to...Migration: Automation ...SWIFT: POC ConversionSync from "Extern...XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • rhel-10.1
    • rhel-10.1
    • kdump-utils
    • kdump-utils-1.0.54-4.el10
    • No
    • Important
    • rhel-kernel-debug
    • 24
    • 26
    • 0
    • False
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • Unspecified
    • Unspecified
    • Unspecified
    • x86_64
    • None

      While RHEL 9 nightly works, I'm facing issues making kdump to NVMe/TCP (boot/NBFT) work on RHEL 10 nightly. I tracked it down to nm-initrd-generator as called by dracut from lib/nm-lib.sh not able to create connection files from the given commandline that is valid:

      [   36.877074] dracut-cmdline[318]: //lib/nm-lib.sh@15(nm_generate_connections): /usr/libexec/nm-initrd-generator -- rd.debug rd.neednet=1 rd.neednet ip=172.18.50.21:::24::nbft0:none ip=172.18.60.22:::24::nbft1:none ifname=nbft0:40:a6:b7:c0:8a:c8 ifname=nbft1:40:a6:b7:c0:8a:c9 rd.lvm.lv=rhel_rhel-storage-108/root rd.nvmf.hostnqn=nqn.2014-08.org.nvmexpress:uuid:12345678-aaaa-bbbb-cccc-123456789012 rd.nvmf.hostid=78563412-aaaa-bbbb-cccc-123456789012 rd.neednet=1 elfcorehdr=0x17000000 'BOOT_IMAGE=(hd4,gpt2)/vmlinuz-6.12.0-98.el10.x86_64' ro resume=UUID=6992ac31-f419-422c-a4a2-836d3f49fe71 console=ttyS0,115200 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 acpi_no_memhotplug transparent_hugepage=never nokaslr hest_disable novmcoredd cma=0 hugetlb_cma=0 pcie_ports=compat disable_cpu_apicid=0 iTCO_wdt.pretimeout=0
      

      First issue is that /run/NetworkManager/system-connections/ doesn't exist, not sure if nm-initrd-generator is supposed to create that. However from the following cycle going over four locations it appears there was no file in either of them:

      [   37.476047] dracut-cmdline[318]: //lib/nm-lib.sh@27(nm_generate_connections): for i in /usr/lib/NetworkManager/system-connections/* /run/NetworkManager/system-connections/* /etc/NetworkManager/system-connections/* /etc/sysconfig/network-scripts/ifcfg-*
      [   37.500033] dracut-cmdline[318]: //lib/nm-lib.sh@31(nm_generate_connections): '[' -f '/usr/lib/NetworkManager/system-connections/*' ']'
      [   37.513030] dracut-cmdline[318]: //lib/nm-lib.sh@31(nm_generate_connections): continue
      [   37.521029] dracut-cmdline[318]: //lib/nm-lib.sh@27(nm_generate_connections): for i in /usr/lib/NetworkManager/system-connections/* /run/NetworkManager/system-connections/* /etc/NetworkManager/system-connections/* /etc/sysconfig/network-scripts/ifcfg-*
      [   37.545029] dracut-cmdline[318]: //lib/nm-lib.sh@31(nm_generate_connections): '[' -f '/run/NetworkManager/system-connections/*' ']'
      [   37.558033] dracut-cmdline[318]: //lib/nm-lib.sh@31(nm_generate_connections): continue
      [   37.566030] dracut-cmdline[318]: //lib/nm-lib.sh@27(nm_generate_connections): for i in /usr/lib/NetworkManager/system-connections/* /run/NetworkManager/system-connections/* /etc/NetworkManager/system-connections/* /etc/sysconfig/network-scripts/ifcfg-*
      [   37.590034] dracut-cmdline[318]: //lib/nm-lib.sh@31(nm_generate_connections): '[' -f '/etc/NetworkManager/system-connections/*' ']'
      [   37.603030] dracut-cmdline[318]: //lib/nm-lib.sh@31(nm_generate_connections): continue
      [   37.611030] dracut-cmdline[318]: //lib/nm-lib.sh@27(nm_generate_connections): for i in /usr/lib/NetworkManager/system-connections/* /run/NetworkManager/system-connections/* /etc/NetworkManager/system-connections/* /etc/sysconfig/network-scripts/ifcfg-*
      [   37.635247] dracut-cmdline[318]: //lib/nm-lib.sh@31(nm_generate_connections): '[' -f '/etc/sysconfig/network-scripts/ifcfg-*' ']'
      [   37.648056] dracut-cmdline[318]: //lib/nm-lib.sh@31(nm_generate_connections): continue
      

      That effectively meant networking was not brought up. I have added some more checks that verify this:

      [   37.272030] dracut-cmdline[318]: //lib/nm-lib.sh@25(nm_generate_connections): ls -al /run/NetworkManager/system-connections/
      [   37.346071] dracut-cmdline[538]: total 0
      [   37.352036] dracut-cmdline[538]: drwxr-xr-x 2 root root 40 Jun 27 11:17 .
      [   37.359030] dracut-cmdline[538]: drwxr-xr-x 4 root root 80 Jun 27 11:17 ..
      [   37.379038] dracut-cmdline[318]: //lib/nm-lib.sh@26(nm_generate_connections): cat '/run/NetworkManager/system-connections/*'
      [   37.453068] dracut-cmdline[539]: cat: '/run/NetworkManager/system-connections/*': No such file or directory
      

      Note that everything works as expected in a regular initrams. When I run the command from the host system, it works as expected:

      # /usr/libexec/nm-initrd-generator --stdout -- rd.debug rd.neednet=1 rd.neednet ip=172.18.50.21:::24::nbft0:none ip=172.18.60.22:::24::nbft1:none ifname=nbft0:40:a6:b7:c0:8a:c8 ifname=nbft1:40:a6:b7:c0:8a:c9 rd.lvm.lv=rhel_rhel-storage-108/root rd.nvmf.hostnqn=nqn.2014-08.org.nvmexpress:uuid:12345678-aaaa-bbbb-cccc-123456789012 rd.nvmf.hostid=78563412-aaaa-bbbb-cccc-123456789012 rd.neednet=1 elfcorehdr=0x17000000 'BOOT_IMAGE=(hd4,gpt2)/vmlinuz-6.12.0-98.el10.x86_64' ro resume=UUID=6992ac31-f419-422c-a4a2-836d3f49fe71 console=ttyS0,115200 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 acpi_no_memhotplug transparent_hugepage=never nokaslr hest_disable novmcoredd cma=0 hugetlb_cma=0 pcie_ports=compat disable_cpu_apicid=0 iTCO_wdt.pretimeout=0
      
      *** Configuration '15-carrier-timeout.conf' ***
      
      [device-15-carrier-timeout]
      match-device=*
      carrier-wait-timeout=10000
      
      [.config]
      enable=env:initrd
      
      
      *** Connection 'nbft0' ***
      
      [connection]
      id=nbft0
      uuid=f1542fac-425a-41d7-a663-257406947db5
      type=ethernet
      autoconnect-priority=-100
      autoconnect-retries=1
      interface-name=nbft0
      multi-connect=1
      wait-device-timeout=60000
      
      [ethernet]
      
      [ipv4]
      address1=172.18.50.21/24
      may-fail=false
      method=manual
      
      [ipv6]
      method=disabled
      
      [proxy]
      
      [user]
      org.freedesktop.NetworkManager.origin=nm-initrd-generator
      
      *** Connection 'nbft1' ***
      
      [connection]
      id=nbft1
      uuid=dbc26049-6c7f-41a1-9283-20dd903f64b2
      type=ethernet
      autoconnect-priority=-100
      autoconnect-retries=1
      interface-name=nbft1
      multi-connect=1
      wait-device-timeout=60000
      
      [ethernet]
      
      [ipv4]
      address1=172.18.60.22/24
      may-fail=false
      method=manual
      
      [ipv6]
      method=disabled
      
      [proxy]
      
      [user]
      org.freedesktop.NetworkManager.origin=nm-initrd-generator
      

      Not sure what's different in the kdump initramfs. Also note this is not relevant to my recent NBFT addition in NetworkManager, this is about the old way of parsing dracut-created kernel commandline.

      Console log with a couple of more prints and checks added in /usr/lib/dracut/modules.d/35network-manager/nm-lib.sh: console-debug.log

              coxu@redhat.com Coiby Xu
              tbzatek Tomáš Bžatek
              kdump team kdump team
              Xiaoying Yan Xiaoying Yan
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

                Created:
                Updated: