Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-22495

'infinite' value for 'dhcp-range' in dnsmasq triggers tear-down of interface by network-manager on openshift nodes

XMLWordPrintable

    • Important
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      'infinite' value for 'dhcp-range' in dnsmasq triggers tear-down of interface by network-manager on openshift
      

      Version-Release number of selected component (if applicable):

      4.7.55
      
      ### PREFACE
      ### Supported Exception for 4.7.55 (looking for RCA really and potential workaround)
      ### /PREFACE

       

      ---------------------------------

      We have case 03649974[1], where DNSMASQ was updated and caused issues. Verizon referenced another case [2] they thought was similar that resulted in a bug [3] that fixed that issue in the NetworkManager RPM provided in OCP 4.7.59 (customer is on OCP 4.7.55).

      However, I'm not convinced this is the same issue. To follow the current, we have to talk about how this is deployed.

      • Cluster is deployed using Openshift IPI from a bastion host
      • This bastion host has the dnsmasq service configured to handle IPs on servers
      • The bonds on the servers are initially created by machine-configs

      So we might start with a config like this on the bastion before deploying the cluster:

      $ less sosreport-bastion-2023-10-26-nrprllh/etc/dnsmasq.d/baremetal.conf
      # dhcp for baremetal network
      no-dhcp-interface=lo
      log-dhcp
      domain=kub2.rcmdva83.vzwops.com
      interface=baremetal
      dhcp-range=set:baremetal,69.100.89.0,static,336h
      dhcp-option=option:T1,12h
      dhcp-option=option:T2,24h
      dhcp-option=tag:baremetal,option:netmask,255.255.255.128
      dhcp-option=tag:baremetal,option:router,69.100.89.1
      dhcp-option=tag:baremetal,option:dns-server,10.215.0.6,10.215.0.7
      dhcp-option=tag:baremetal,option:ntp-server,69.100.89.4
      dhcp-option-force=tag:baremetal,option:mtu,9000# Bootstrap Node
      dhcp-host=52:54:00:*:*:*,69.100.89.115,set:baremetal# Master and Worker Nodes
      dhcp-host=88:e9:a4:06:fd:68,88:e9:a4:06:fd:69,master-0,69.100.89.10,set:baremetal
      dhcp-host=88:e9:a4:06:fd:3c,88:e9:a4:06:fd:3d,master-1,69.100.89.11,set:baremetal
      dhcp-host=88:e9:a4:1b:37:38,88:e9:a4:1b:37:39,master-2,69.100.89.12,set:baremetal
      dhcp-host=88:e9:a4:15:6b:4c,88:e9:a4:15:6b:4d,worker-20,69.100.89.33,set:baremetal 

      When the cluster is deployed, a machine-config lays down the bonds – this is what that machine-config might look like (only showing bit concerning bond0)

      $ oc get mc 11-worker-bonding -o yaml
      spec:
        config:
          ignition:
            version: 3.2.0
          storage:
            files:
            - contents:
                source: data:;base64,W2Nvbm5lY3Rpb25dCmlkPWJvbmQwCnR5cGU9Ym9uZAppbnRlcmZhY2UtbmFtZT1ib25kMAphdXRvY29ubmVjdD10cnVlCmNvbm5lY3Rpb24uYXV0b2Nvbm5lY3Qtc2xhdmVzPTEKCltldGhlcm5ldF0KbXR1PTkwMDAKCltib25kXQptb2RlPWFjdGl2ZS1iYWNrdXAKbWlpbW9uPTEwMApmYWlsX292ZXJfbWFjPWFjdGl2ZQpudW1fZ3JhdF9hcnA9MTAKCltpcHY0XQptZXRob2Q9YXV0bwpkaGNwLXRpbWVvdXQ9MjE0NzQ4MzY0NwoKW2lwdjZdCm1ldGhvZD1kaXNhYmxlZA==
              filesystem: root
              mode: 384
              path: /etc/NetworkManager/system-connections/bond0.nmconnection
            - contents:
                source: data:;base64,W2Nvbm5lY3Rpb25dCmlkPWVuczFmMAp0eXBlPWV0aGVybmV0CmludGVyZmFjZS1uYW1lPWVuczFmMAptYXN0ZXI9Ym9uZDAKc2xhdmUtdHlwZT1ib25kCmF1dG9jb25uZWN0PXRydWUKCltldGhlcm5ldF0KbXR1PTkwMDA=
              filesystem: root
              mode: 384
              path: /etc/NetworkManager/system-connections/ens1f0.nmconnection
            - contents:
                source: data:;base64,W2Nvbm5lY3Rpb25dCmlkPWVuczFmMQp0eXBlPWV0aGVybmV0CmludGVyZmFjZS1uYW1lPWVuczFmMQptYXN0ZXI9Ym9uZDAKc2xhdmUtdHlwZT1ib25kCmF1dG9jb25uZWN0PXRydWUKCltldGhlcm5ldF0KbXR1PTkwMDA=
              filesystem: root
              mode: 384
              path: /etc/NetworkManager/system-connections/ens1f1.nmconnection 
      $ oc get mc 00-worker -o yaml       
      - contents:           
          source: data:,%23!%2Fbin%2Fbash%0Aset%20-ex%20-o%20pipefail%0A%0Aif%20%5B%5B%20%22OVNKubernetes%22%20%3D%3D%20%22OVNKubernetes%22%20%26%26%20%22%24CONNECTION_ID%22%20%3D%3D%20%22Wired%20Connection%22%20%5D%5D%0Athen%0A%20%20%20%20%3E%262%20echo%20%22Refusing%20to%20modify%20default%20connection.%22%0A%20%20%20%20exit%200%0Afi%0A%0Aif%20%5B%20-z%20%24%7BDHCP4_IP_ADDRESS%3A-%7D%20%5D%0Athen%0A%20%20%20%20%3E%262%20echo%20%22Not%20a%20DHCP4%20address.%20Ignoring.%22%0A%20%20%20%20exit%200%0Afi%0A%0Aif%20%5B%20%24%7BDHCP4_DHCP_LEASE_TIME%3A-0%7D%20-lt%204294967295%20%5D%0Athen%0A%20%20%20%20%3E%262%20echo%20%22Not%20an%20infinite%20DHCP4%20lease.%20Ignoring.%22%0A%20%20%20%20exit%200%0Afi%0A%0AIPS%3D(%24IP4_ADDRESS_0)%0ACIDR%3D%24%7BIPS%5B0%5D%7D%0AGATEWAY%3D%24%7BIPS%5B1%5D%7D%0A%0ATYPE%3D%24(nmcli%20--get-values%20connection.type%20connection%20show%20%22%24CONNECTION_ID%22)%0A%0A%23%20Modifying%20the%20default%20connection%20id%20directly%20doesn't%20do%20what%20we%20want.%0A%23%20If%20we%20see%20that%2C%20then%20we%20need%20to%20create%20a%20new%20connection.%0Aif%20%5B%20%22%24CONNECTION_ID%22%20%3D%3D%20%22Wired%20Connection%22%20%5D%0Athen%0A%20%20%20%20if%20!%20nmcli%20con%20show%20inf-lease-to-static%0A%20%20%20%20then%0A%20%20%20%20%20%20%20%20nmcli%20con%20add%20type%20%22%24TYPE%22%20con-name%20inf-lease-to-static%0A%20%20%20%20fi%0A%20%20%20%20STATIC_INT_NAME%3Dinf-lease-to-static%0Aelse%0A%20%20%20%20STATIC_INT_NAME%3D%22%24CONNECTION_ID%22%0Afi%0Anmcli%20con%20mod%20%22%24STATIC_INT_NAME%22%20%5C%0A%20%20conn.interface%20%22%241%22%20%5C%0A%20%20connection.autoconnect%20yes%20%5C%0A%20%20ipv4.addresses%20%22%24CIDR%22%20%5C%0A%20%20ipv4.method%20manual%20%5C%0A%20%20ipv4.gateway%20%22%24GATEWAY%22%20%5C%0A%20%20ipv4.dns%20%22%24IP4_NAMESERVERS%22%0A%0Aif%20%5B%20-n%20%22%24IP4_DOMAINS%22%20%5D%3B%20then%0A%20%20%20%20nmcli%20con%20mod%20%22%24STATIC_INT_NAME%22%20ipv4.dns-search%20%22%24IP4_DOMAINS%22%0Afi%0Aplus%3D''%0Afor%20i%20in%20%24(seq%200%20%24((%24IP4_NUM_ROUTES-1))%20)%0Ado%0A%20%20%20%20varname%3D%22IP4_ROUTE_%24i%22%0A%20%20%20%20nmcli%20con%20mod%20%22%24STATIC_INT_NAME%22%20%24%7Bplus%7Dipv4.routes%20%22%24%7B!varname%7D%22%0A%20%20%20%20plus%3D'%2B'%0Adone%0A%0Anmcli%20con%20up%20%22%24STATIC_INT_NAME%22%0A%0A%23%20Copy%20it%20from%20the%20OverlayFS%20mount%20to%20the%20persistent%20lowerdir%0Acp%20%22%2Fetc%2FNetworkManager%2FsystemConnectionsMerged%2F%24%7BSTATIC_INT_NAME%7D.nmconnection%22%20%2Fetc%2FNetworkManager%2Fsystem-connections%0A%0Aif%20%5B%20-n%20%22%24%7BDHCP4_HOST_NAME%3A-%7D%22%20%5D%0Athen%0A%20%20%20%20hostnamectl%20set-hostname%20--static%20--transient%20%22%24DHCP4_HOST_NAME%22%0Afi%0A         
          mode: 493         
          overwrite: true         
          path: /etc/NetworkManager/dispatcher.d/30-static-dhcp
      
      $ echo "W2Nvbm5lY3Rpb25dCmlkPWJvbmQwCnR5cGU9Ym9uZAppbnRlcmZhY2UtbmFtZT1ib25kMAphdXRvY29ubmVjdD10cnVlCmNvbm5lY3Rpb24uYXV0b2Nvbm5lY3Qtc2xhdmVzPTEKCltldGhlcm5ldF0KbXR1PTkwMDAKCltib25kXQptb2RlPWFjdGl2ZS1iYWNrdXAKbWlpbW9uPTEwMApmYWlsX292ZXJfbWFjPWFjdGl2ZQpudW1fZ3JhdF9hcnA9MTAKCltpcHY0XQptZXRob2Q9YXV0bwpkaGNwLXRpbWVvdXQ9MjE0NzQ4MzY0NwoKW2lwdjZdCm1ldGhvZD1kaXNhYmxlZA==" | base64 -d
      [connection]
      id=bond0
      type=bond
      interface-name=bond0
      autoconnect=true
      connection.autoconnect-slaves=1[ethernet]
      mtu=9000
      
      [bond]
      mode=active-backup
      miimon=100
      fail_over_mac=active
      num_grat_arp=10
      
      [ipv4]
      method=auto
      dhcp-timeout=2147483647
      
      [ipv6]
      method=disabled 
      $ echo "W2Nvbm5lY3Rpb25dCmlkPWVuczFmMAp0eXBlPWV0aGVybmV0CmludGVyZmFjZS1uYW1lPWVuczFmMAptYXN0ZXI9Ym9uZDAKc2xhdmUtdHlwZT1ib25kCmF1dG9jb25uZWN0PXRydWUKCltldGhlcm5ldF0KbXR1PTkwMDA=" | base64 -d 
      [connection] 
      id=ens1f0 
      type=ethernet 
      interface-name=ens1f0 
      master=bond0 
      slave-type=bond 
      autoconnect=true
      
      [ethernet] 
      mtu=9000
      
      $ echo "W2Nvbm5lY3Rpb25dCmlkPWVuczFmMQp0eXBlPWV0aGVybmV0CmludGVyZmFjZS1uYW1lPWVuczFmMQptYXN0ZXI9Ym9uZDAKc2xhdmUtdHlwZT1ib25kCmF1dG9jb25uZWN0PXRydWUKCltldGhlcm5ldF0KbXR1PTkwMDA="| base64 -d
      [connection]
      id=ens1f1
      type=ethernet
      interface-name=ens1f1
      master=bond0
      slave-type=bond
      autoconnect=true
      
      [ethernet]
      mtu=9000 

      When deploying, cluster looks fine and runs fine as-is and this bond0 config looks like this as expected:

      [connection]
      id=bond0
      type=bond
      interface-name=bond0
      autoconnect=true
      autoconnect-priority=98
      connection.autoconnect-slaves=1
      
      [ethernet]
      mtu=9000
      
      [bond]
      mode=802.3ad
      miimon=100
      lacp_rate=1
      
      [ipv4]
      method=auto
      dhcp-timeout=2147483647
      
      [ipv6]
      method=disabled 

      It's been running like this without issue. 

      Then around '2023-10-26 13:27:37', VZ modified the line 'dhcp-range' from the file 'sosreport-bastion-2023-10-26-nrprllh/etc/dnsmasq.d/baremetal.conf' and restarted dnsmasq

          1. BEFORE
            dhcp-range=set:baremetal,69.100.89.0,static,336h 
          2. AFTER
            dhcp-range=set:baremetal,69.100.89.0,static,infinite 
          3. Restart
            $ less sosreport-bastion-2023-10-26-nrprllh/sos_commands/systemd/systemctl_status_--all
            * dnsmasq.service - DNS caching server.
               Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; enabled; vendor preset: disabled)
               Active: active (running) since Thu 2023-10-26 13:27:37 UTC; 6h ago
             Main PID: 1983632 (dnsmasq)
                Tasks: 1 (limit: 3293354)
               Memory: 1.2M
               CGroup: /system.slice/dnsmasq.service
                       `-1983632 /usr/sbin/dnsmasq -k 

            About 5+ hours later, at 'Oct 26 18:40:52', we see 'worker-20' perform a DHCP request:

            $ less sosreport-bastion-2023-10-26-nrprllh/var/log/messages
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 available DHCP subnet: 69.100.89.0/255.255.255.128
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 DHCPREQUEST(baremetal) 69.100.89.33 88:e9:a4:15:6b:4c
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 tags: known, baremetal
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 DHCPACK(baremetal) 69.100.89.33 88:e9:a4:15:6b:4c worker-20
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: Ignoring duplicate dhcp-option 59
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: Ignoring duplicate dhcp-option 58
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: Ignoring duplicate dhcp-option 59
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: Ignoring duplicate dhcp-option 58
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 requested options: 1:netmask, 2:time-offset, 6:dns-server, 12:hostname,
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 requested options: 15:domain-name, 26:mtu, 28:broadcast, 121:classless-static-route,
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 requested options: 3:router, 33:static-route, 40:nis-domain,
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 requested options: 41:nis-server, 42:ntp-server, 119:domain-search,
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 requested options: 249, 252, 17:root-path
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 next server: 69.100.89.4
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 sent size:  1 option: 53 message-type  5
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 sent size:  4 option: 54 server-identifier  69.100.89.4
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 sent size:  4 option: 51 lease-time  infinite
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 sent size:  4 option: 28 broadcast  69.100.89.127
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 sent size: 24 option: 15 domain-name  kub2.rcmdva83.vzwops.com
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 sent size:  9 option: 12 hostname  worker-20
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 sent size:  2 option: 26 mtu  9000
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 sent size:  4 option: 42 ntp-server  69.100.89.4
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 sent size:  8 option:  6 dns-server  10.215.0.6, 10.215.0.7
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 sent size:  4 option:  3 router  69.100.89.1
            Oct 26 18:40:52 bastion dnsmasq-dhcp[1983632]: 1832159419 sent size:  4 option:  1 netmask  255.255.255.128 

            We can see this correlation from the server side where the 'bond0' got torn down. Keep in mind that the IP on this bond0 doesn't change but it still triggers the teardown. 

            $ less sosreport-worker-20-2023-10-26-flrusio/sos_commands/networkmanager/journalctl_--no-pager_--unit_NetworkManager 
            Oct 26 18:40:52 worker-20 NetworkManager[14930]: <info>  [1698345652.4842] dhcp4 (bond0): state changed extended -> extended, address=69.100.89.33
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.2215] audit: op="connection-update" uuid="52eecf5a-df5e-30ae-9ca1-6297f0239027" name="bond0" args="ipv4.addresses,ipv4.gateway,ipv4.dns,ipv4.method,connection.timestamp" pid=1995679 uid=0 result="success"
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.2779] audit: op="connection-update" uuid="52eecf5a-df5e-30ae-9ca1-6297f0239027" name="bond0" args="ipv4.dns-search" pid=1995708 uid=0 result="success"
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.3366] audit: op="connection-update" uuid="52eecf5a-df5e-30ae-9ca1-6297f0239027" name="bond0" args="ipv4.routes" pid=1995723 uid=0 result="success"
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.3935] audit: op="connection-update" uuid="52eecf5a-df5e-30ae-9ca1-6297f0239027" name="bond0" args="ipv4.routes" pid=1995772 uid=0 result="success"
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.4506] agent-manager: agent[8589b4812ce15f8a,:1.951751/nmcli-connect/0]: agent registered
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.4510] device (bond0): state change: activated -> deactivating (reason 'new-activation', sys-iface-state: 'managed')
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.4518] manager: NetworkManager state is now CONNECTED_LOCAL
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.4522] device (bond0): disconnecting for new activation request.
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.4522] audit: op="connection-activate" uuid="52eecf5a-df5e-30ae-9ca1-6297f0239027" name="bond0" pid=1995776 uid=0 result="success"
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.4526] device (bond0): state change: deactivating -> disconnected (reason 'new-activation', sys-iface-state: 'managed')
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.4533] dhcp4 (bond0): canceled DHCP transaction
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.4533] dhcp4 (bond0): state changed extended -> done
            Oct 26 18:40:53 worker-20 NetworkManager[14930]: <info>  [1698345653.8544] device (bond0): released bond slave ens1f0
            Oct 26 18:40:55 worker-20 NetworkManager[14930]: <info>  [1698345655.7513] device (bond0): released bond slave ens1f1
            Oct 26 18:40:57 worker-20 NetworkManager[14930]: <info>  [1698345657.3925] device (bond0.696): set-hw-addr: set MAC address to EE:7E:E2:D4:98:7C (vlan-parent)
            Oct 26 18:40:57 worker-20 NetworkManager[14930]: <info>  [1698345657.3933] device (bond0.697): set-hw-addr: set MAC address to EE:7E:E2:D4:98:7C (vlan-parent)
            Oct 26 18:40:57 worker-20 NetworkManager[14930]: <info>  [1698345657.4024] device (bond0): Activation: starting connection 'bond0' (52eecf5a-df5e-30ae-9ca1-6297f0239027)
            Oct 26 18:41:24 worker-20 NetworkManager[14930]: <warn>  [1698345684.2104] dispatcher: (43259) /etc/NetworkManager/dispatcher.d/30-static-dhcp failed (failed): Script '/etc/NetworkManager/dispatcher.d/30-static-dhcp' exited with error status 1. 

            We then see the '/etc/NetworkManager/dispatcher.d/30-static-dhcp' script get executed (last line from above) which appears to then generate a new 'bond0.nmconnection'. No other '.nmconnection' files were touched.

            $ less sosreport-worker-20-2023-10-26-flrusio/etc/NetworkManager/dispatcher.d/30-static-dhcp
            # Copy it from the OverlayFS mount to the persistent lowerdir
            cp "/etc/NetworkManager/systemConnectionsMerged/${STATIC_INT_NAME}.nmconnection" /etc/NetworkManager/system-connections
            
            $ ll sosreport-worker-20-2023-10-26-flrusio/etc/NetworkManager/system-connections/bond0.nmconnection
            total 72
            -rw-rw-rw-+ 1 yank yank 520 Oct 26 18:40 bond0.nmconnection 

            and now our bond config is completely different than before

            $ cat sosreport-worker-20-2023-10-26-flrusio/etc/NetworkManager/system-connections/bond0.nmconnection 
            [connection]
            id=bond0
            uuid=52eecf5a-df5e-30ae-9ca1-6297f0239027
            type=bond
            autoconnect-priority=98
            interface-name=bond0
            permissions=
            timestamp=1698345392
            
            [ethernet]
            mac-address-blacklist=
            mtu=9000
            
            [bond]
            lacp_rate=1
            miimon=100
            mode=802.3ad
            
            [ipv4]
            address1=69.100.89.33/25,69.100.89.1
            dhcp-timeout=2147483647
            dns=10.215.0.6;10.215.0.7;
            dns-search=kub2.rcmdva83.vzwops.com;
            method=manual
            route1=69.100.89.0/25,0.0.0.0,303
            route2=69.100.89.33/32,0.0.0.0,0
            
            [ipv6]
            addr-gen-mode=eui64
            dns-search=
            method=disabled
            
            [proxy] 

            We need to discuss where the placement needs to be so we're opening a bug with engineering...

      At the end of the day, the expectation was that this would've been non-impactful to update the DHCP lease from dnsmasq on the bastion to a value of 'infinite' from '336h' as they thought this would only extend the lease provided by DNSMASQ, not trigger NetworkManager to tear down the interface and restart it.

          1. Bonus Points:
      • When bond0 gets torn down, it affects
        • bond0 (used for NodeIP and SSH access)
        • bond0.696 (used for ClusterNetwork [pods])
        • bond0.697 (Block Storage IP [Trident Netapp PVC])
        • br-ex (hangs off of bond0.696 for ICNIVLAN traffic)
        • If pods are using SRIOV from interfaces ens1f0 and ens1f1
         $ grep -Ri "6b:4c" sosreport-bastion-2023-10-26-nrprllh/etc/dnsmasq.d/
         0050-sosreport-bastion-2023-10-26-nrprllh.tar.xz/sosreport-bastion-2023-10-26-nrprllh/etc/dnsmasq.d/baremetal.conf:dhcp- 
         host=88:e9:a4:15:6b:4c,88:e9:a4:15:6b:4d,worker-20,69.100.89.33,set:baremetal
         
         0050-sosreport-bastion-2023-10-26-nrprllh.tar.xz/sosreport-bastion-2023-10-26-nrprllh/etc/dnsmasq.d/icnivlan.conf:dhcp- 
         host=88:e9:a4:15:6b:4c,88:e9:a4:15:6b:4d,worker-20,198.19.0.33,set:icnivlan
          
         0050-sosreport-bastion-2023-10-26-nrprllh.tar.xz/sosreport-bastion-2023-10-26-nrprllh/etc/dnsmasq.d/blockstorage.conf:dhcp- 
         host=88:e9:a4:15:6b:4c,88:e9:a4:15:6b:4d,worker-20,69.99.4.28,set:blockstorage 

      [1] https://access.redhat.com/support/cases/03649974
      [2] https://access.redhat.com/support/cases/03272556
      [3] https://bugzilla.redhat.com/show_bug.cgi?id=2110000

            bnemec@redhat.com Benjamin Nemec
            rhn-support-acardena Albert Cardenas
            Sergio Regidor de la Rosa Sergio Regidor de la Rosa
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: