Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-5696

Multiple dhclient instances for the same interface are spawned when network.service is started after another systemd service has performed ifup on the same interface

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Minor Minor
    • None
    • rhel-7.9.z
    • initscripts
    • None
    • None
    • sst_cs_plumbers
    • ssg_core_services
    • 8
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • If docs needed, set a value
    • None

      In systems using network scripts (network.service), when generic systemd services with selinux context "system_u:system_r:unconfined_service_t" run /usr/sbin/ifup to bring up an interface set via dhcp, the dhclient that is spawned runs with selinux context "system_u:system_r:unconfined_service_t". That is different from the context that dhclient acquires if executed by /etc/init.d/network (system_u:system_r:dhcpc_t) or by the context dhclient acquires when spawn by a root user running /usr/sbin/ifup from the terminal:

      1. dhclient spwaned from generic systemd service via ifup:
        $ ps auxZ | grep dhclient
        system_u:system_r:unconfined_service_t:s0 root 26827 1 0 25751 3028 2 10:36 ? 00:00:00 /sbin/dhclient 1 -q -lf /var/lib/dhclient/dhclient-eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
      1. dhclient spwaned from /etc/init.d/network:
        $ ps auxZ | grep dhclient
        unconfined_u:system_r:dhcpc_t:s0-s0:c0.c1023 root 29114 1 0 25751 3032 2 10:38 ? 00:00:00 /sbin/dhclient 1 -q -lf /var/lib/dhclient/dhclient-eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3

      If we then run "/etc/init.d/network start" after the systemd service has performed /usr/sbin/ifup on a dhcp interface, /etc/init.d/network spwans additional dhclient instances for the same interface, which pile up in the host:

      1. After running a systemd service that does /usr/sbin/ifup on eth3, we run /etc/init.d/network start
        $ ps auxZ | grep dhclient
        system_u:system_r:unconfined_service_t:s0 root 17084 0.0 0.0 103004 2980 ? Ss 07:55 0:00 /sbin/dhclient 1 -q -lf /var/lib/dhclient/dhclient-eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
        system_u:system_r:dhcpc_t:s0 root 17464 0.0 0.0 103004 4160 ? Ss 07:55 0:00 /sbin/dhclient 1 -q -lf /var/lib/dhclient/dhclient-eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
        system_u:system_r:dhcpc_t:s0 root 18187 0.0 0.0 103004 4036 ? Ss 07:55 0:00 /sbin/dhclient 1 -q -lf /var/lib/dhclient/dhclient-eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
        unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 18265 0.0 0.0 112824 2248 pts/0 S+ 07:55 0:00 grep --color=auto dhclient

      dhclient shall notice that the interface is up and already configured, and that another dhclient is already running.

      By digging further, the root cause seems due to dhclient spawned by "/etc/init.d/network" being unable to replace the PID file in /var/run/dhclient-eth3.pid. This causes dhclient to fail to recognise that another dhclient instance is running, and ultimately causes several other instances to pile up. In particular, this seems due to the wrong selinux context that dhclient has when run from a generic systemd service, versus the selinux context it has when run from "/etc/init.d/network". Notice the error message in network.service logs when interface are brought up:

      Jul 25 16:30:24 timberlake-4 network[18988]: Bringing up interface eth0: [ OK ]
      Jul 25 16:30:24 timberlake-4 network[18988]: Bringing up interface eth1: [ OK ]
      Jul 25 16:30:25 timberlake-4 network[18988]: Bringing up interface eth2: [ OK ]
      Jul 25 16:30:25 timberlake-4 network[18988]: Bringing up interface eth3:
      Jul 25 16:30:25 timberlake-4 dhclient[20024]: Can't create /var/run/dhclient-eth3.pid: Permission denied
      Jul 25 16:30:25 timberlake-4 network[18988]: Determining IP information for eth3...Can't create /var/run/dhclient-eth3.pid: Permission denied
      Jul 25 16:30:25 timberlake-4 dhclient[20024]: DHCPREQUEST on eth3 to 255.255.255.255 port 67 (xid=0x26dd3b44)
      Jul 25 16:30:25 timberlake-4 dhclient[20024]: DHCPACK from 10.53.96.1 (xid=0x26dd3b44)
      Jul 25 16:30:27 timberlake-4 dhclient[20024]: bound to 10.53.102.56 – renewal in 1404 seconds.
      Jul 25 16:30:27 timberlake-4 dhclient[20119]: Can't create /var/run/dhclient-eth3.pid: Permission denied
      Jul 25 16:30:27 timberlake-4 network[18988]: done.
      Jul 25 16:30:27 timberlake-4 network[18988]: [ OK ]
      Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
      Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
      Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
      Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
      Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
      Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
      Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
      Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
      Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists

      If we repeat the same experiment, but this time performing "restorecon -R /var/run/" after the systemd service runs /usr/sbin/ifup (to fix the labelling of the PID file /var/run/dhclient-eth3.pid), we notice that dhclient spawned from "/etc/init.d/network start" can tell that the interface is up and configured, and thus refrains from running again:

      Determining IP information for eth3...dhclient(22807) is already running - exiting

      Version-Release number of selected component (if applicable):

      [root@localhost nutanix]# cat /etc/redhat-release
      Red Hat Enterprise Linux Server release 7.9 (Maipo)

      [root@localhost nutanix]# rpm -qa | grep selinux
      selinux-policy-targeted-3.13.1-268.el7.noarch
      libselinux-python-2.5-15.el7.x86_64
      selinux-policy-3.13.1-268.el7.noarch
      libselinux-2.5-15.el7.x86_64
      libselinux-utils-2.5-15.el7.x86_64

      How reproducible: Every time

      Steps to Reproduce:

      1. Configure an interface to get IP address via dhcp. E.g,: /etc/sysconfig/network-scripts/ifcfg-eth1:
      TYPE="Ethernet"
      PROXY_METHOD="none"
      BROWSER_ONLY="no"
      BOOTPROTO="dhcp"
      DEFROUTE="yes"
      IPV4_FAILURE_FATAL="no"
      IPV6INIT="yes"
      IPV6_AUTOCONF="yes"
      IPV6_DEFROUTE="yes"
      IPV6_FAILURE_FATAL="no"
      IPV6_ADDR_GEN_MODE="stable-privacy"
      NAME="eth1"
      DEVICE="eth1"
      ONBOOT="yes"
      NM_CONTROLLED=no

      2. Create a systemd service that brings up eth1 via ifup. E.g.:
      [nutanix@localhost ~]$ cat /usr/example.py
      #! /usr/bin/env python

      import subprocess

      def run(cmd_tokens):
      subprocess.call(cmd_tokens)

      if _name_ == '_main_':

      1. We first stop network.service, so we can bring up interface again.
        run(["service", "network", "stop"])
      1. We manually bring up eth1.
        run(["/sbin/ifdown", "eth1"])
        run(["/sbin/ifup", "eth1"])
      1. We use network.service to bring-up the already up eth1
        run(["service", "network", "start"])

      [nutanix@localhost ~]$ systemctl cat example.service

      1. /etc/systemd/system/example.service
        [Unit]
        Description=Example
        After=network.target
        StartLimitIntervalSec=0
        [Service]
        Type=oneshot
        ExecStart=/usr/example.py

      [Install]
      WantedBy=multi-user.target

      [Install]
      WantedBy=multi-user.target

      3. Start the systemd service example.service
      $ systemctl start example.service

      4. Check "ps auxZ | grep dhclient" and "ls -lhZ /var/run" and notice the multiple instances of dhclient

      Actual results:

      $ ps auxZ | grep dhclient ; ls -lhZ /var/run/ | grep dhclient
      system_u:system_r:unconfined_service_t:s0 root 17084 0.0 0.0 103004 2980 ? Ss 07:55 0:00 /sbin/dhclient 1 -q -lf /var/lib/dhclient/dhclient-eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
      system_u:system_r:dhcpc_t:s0 root 17464 0.0 0.0 103004 4160 ? Ss 07:55 0:00 /sbin/dhclient 1 -q -lf /var/lib/dhclient/dhclient-eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
      system_u:system_r:dhcpc_t:s0 root 18187 0.0 0.0 103004 4036 ? Ss 07:55 0:00 /sbin/dhclient 1 -q -lf /var/lib/dhclient/dhclient-eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
      unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 18265 0.0 0.0 112824 2248 pts/0 S+ 07:55 0:00 grep --color=auto dhclient

      Expected results:

      ps auxZ | grep dhcli ; ls -lhZ /var/run/ | grep dhcl ; systemctl status network
      unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 6941 0.0 0.0 112824 2280 pts/1 S+ 16:35 0:00 grep --color=auto dhcli
      system_u:system_r:dhcpc_t:s0 root 20734 0.0 0.0 103004 2980 ? Ss 16:31 0:00 /sbin/dhclient 1 -q -lf /var/lib/dhclient/dhclient-eth3.lease -pf /var/run/dhclient-eth3.pid -H seabass-3 eth3
      rw-rr-. root root system_u:object_r:dhcpc_var_run_t:s0 dhclient-eth3.pid

            jamacku@redhat.com Jan Macku
            jira-bugzilla-migration RH Bugzilla Integration
            Jan Macku Jan Macku
            Daniel Rusek Daniel Rusek
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: