Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-22109

After EDPM node reboots, chrony is not synced with the assigned server when ifcfg method is used

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • rhos-18.0.z
    • os-net-config
    • None
    • Bugs Pending Verification
    • 1
    • Important

      To Reproduce Steps to reproduce the behavior:

      1. Deploy RHOSO18 with ifcfg network method for EDPM nodes and NTP servers are assigned for EDPM nodes
      2. Reboot the EDPM nodes.
      3. chrony is not synced with the assigned server.

      Expected behavior

      • chrony synced with the assigned NTP servers.

      Bug impact

      • After system rebooted, EDPM node is out of sync with NTP servers and time difference happens with OCP nodes or the rest of EDPM nodes.

      Known workaround

      • Run chronyc onoffline after reboot
      • Disable NetworkManager

      Additional context

      • If I understand correctly, ifcfg method in os-net-config is still supported and the older method used customer in RHOSO18 are existing.
      • This behavior happens the following reasons.
        When ifcfg method is used, NetworkManager(NM) is still enabled. In ifcfg-XXX configuratin, ifcfg-lo file is a single device file which is not managed by os-net-config and it doesn't have NM_CONTROLLED=no. Due to that, lo device is managed by NM. As a result NM tries to up lo device then it sends connectivity-change event.
        Nov 18 05:53:27 edpm-compute-0.rhoso.example.com chronyd[1007]: chronyd version 4.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 +DEBUG)
        Nov 18 05:53:27 edpm-compute-0.rhoso.example.com chronyd[1007]: Frequency -13.775 +/- 0.018 ppm read from /var/lib/chrony/drift
        :
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4225] monotonic timestamp started counting 1.479607127 seconds ago with an offset of 8.0 seconds to CLOCK_BOOTTIME (local time is 2025-11-18 05:53:29)
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <info>  [1763445209.4229] NetworkManager (version 1.46.0-33.el9_4) is starting... (boot:db1c4e38-58ff-44b7-b14d-3088bf2e9931)
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <info>  [1763445209.4229] Read config: /etc/NetworkManager/NetworkManager.conf (etc: 95-nm-debug.conf, 99-cloud-init.conf)
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4229] CONFIG: config-data[0x5646b85db020]: 2 groups
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4229] CONFIG: 
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4229] CONFIG: [main]
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4230] CONFIG:   dns=none
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4230] CONFIG:   rc-manager=unmanaged
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4230] CONFIG:   no-auto-default=*
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4230] CONFIG:   configure-and-quit=no
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4230] CONFIG: 
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4230] CONFIG: [logging]
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4230] CONFIG:   domains=ALL:TRACE
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4230] CONFIG: 
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4230] CONFIG: # no-auto-default file "/var/lib/NetworkManager/no-auto-default.state"
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4232] config: state: successfully read state file "/var/lib/NetworkManager/NetworkManager.state"
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4232] WEXT support is disabled
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.4237] create NMDBusManager singleton (67bfecf95168a207)
        :
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.6263] device[d0821320cb7af9be] (lo): connectivity state changed from NONE to LIMITED
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.6264] device[d0821320cb7af9be] (lo): connectivity state changed from NONE to LIMITED
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.6264] device[d0821320cb7af9be] (lo): remove_pending_action (1): 'in-state-change'
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.6265] device[d0821320cb7af9be] (lo): remove_pending_action (0): 'queued-state-change-activated'
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <info>  [1763445209.6265] manager: startup complete
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <trace> [1763445209.6265] settings: [timestamps-keyfile]: prune keyfile of old entries: "/var/lib/NetworkManager/timestamps"
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:4 'up' [lo]: new request (4 scripts)
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <trace> [1763445209.6285] settings: [timestamps-keyfile]: write keyfile: "/var/lib/NetworkManager/timestamps"
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:4 'up' [lo]: environment: CONNECTION_DBUS_PATH=/org/freedesktop/NetworkManager/Settings/1
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.6285] pacrunner: pacrunner not on D-Bus
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:4 'up' [lo]: environment: CONNECTION_EXTERNAL=1
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.6285] pacrunner: try D-Bus activating pacrunner...
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:4 'up' [lo]: environment: CONNECTION_FILENAME=/run/NetworkManager/system-connections/lo.nmconnection
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:4 'up' [lo]: environment: CONNECTION_UUID=bdf13464-cf19-48ae-8152-656ef877f6c5
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:4 'up' [lo]: environment: CONNECTION_ID=lo
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:4 'up' [lo]: environment: DEVICE_IFACE=lo
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:4 'up' [lo]: environment: DEVICE_IP_IFACE=lo
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:4 'up' [lo]: environment: IP4_NUM_ROUTES=0
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:4 'up' [lo]: environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:4 'up' [lo]: environment: NM_DISPATCHER_ACTION=up
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:1 'hostname', "/usr/lib/NetworkManager/dispatcher.d/11-dhclient": complete: process succeeded
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:1 'hostname', "/usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp": run script
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com systemd[1]: Finished Network Manager Wait Online.
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:1 'hostname', "/usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp": complete: process succeeded
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:1 'hostname', "/usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline": run script
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com systemd[1]: Starting LSB: Bring up/down networking...
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:1 'hostname', "/usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline": complete: process succeeded
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:1 'hostname': completed (4 scripts)
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:2 'connectivity-change': start running ordered scripts...
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:2 'connectivity-change', "/usr/lib/NetworkManager/dispatcher.d/04-iscsi": run script
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.6386] dispatcher: (1) succeeded (after 0.076 sec, 4 scripts invoked)
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.6387] dispatcher: (1) /usr/lib/NetworkManager/dispatcher.d/04-iscsi succeeded
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.6387] dispatcher: (1) /usr/lib/NetworkManager/dispatcher.d/11-dhclient succeeded
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.6387] dispatcher: (1) /usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp succeeded
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com NetworkManager[1186]: <debug> [1763445209.6387] dispatcher: (1) /usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline succeeded
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:2 'connectivity-change', "/usr/lib/NetworkManager/dispatcher.d/04-iscsi": complete: process succeeded
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:2 'connectivity-change', "/usr/lib/NetworkManager/dispatcher.d/11-dhclient": run script
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:2 'connectivity-change', "/usr/lib/NetworkManager/dispatcher.d/11-dhclient": complete: process succeeded
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:2 'connectivity-change', "/usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp": run script
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:2 'connectivity-change', "/usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp": complete: process succeeded
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com nm-dispatcher[1193]: req:2 'connectivity-change', "/usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline": run script
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com chronyd[1007]: Source 192.168.122.1 offline
        Nov 18 05:53:29 edpm-compute-0.rhoso.example.com network[1205]: WARN      : [network] You are using 'network' service provided by 'network-scripts', which are now deprecated.
        

      connectivity-change events triggers NM dispatcher script and the script runs /usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline

      #!/usr/bin/sh
      # This is a NetworkManager dispatcher / networkd-dispatcher script for
      # chronyd to set its NTP sources online or offline when a network interface
      # is configured or removed
      
      export LC_ALL=C
      
      chronyc=/usr/bin/chronyc
      
      # For NetworkManager consider only selected events
      if [ $# -ge 2 ]; then
          case "$2" in
              up|down|connectivity-change)
                  ;;
              dhcp4-change|dhcp6-change)
                  # Actions "up" and "connectivity-change" in some cases do not
                  # guarantee that the interface has a route (e.g. a bond).
                  # dhcp(x)-change handles at least cases that use DHCP.
                  ;;
              *)
                  exit 0;;
          esac
      fi
      
      # Note: for networkd-dispatcher routable.d ~= on and off.d ~= off
      
      $chronyc onoffline > /dev/null 2>&1
      
      exit 0
      

      By running chronyc onoffline and the script isn't run again, the chronyc keeps sources are offline.

      To prevent this behavior, we should disable NetworkManager for ifcfg method.

              vcandapp@redhat.com Vijayalakshmi Candappa
              rhn-support-knoha Keigo Noha
              rhos-dfg-nfv
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: