Uploaded image for project: 'Openshift sandboxed containers'
  1. Openshift sandboxed containers
  2. KATA-4007

Clock drift noticed in Openshift sandbox containers

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: High High
    • OSC 1.10.0
    • OSC 1.9.0
    • None
    • None
    • Quality / Stability / Reliability
    • 2
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide
      .Clock drift noticed in Openshift sandbox containers
      Cause: Clock drift is observed in sandboxed containers on baremetal.
      Consequence: Time critical workloads might have a degraded behavior or fail.
      Fix: Kata guest clock is now synchronized with the host clock.
      Result: No more drift is observed.
      Show
      .Clock drift noticed in Openshift sandbox containers Cause: Clock drift is observed in sandboxed containers on baremetal. Consequence: Time critical workloads might have a degraded behavior or fail. Fix: Kata guest clock is now synchronized with the host clock. Result: No more drift is observed.
    • Bug Fix
    • Done
    • Kata Sprint #272

      Description of problem:

      Whats the best recommendation to avoid clock drift for katacontainers ?
      
      clock drift noticed on Openshift Sandbox containers affecting time sensitive application. Time in the container is either ahead or behind.
      
      System clock is synced on the underlying baremetal host using chrony. There are no issues noticed with time sync on the underlying node. 
      
      Using clockdrift, customer was able to measure the drift. clockdrift shows that the Kata Pod is slowly drifting by about 1ms every 20-30 seconds:
      
      $ kubectl --context 1024 -n d192935 exec -it deployment/shell -- /bin/bash
      [root@shell-bfbc548f7-rvmtt /]# while true; do clockdiff -o1 time.XXX.com; echo; sleep 10; done
      ..................................................
      host=time.XXX.com rtt=3(0)ms/3ms delta=179ms/179ms Thu Jul  3 11:05:12 2025
      ..................................................
      host=time.XXX.com rtt=5(0)ms/4ms delta=179ms/180ms Thu Jul  3 11:05:28 2025
      ..................................................
      host=time1.XXX.com rtt=4(0)ms/4ms delta=180ms/180ms Thu Jul  3 11:05:45 2025
      ..................................................
      host=time1.XXX.com rtt=3(0)ms/3ms delta=180ms/180ms Thu Jul  3 11:06:01 2025
      ..................................................
      host=time1.XXX.com rtt=3(0)ms/3ms delta=180ms/180ms Thu Jul  3 11:06:17 2025
      ..................................................
      host=time.XXX.com rtt=4(0)ms/4ms delta=181ms/181ms Thu Jul  3 11:06:34 2025
      ..................................................
      host=time1.XXX.com rtt=3(0)ms/3ms delta=181ms/181ms Thu Jul  3 11:06:50 2025
      ..................................................
      host=time1.XXX.com rtt=4(0)ms/4ms delta=182ms/182ms Thu Jul  3 11:07:06 2025
      ...................................................
      
      while running the same command on the baremetal node, no drift is observed. Hardware: PowerEdge R6515 with AMD EPYC 7502P 32-Core Processor
      
      To avoid clock drift on VM, we typically make use of ntp/chrony on the Virtual Machine. 
      
          

      Version-Release number of selected component (if applicable):

      sandboxed-containers-operator.v1.9.0
      Openshift 4.18.15
          

      How reproducible:

      Always for the customer
          

      Steps to Reproduce:

          1. OCP 4.18.15 with Baremetal nodes
          2. KataContainers to be deployed
          3. Check clock drift using clockdiff 
          

      Actual results:

        clock drift is noticed 
          

      Expected results:

         avoid clock drift to prevent time sensitive application from getting affected
          

      Additional info:

      Customer **** have reported this problem. Due to this issue, their production application deployed through KataContainers have been rolled back. 
          

              rhgkurz Greg Kurz
              rhn-support-rrajaram Ranjith Rajaram
              Victor Voronkov Victor Voronkov
              Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: