Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-25692

SNO DU "PTP failed to get time" ice driver errors in kernel logs

XMLWordPrintable

    • No
    • CNF RAN Sprint 249, CNF RAN Sprint 250, CNF RAN Sprint 251
    • 3
    • False
    • Hide

      None

      Show
      None
    • Hide
      3/14: waiting for the forecast to available downstream
      3/7: 9.2 test kernel is made available by RHEL team, to be tested internally
      2/29: Patch under review from Intel.
      2/22: RHEL found issues with ice driver actually locking up . Intel is review the findings
      2/15: short term solution to downgrade ptp failed message from error to debug
      2/8: blocked by RHEL-19000
      1/4: bpftrace script was run on qe lab and updated kernel team
      1/2/2024: This was fixed to reduce the warning log message, either we need to fix to suppress logs or check with rhel/intel if there is any solution to prevent locking,
      Show
      3/14: waiting for the forecast to available downstream 3/7: 9.2 test kernel is made available by RHEL team, to be tested internally 2/29: Patch under review from Intel. 2/22: RHEL found issues with ice driver actually locking up . Intel is review the findings 2/15: short term solution to downgrade ptp failed message from error to debug 2/8: blocked by RHEL-19000 1/4: bpftrace script was run on qe lab and updated kernel team 1/2/2024: This was fixed to reduce the warning log message, either we need to fix to suppress logs or check with rhel/intel if there is any solution to prevent locking,

      Description of problem:

      Observing repeated ice driver errors "PTP failed to get time" on the console even though the SNO is PTP synchronized


Dec 15 10:31:38 e28-h11-r750 kernel: ice 0000:4b:00.0: PTP failed to get time
Dec 15 10:37:12 e28-h11-r750 kernel: ice 0000:4b:00.0: PTP failed to get time
Dec 15 10:49:50 e28-h11-r750 kernel: ice 0000:4b:00.0: PTP failed to get time
Dec 15 12:39:11 e28-h11-r750 kernel: ice 0000:4b:00.0: PTP failed to get time
Dec 15 13:07:33 e28-h11-r750 kernel: ice 0000:4b:00.0: PTP failed to get time


linux-daemon-container logs:
phc2sys[365287.444]: [ptp4l.0.config] CLOCK_REALTIME rms    3 max    3 freq  -7048 +/-   0 delay   531 +/-   0
phc2sys[365287.507]: [ptp4l.0.config] CLOCK_REALTIME rms    3 max    3 freq  -7043 +/-   0 delay   525 +/-   0
phc2sys[365287.569]: [ptp4l.0.config] CLOCK_REALTIME rms   17 max   17 freq  -7062 +/-   0 delay   528 +/-   0
phc2sys[365287.632]: [ptp4l.0.config] CLOCK_REALTIME rms    4 max    4 freq  -7054 +/-   0 delay   527 +/-   0
phc2sys[365287.695]: [ptp4l.0.config] CLOCK_REALTIME rms   12 max   12 freq  -7063 +/-   0 delay   528 +/-   0
phc2sys[365287.757]: [ptp4l.0.config] CLOCK_REALTIME rms    2 max    2 freq  -7053 +/-   0 delay   528 +/-   0
phc2sys[365287.820]: [ptp4l.0.config] CLOCK_REALTIME rms   14 max   14 freq  -7040 +/-   0 delay   524 +/-   0
phc2sys[365287.883]: [ptp4l.0.config] CLOCK_REALTIME rms   19 max   19 freq  -7031 +/-   0 delay   524 +/-   0
phc2sys[365287.945]: [ptp4l.0.config] CLOCK_REALTIME rms   14 max   14 freq  -7030 +/-   0 delay   517 +/-   0
phc2sys[365288.008]: [ptp4l.0.config] CLOCK_REALTIME rms   18 max   18 freq  -7022 +/-   0 delay   529 +/-   0
phc2sys[365288.070]: [ptp4l.0.config] CLOCK_REALTIME rms   17 max   17 freq  -7018 +/-   0 delay   527 +/-   0
phc2sys[365288.133]: [ptp4l.0.config] CLOCK_REALTIME rms    8 max    8 freq  -7022 +/-   0 delay   527 +/-   0
phc2sys[365288.196]: [ptp4l.0.config] CLOCK_REALTIME rms   18 max   18 freq  -7009 +/-   0 delay   525 +/-   0
phc2sys[365288.258]: [ptp4l.0.config] CLOCK_REALTIME rms    6 max    6 freq  -7016 +/-   0 delay   526 +/-   0

      Version-Release number of selected component (if applicable):

       ptp-operator.v4.14.0-202311092032
OCP Version : 4.14.4
Kernel Version: 5.14.0-284.41.1.rt14.326.el9_2.x86_64

      How reproducible:

      These kernel messages appear at a regular frequency of ~20 min on the console on a PTP configured DU

      Actual results:

          Dec 15 10:31:38 e28-h11-r750 kernel: ice 0000:4b:00.0: PTP failed to get time
Dec 15 10:37:12 e28-h11-r750 kernel: ice 0000:4b:00.0: PTP failed to get time
Dec 15 10:49:50 e28-h11-r750 kernel: ice 0000:4b:00.0: PTP failed to get time
Dec 15 12:39:11 e28-h11-r750 kernel: ice 0000:4b:00.0: PTP failed to get time
Dec 15 13:07:33 e28-h11-r750 kernel: ice 0000:4b:00.0: PTP failed to get time

      Expected results:

      Errors are not expected when the node is PTP synced

      Additional info:

      PTP config and kernel logs attached

            aputtur@redhat.com Aneesh Puttur
            nchhabra@redhat.com Noreen Chhabra
            Hen Shay Hassid Hen Shay Hassid
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: