Loading...

XML

Word

Printable

Type: Bug
Resolution: Done-Errata
Priority: Undefined
Fix Version/s: None
Affects Version/s: 4.15
Component/s: Networking / ptp
Labels:

Severity:
Moderate
Regression:
No
Sprint:
CNF RAN Sprint 247, CNF RAN Sprint 248, CNF RAN Sprint 249
sprint_count:
3
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Release Note Text:
N/A
Release Note Type:
Release Note Not Required
Release Note Status:
In Progress
Latest Status Summary:
1/2: In development .Fix will be available in next two days
Target Version:

4.13.z

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

This is a clone of issue ~~OCPBUGS-25547~~. The following is the description of the original issue:
—
Description of problem:

Cluster had two ptpconfigs (GM and OC), after deleting the OC config, ptp metrics still includes metrics for the old configs.

Version-Release number of selected component (if applicable):

4.15

How reproducible:

100%

Steps to Reproduce:

    1. Apply two ptpconfigs at the same time - GM and OC
    2. Delete OC ptpconfig
    3. Check ptp metrics

Actual results:

  -   PTP metrics still includes metrics for the delete OC config
  -   PTP metrics is showing the correct metrics only after restarting the ptp linuxdeamon pod.

Expected results:

    PTP metrics only includes metrics for the remaining GM config without restarting ptp linux daemon pod

Additional info:

eno12399 was configured as OC and the ptpconfig was deleted. However, the ptp metrics still includes eno12399 related metrics, while the ptp linuxdaemon logs is showing only the expected offsets for the remaining GM config. 

[kni@registry.kni-qe-61 ~]$ oc get ptpconfigs.ptp.openshift.io -A NAMESPACE       NAME           AGE openshift-ptp   grandmaster1   14h [kni@registry.kni-qe-61 ~]$ 

[kni@registry.kni-qe-61 ~]$ oc logs -n openshift-ptp linuxptp-daemon-vfhjr linuxptp-daemon-container --since=10m | grep -i eno [kni@registry.kni-qe-61 ~]$  


[root@helix60 /]# curl http://localhost:9091/metrics
# HELP cne_api_events_published Metric to get number of events published by the rest api
# TYPE cne_api_events_published gauge
cne_api_events_published{address="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/gnss-status/gnss-sync-status",status="success"} 4
cne_api_events_published{address="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/ptp-status/lock-state",status="success"} 59
cne_api_events_published{address="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/ptp-status/ptp-clock-class-change",status="success"} 12
cne_api_events_published{address="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/sync-status/os-clock-sync-state",status="success"} 74
# HELP cne_api_publishers Metric to get number of publishers
# TYPE cne_api_publishers gauge
cne_api_publishers{status="active"} 4
# HELP cne_events_ack Metric to get number of events produced
# TYPE cne_events_ack gauge
cne_events_ack{status="failed",type="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/gnss-status/gnss-sync-status"} 2
cne_events_ack{status="failed",type="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/ptp-status/lock-state"} 6
cne_events_ack{status="failed",type="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/ptp-status/ptp-clock-class-change"} 5
cne_events_ack{status="failed",type="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/sync-status/os-clock-sync-state"} 9
cne_events_ack{status="success",type="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/gnss-status/gnss-sync-status"} 1
cne_events_ack{status="success",type="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/ptp-status/lock-state"} 50
cne_events_ack{status="success",type="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/ptp-status/ptp-clock-class-change"} 2
cne_events_ack{status="success",type="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/sync-status/os-clock-sync-state"} 38
# HELP cne_transport_events_published Metric to get number of events published by the transport
# TYPE cne_transport_events_published gauge
cne_transport_events_published{address="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/gnss-status/gnss-sync-status",status="failed"} 198
cne_transport_events_published{address="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/gnss-status/gnss-sync-status",status="success"} 1
cne_transport_events_published{address="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/ptp-status/lock-state",status="failed"} -2
cne_transport_events_published{address="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/ptp-status/lock-state",status="success"} 248
cne_transport_events_published{address="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/ptp-status/ptp-clock-class-change",status="failed"} -3
cne_transport_events_published{address="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/ptp-status/ptp-clock-class-change",status="success"} 200
cne_transport_events_published{address="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/sync-status/os-clock-sync-state",status="failed"} 4
cne_transport_events_published{address="/cluster/node/helix60.lab.eng.rdu2.redhat.com/sync/sync-status/os-clock-sync-state",status="success"} 235
# HELP cne_transport_sender Metric to get number of sender created
# TYPE cne_transport_sender gauge
cne_transport_sender{address="89cf9237-5a94-384b-bbb0-67739967f256",status="active"} 1
# HELP openshift_ptp_clock_class 6 = Locked, 7 = PRC unlocked in-spec, 52/187 = PRC unlocked out-of-spec, 248 = Default, 255 = Slave Only Clock
# TYPE openshift_ptp_clock_class gauge
openshift_ptp_clock_class{node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 6
# HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER
# TYPE openshift_ptp_clock_state gauge
openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="helix60.lab.eng.rdu2.redhat.com",process="phc2sys"} 1
openshift_ptp_clock_state{iface="eno1239x",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 2
openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="GM"} 1
openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="dpll"} 1
openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="gnss"} 1
openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1
# HELP openshift_ptp_delay_ns 
# TYPE openshift_ptp_delay_ns gauge
openshift_ptp_delay_ns{from="master",iface="eno1239x",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 627
openshift_ptp_delay_ns{from="master",iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 0
openshift_ptp_delay_ns{from="phc",iface="CLOCK_REALTIME",node="helix60.lab.eng.rdu2.redhat.com",process="phc2sys"} 508
# HELP openshift_ptp_frequency_adjustment_ns 
# TYPE openshift_ptp_frequency_adjustment_ns gauge
openshift_ptp_frequency_adjustment_ns{from="master",iface="eno1239x",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} -12876
openshift_ptp_frequency_adjustment_ns{from="master",iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 0
openshift_ptp_frequency_adjustment_ns{from="phc",iface="CLOCK_REALTIME",node="helix60.lab.eng.rdu2.redhat.com",process="phc2sys"} -8064
# HELP openshift_ptp_frequency_status -1=UNKNOWN, 0=INVALID, 1=FREERUN, 2=LOCKED, 3=LOCKED_HO_ACQ, 4=HOLDOVER
# TYPE openshift_ptp_frequency_status gauge
openshift_ptp_frequency_status{from="dpll",iface="ens7fx",node="",process="dpll"} 3
# HELP openshift_ptp_gnss_status 0=NOFIX, 1=Dead Reckoning Only, 2=2D-FIX, 3=3D-FIX, 4=GPS+dead reckoning fix, 5=Time only fix
# TYPE openshift_ptp_gnss_status gauge
openshift_ptp_gnss_status{from="gnss",iface="ens7f0",node="",process="gnss"} 3
openshift_ptp_gnss_status{from="gnss",iface="ens7fx",node="",process="gnss"} 3
# HELP openshift_ptp_interface_role 0 = PASSIVE, 1 = SLAVE, 2 = MASTER, 3 = FAULTY, 4 = UNKNOWN, 5 = LISTENING
# TYPE openshift_ptp_interface_role gauge
openshift_ptp_interface_role{iface="ens7f0",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 3
openshift_ptp_interface_role{iface="ens7f1",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 3
openshift_ptp_interface_role{iface="ens7f2",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 3
openshift_ptp_interface_role{iface="ens7f3",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 2
# HELP openshift_ptp_max_offset_ns 
# TYPE openshift_ptp_max_offset_ns gauge
openshift_ptp_max_offset_ns{from="master",iface="eno1239x",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 19
openshift_ptp_max_offset_ns{from="master",iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1
openshift_ptp_max_offset_ns{from="phc",iface="CLOCK_REALTIME",node="helix60.lab.eng.rdu2.redhat.com",process="phc2sys"} 98
# HELP openshift_ptp_nmea_status 0 = UNAVAILABLE, 1 = AVAILABLE
# TYPE openshift_ptp_nmea_status gauge
openshift_ptp_nmea_status{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1
# HELP openshift_ptp_offset_ns 
# TYPE openshift_ptp_offset_ns gauge
openshift_ptp_offset_ns{from="dpll",iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="dpll"} 0
openshift_ptp_offset_ns{from="gnss",iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="gnss"} 5
openshift_ptp_offset_ns{from="master",iface="eno1239x",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 7
openshift_ptp_offset_ns{from="master",iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 0
openshift_ptp_offset_ns{from="phc",iface="CLOCK_REALTIME",node="helix60.lab.eng.rdu2.redhat.com",process="phc2sys"} 7
# HELP openshift_ptp_phase_status -1=UNKNOWN, 0=INVALID, 1=FREERUN, 2=LOCKED, 3=LOCKED_HO_ACQ, 4=HOLDOVER
# TYPE openshift_ptp_phase_status gauge
openshift_ptp_phase_status{from="dpll",iface="ens7fx",node="",process="dpll"} 3
# HELP openshift_ptp_pps_status 0 = UNAVAILABLE, 1 = AVAILABLE
# TYPE openshift_ptp_pps_status gauge
openshift_ptp_pps_status{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="dpll"} 1
# HELP openshift_ptp_process_restart_count 
# TYPE openshift_ptp_process_restart_count counter
openshift_ptp_process_restart_count{config="ptp4l.0.config",node="helix60.lab.eng.rdu2.redhat.com",process="phc2sys"} 1
openshift_ptp_process_restart_count{config="ptp4l.0.config",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 1
openshift_ptp_process_restart_count{config="ts2phc.0.config",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1
# HELP openshift_ptp_process_status 0 = DOWN, 1 = UP
# TYPE openshift_ptp_process_status gauge
openshift_ptp_process_status{config="ptp4l.0.config",node="helix60.lab.eng.rdu2.redhat.com",process="phc2sys"} 1
openshift_ptp_process_status{config="ptp4l.0.config",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 1
openshift_ptp_process_status{config="ts2phc.0.config",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1
# HELP openshift_ptp_threshold 
# TYPE openshift_ptp_threshold gauge
openshift_ptp_threshold{node="helix60.lab.eng.rdu2.redhat.com",profile="grandmaster1",threshold="HoldOverTimeout"} 5
openshift_ptp_threshold{node="helix60.lab.eng.rdu2.redhat.com",profile="grandmaster1",threshold="MaxOffsetThreshold"} 100
openshift_ptp_threshold{node="helix60.lab.eng.rdu2.redhat.com",profile="grandmaster1",threshold="MinOffsetThreshold"} -100
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 2891
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0


## otp linux daemon log: 
ts2phc[50336.804]: [ts2phc.0.config] nmea delay: 105064652 ns
ts2phc[50336.804]: [ts2phc.0.config] ens7f0 extts index 0 at 1702744639.000000000 corr 0 src 1702744639.894966450 diff 0
ts2phc[50336.804]: [ts2phc.0.config] ens7f0 master offset          0 s2 freq      +0
I1216 16:36:42.000225   23019 stats.go:57] state updated for ts2phc =s2
I1216 16:36:42.000286   23019 event.go:412] dpll State s2, gnss State s2, tsphc state s2, gm state s2,
ts2phc[1702744602]:[ts2phc.0.config] ens7f0 nmea_status 1 offset 0 s2
GM[1702744602]:[ts2phc.0.config] ens7f0 T-GM-STATUS s2
phc2sys[50336.853]: [ptp4l.0.config] CLOCK_REALTIME phc offset         0 s2 freq   -8078 delay    518
ts2phc[50336.912]: [ts2phc.0.config] nmea sentence: GNRMC,163642.00,A,3551.01745,N,07849.70772,W,0.000,,161223,,,A,V
I1216 16:36:42.108196   23019 stats.go:57] state updated for gnss =s2
gnss[1702744602]:[ts2phc.0.config] ens7f0 gnss_status 3 offset 5 s2
phc2sys[50336.916]: [ptp4l.0.config] CLOCK_REALTIME phc offset         0 s2 freq   -8078 delay    508
phc2sys[50336.978]: [ptp4l.0.config] CLOCK_REALTIME phc offset         0 s2 freq   -8078 delay    508
phc2sys[50337.041]: [ptp4l.0.config] CLOCK_REALTIME phc offset        -2 s2 freq   -8080 delay    514
phc2sys[50337.104]: [ptp4l.0.config] CLOCK_REALTIME phc offset         2 s2 freq   -8077 delay    507
phc2sys[50337.166]: [ptp4l.0.config] CLOCK_REALTIME phc offset       -16 s2 freq   -8094 delay    517
phc2sys[50337.229]: [ptp4l.0.config] CLOCK_REALTIME phc offset       -14 s2 freq   -8097 delay    508
phc2sys[50337.291]: [ptp4l.0.config] CLOCK_REALTIME phc offset        -2 s2 freq   -8089 delay    513
phc2sys[50337.354]: [ptp4l.0.config] CLOCK_REALTIME phc offset       -12 s2 freq   -8100 delay    509
phc2sys[50337.416]: [ptp4l.0.config] CLOCK_REALTIME phc offset        -9 s2 freq   -8100 delay    508
phc2sys[50337.479]: [ptp4l.0.config] CLOCK_REALTIME phc offset        -6 s2 freq   -8100 delay    513
phc2sys[50337.542]: [ptp4l.0.config] CLOCK_REALTIME phc offset       -16 s2 freq   -8112 delay    515
phc2sys[50337.604]: [ptp4l.0.config] CLOCK_REALTIME phc offset         2 s2 freq   -8099 delay    514
phc2sys[50337.667]: [ptp4l.0.config] CLOCK_REALTIME phc offset        -2 s2 freq   -8102 delay    524
phc2sys[50337.729]: [ptp4l.0.config] CLOCK_REALTIME phc offset         0 s2 freq   -8101 delay    512
phc2sys[50337.792]: [ptp4l.0.config] CLOCK_REALTIME phc offset        -4 s2 freq   -8105 delay    507
ts2phc[50337.804]: [ts2phc.0.config] nmea delay: 107964254 ns
ts2phc[50337.804]: [ts2phc.0.config] ens7f0 extts index 0 at 1702744640.000000000 corr 0 src 1702744640.892071404 diff 0
ts2phc[50337.804]: [ts2phc.0.config] ens7f0 master offset          0 s2 freq      +0
ts2phc[1702744603]:[ts2phc.0.config] ens7f0 nmea_status 1 offset 0 s2
GM[1702744603]:[ts2phc.0.config] ens7f0 T-GM-STATUS s2
I1216 16:36:43.000187   23019 stats.go:57] state updated for ts2phc =s2
I1216 16:36:43.000208   23019 event.go:412] dpll State s2, gnss State s2, tsphc state s2, gm state s2,
phc2sys[50337.855]: [ptp4l.0.config] CLOCK_REALTIME phc offset       -18 s2 freq   -8120 delay    518
ts2phc[50337.895]: [ts2phc.0.config] nmea sentence: GNRMC,163643.00,A,3551.01745,N,07849.70772,W,0.000,,161223,,,A,V
I1216 16:36:43.091104   23019 stats.go:57] state updated for gnss =s2
gnss[1702744603]:[ts2phc.0.config] ens7f0 gnss_status 3 offset 5 s2
phc2sys[50337.917]: [ptp4l.0.config] CLOCK_REALTIME phc offset        -3 s2 freq   -8110 delay    518
phc2sys[50337.980]: [ptp4l.0.config] CLOCK_REALTIME phc offset       -10 s2 freq   -8118 delay    508
phc2sys[50338.042]: [ptp4l.0.config] CLOCK_REALTIME phc offset        -6 s2 freq   -8117 delay    510
phc2sys[50338.105]: [ptp4l.0.config] CLOCK_REALTIME phc offset        -2 s2 freq   -8115 delay    510
phc2sys[50338.168]: [ptp4l.0.config] CLOCK_REALTIME phc offset        14 s2 freq   -8099 delay    508

clones

OCPBUGS-26077 PTP metrics still includes metrics for old config after deleting a ptpconfig

Closed

depends on

OCPBUGS-26238 Backport 2 WPC Cards to 4.14: ptp

Closed

links to

redhat-cne/cloud-event-proxy#284: [release-4.13] Bug OCPBUGS-26508: PTP metrics still includes metrics for old config after deleting a ptpconfig

RHBA-2024:0741 OpenShift Container Platform 4.13.z bug fix update

Assignee:: Aneesh Puttur

Reporter:: OpenShift Prow Bot

QA Contact:: Bonnie Block

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Created:: 2024/01/09 2:33 AM

Updated:: 2024/02/14 6:34 AM

Resolved:: 2024/02/14 6:34 AM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates