Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-30649

PTP CI TC failure: PTP Outage recovery The slave node network interface is taken down and up (intermittent)

XMLWordPrintable

    • No
    • False
    • Hide

      None

      Show
      None
    • Hide
      4/15: Currently resolving issues with the pipeline, will close these issues once the pipeline is green again.
      3/21: Keeping this bug open for 3 more weeks to watch any re occurrence
      Show
      4/15: Currently resolving issues with the pipeline, will close these issues once the pipeline is green again. 3/21: Keeping this bug open for 3 more weeks to watch any re occurrence

      Failed TC in 4.13 CI:

      [It] [oc-serial] PTP e2e tests PTP Outage recovery The slave node network interface is taken down and up

      Observed once in the first 2 4.13 periodic runs.

      https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.15-e2e-telco5g-ptp/1762689340414103552/artifacts/e2e-telco5g-ptp/telco5g-ptp-tests/artifacts/test_results_all.html

       

      ft9.20: �[36mINFO   �[0m[Mar  8 18:44:52.931][testconfig.go: 528] Getting ptp configuration for namespace:openshift-ptp 
      �[36mINFO   �[0m[Mar  8 18:44:52.948][testconfig.go: 1016] 2 ptpconfig objects recovered                
      �[37mDEBUG  �[0m[Mar  8 18:44:52.965][ptphelper.go: 578] GetFirstNode err=nodeName not found          
      �[36mINFO   �[0m[Mar  8 18:44:52.982][testconfig.go: 537] Current PTP test config=PtpModeDesired= OC, PtpModeDiscovered= OC, Status= discoverySuccess, DiscoveredClockUnderTestPtpConfig= test-slave1, DiscoveredClockUnderTestSecondaryPtpConfig= nil, DiscoveredGrandMasterPtpConfig= test-grandmaster, DiscoveredSlave1PtpConfig= nil, DiscoveredSlave2PtpConfig= nil, PtpEventsIsSidecarReady= false, DiscoveredClockUnderTestPodName=linuxptp-daemon-hz49k, DiscoveredClockUnderTestNodeName=cnfdf31.telco5gran.eng.rdu2.redhat.com 
      �[36mINFO   �[0m[Mar  8 18:44:52.983][ptp.go: 823] skipINterfacesmap[eno12409:true eno8303:true eno8303np0:true eno8403:true eno8403np1:true eno8503np2:true eno8603np3:true ens6f0np0:true ens6f1np1:true ens7f0np0:true ens7f1np1:true] 
      �[36mINFO   �[0m[Mar  8 18:44:52.983][ptptesthelper.go: 219] Recovery PTP outage begins ...........       
      �[37mDEBUG  �[0m[Mar  8 18:44:53.002][ptphelper.go: 578] GetFirstNode err=nodeName not found          
      �[36mINFO   �[0m[Mar  8 18:44:53.019][ptptesthelper.go: 228] slave node name is cnfdf31.telco5gran.eng.rdu2.redhat.com 
      �[37mDEBUG  �[0m[Mar  8 18:44:53.038][main.go: 391] Is Present err=namespaces "ptp-network-outage-recovery" not found 
      �[36mINFO   �[0m[Mar  8 18:44:53.149][main.go: 177] Checking if the daemonset exists             
      �[36mINFO   �[0m[Mar  8 18:44:53.186][main.go: 180] daemonset ptp-network-outage-recovery does not exist, err=daemonsets.apps "ptp-network-outage-recovery" not found 
      �[36mINFO   �[0m[Mar  8 18:44:53.187][main.go: 228] Creating daemonset ptp-network-outage-recovery 
      �[33mWARNING�[0m[Mar  8 18:44:53.252][main.go: 272] Daemonset ptp-network-outage-recovery (ns ptp-network-outage-recovery) could not be deployed: DesiredNumberScheduled=0 - NodesCount=6 
      �[36mINFO   �[0m[Mar  8 18:44:58.268][main.go: 266] Waiting for (6) daemonset pods to be ready: {CurrentNumberScheduled:6 NumberMisscheduled:0 DesiredNumberScheduled:6 NumberReady:6 ObservedGeneration:1 UpdatedNumberScheduled:6 NumberAvailable:6 NumberUnavailable:0 CollisionCount:<nil> Conditions:[]} 
      �[36mINFO   �[0m[Mar  8 18:44:58.268][main.go: 283] All the debug pods are ready.                
      �[36mINFO   �[0m[Mar  8 18:44:58.268][main.go: 239] Daemonset is ready                           
      �[36mINFO   �[0m[Mar  8 18:44:58.298][main.go: 245] Successfully created daemonset ptp-network-outage-recovery 
      �[36mINFO   �[0m[Mar  8 18:44:58.298][ptptesthelper.go: 244] outage recovery pod name is ptp-network-outage-recovery-7m7sd 
      �[36mINFO   �[0m[Mar  8 18:44:58.298][ptptesthelper.go: 248] Slave interfaces are ["ens2f0"]              
      �[36mINFO   �[0m[Mar  8 18:44:58.298][ptptesthelper.go: 255] Simulating PTP outage using interface ens2f0 
      �[36mINFO   �[0m[Mar  8 18:44:58.298][ptptesthelper.go: 271] Setting the interface ens2f0 down            
      �[36mINFO   �[0m[Mar  8 18:44:58.538][ptptesthelper.go: 273] Interface ens2f0 is set down                 
      > Enter [BeforeEach] [oc-serial] - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/conformance/serial/ptp.go:46 @ 03/08/24 18:44:52.865
      < Exit [BeforeEach] [oc-serial] - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/conformance/serial/ptp.go:46 @ 03/08/24 18:44:52.865 (0s)
      > Enter [BeforeEach] PTP e2e tests - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/pkg/execute/ginkgo.go:9 @ 03/08/24 18:44:52.865
      < Exit [BeforeEach] PTP e2e tests - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/pkg/execute/ginkgo.go:9 @ 03/08/24 18:44:52.865 (0s)
      > Enter [BeforeEach] PTP Outage recovery - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/conformance/serial/ptp.go:800 @ 03/08/24 18:44:52.865
      STEP: Refreshing configuration - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/conformance/serial/ptp.go:801 @ 03/08/24 18:44:52.866
      END STEP: Refreshing configuration - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/conformance/serial/ptp.go:801 @ 03/08/24 18:44:52.983 (117ms)
      < Exit [BeforeEach] PTP Outage recovery - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/conformance/serial/ptp.go:800 @ 03/08/24 18:44:52.983 (117ms)
      > Enter [It] The slave node network interface is taken down and up - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/conformance/serial/ptp.go:810 @ 03/08/24 18:44:52.983
      STEP: toggling network interfaces and syncing - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/conformance/serial/ptp.go:811 @ 03/08/24 18:44:52.983
      STEP: Setting interface down then wait - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/pkg/ptptesthelper/ptptesthelper.go:269 @ 03/08/24 18:44:58.298
      STEP: Checking that the port role is FAULTY after wait. Set the interface UP again and wait - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/pkg/ptptesthelper/ptptesthelper.go:277 @ 03/08/24 18:49:58.539
      END STEP: toggling network interfaces and syncing - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/conformance/serial/ptp.go:811 @ 03/08/24 18:49:58.753 (5m5.77s)
      [FAILED] Unexpected error:
          <*errors.errorString | 0xc0000be0e0>: 
          incorrect role, role expected=3, role observed=1(1)
          {
              s: "incorrect role, role expected=3, role observed=1(1)",
          }
      occurred
      In [It] at: /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/pkg/ptptesthelper/ptptesthelper.go:281 @ 03/08/24 18:49:58.753
      < Exit [It] The slave node network interface is taken down and up - /tmp/cnf-vK8rw/ptp-operator-conformance-test/test/conformance/serial/ptp.go:810 @ 03/08/24 18:49:58.753 (5m5.771s)
       

            nparekh@redhat.com Nishant Parekh
            jacding@redhat.com Jack Ding
            Hen Shay Hassid Hen Shay Hassid
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: