Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-39022

Single worker host-to-pod disruption while network operator progressing

XMLWordPrintable

    • Moderate
    • None
    • False
    • Hide

      None

      Show
      None

      TRT tooling has detected a bug where we appear to occasionally be briefly losing host-to-pod networking from a worker to all three masters during the time when the network operator is upgrading.

      The the below job runs, look for host-to-pod 1s outages to multiple endpoints. Closer examination seems to show that the outage is all originating from the same host and only affects new connections.

      Sample job runs:
      https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.17-e2e-azure-ovn-upgrade/1828143251106828288
      Intervals: https://sippy.dptools.openshift.org/sippy-ng/job_runs/1828143251106828288/periodic-ci-openshift-release-master-ci-4.17-e2e-azure-ovn-upgrade/intervals?filterText=&intervalFile=e2e-timelines_spyglass_20240826-195928.json&overrideDisplayFlag=1&selectedSources=OperatorAvailable&selectedSources=OperatorProgressing&selectedSources=OperatorDegraded&selectedSources=KubeletLog&selectedSources=EtcdLog&selectedSources=EtcdLeadership&selectedSources=Alert&selectedSources=Disruption&selectedSources=E2EFailed&selectedSources=APIServerGracefulShutdown&selectedSources=KubeEvent

      https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.17-e2e-azure-ovn-upgrade/1828143244467245056
      Intervals: https://sippy.dptools.openshift.org/sippy-ng/job_runs/1828143244467245056/periodic-ci-openshift-release-master-ci-4.17-e2e-azure-ovn-upgrade/intervals?filterText=&intervalFile=e2e-timelines_spyglass_20240826-200245.json&overrideDisplayFlag=1&selectedSources=OperatorAvailable&selectedSources=OperatorProgressing&selectedSources=OperatorDegraded&selectedSources=KubeletLog&selectedSources=EtcdLog&selectedSources=EtcdLeadership&selectedSources=Alert&selectedSources=Disruption&selectedSources=E2EFailed&selectedSources=APIServerGracefulShutdown&selectedSources=KubeEvent&selectedSources=NodeState

      https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.17-e2e-azure-ovn-upgrade/1828143239379554304
      Intervals: https://sippy.dptools.openshift.org/sippy-ng/job_runs/1828143239379554304/periodic-ci-openshift-release-master-ci-4.17-e2e-azure-ovn-upgrade/intervals?filterText=&intervalFile=e2e-timelines_spyglass_20240826-200958.json&overrideDisplayFlag=1&selectedSources=OperatorAvailable&selectedSources=OperatorProgressing&selectedSources=OperatorDegraded&selectedSources=KubeletLog&selectedSources=EtcdLog&selectedSources=EtcdLeadership&selectedSources=Alert&selectedSources=Disruption&selectedSources=E2EFailed&selectedSources=APIServerGracefulShutdown&selectedSources=KubeEvent&selectedSources=NodeState

      https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.17-e2e-azure-ovn-upgrade/1826046480729772032
      Intervals: https://sippy.dptools.openshift.org/sippy-ng/job_runs/1826046480729772032/periodic-ci-openshift-release-master-ci-4.17-e2e-azure-ovn-upgrade/intervals?filterText=&intervalFile=e2e-timelines_spyglass_20240821-011912.json&overrideDisplayFlag=1&selectedSources=OperatorAvailable&selectedSources=OperatorProgressing&selectedSources=OperatorDegraded&selectedSources=KubeletLog&selectedSources=EtcdLog&selectedSources=EtcdLeadership&selectedSources=Alert&selectedSources=Disruption&selectedSources=E2EFailed&selectedSources=APIServerGracefulShutdown&selectedSources=KubeEvent&selectedSources=NodeState

              jluhrsen Jamo Luhrsen
              rhn-engineering-dgoodwin Devan Goodwin
              Anurag Saxena Anurag Saxena
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: