Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-11710

Connection problems with OVN-Kubernetes on OpenShift Container Platform 4.12 on AWS post hibernation

    XMLWordPrintable

Details

    • Moderate
    • No
    • SDN Sprint 235, SDN Sprint 236, SDN Sprint 237, SDN Sprint 238, SDN Sprint 239, SDN Sprint 240, SDN Sprint 241, SDN Sprint 242, SDN Sprint 243, SDN Sprint 244, SDN Sprint 245
    • 11
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, an external neighbor could have its MAC address changed while the cluster was shutting down or hibernating. Although a Gratuitous Address Resolution Protocol (GARP) should notify other neighbors about this change, the cluster would not process the GARP because it was not running. When the cluster was brought back up, that neighbor might not be reachable from the OVN-Kubernetes cluster network because the stale MAC address was being used. This update enables an aging mechanism so that a neighbor's MAC address is periodically refreshed every 300 seconds. (link:https://issues.redhat.com/browse/OCPBUGS-11710[*OCPBUGS-11710*])

      While the cluster is shut down or hibernating, an external neighbor could have its mac address changed. Although a GARP should sent to notify other neighbors about this change, the cluster would not process this GARP as it is not running. Then, when the cluster is brought back up, that neighbor might not be reachable from the OVN cluster network because the old stale mac address is being used. This change enables an aging mechanism so that neighbors mac addresses are periodically refreshed every 300s.
      Show
      * Previously, an external neighbor could have its MAC address changed while the cluster was shutting down or hibernating. Although a Gratuitous Address Resolution Protocol (GARP) should notify other neighbors about this change, the cluster would not process the GARP because it was not running. When the cluster was brought back up, that neighbor might not be reachable from the OVN-Kubernetes cluster network because the stale MAC address was being used. This update enables an aging mechanism so that a neighbor's MAC address is periodically refreshed every 300 seconds. (link: https://issues.redhat.com/browse/OCPBUGS-11710 [* OCPBUGS-11710 *]) While the cluster is shut down or hibernating, an external neighbor could have its mac address changed. Although a GARP should sent to notify other neighbors about this change, the cluster would not process this GARP as it is not running. Then, when the cluster is brought back up, that neighbor might not be reachable from the OVN cluster network because the old stale mac address is being used. This change enables an aging mechanism so that neighbors mac addresses are periodically refreshed every 300s.
    • Bug Fix
    • Done
    • Needs PxE follow up

    Description

      Description of problem:

       

      Version-Release number of selected component (if applicable):

       

      How reproducible:

       

      Steps to Reproduce:

      1.

      2.

      3.

       

      Actual results:

       

      Expected results:

       

      Additional info:

      Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.

      Affected Platforms:

      Is it an

      1. internal CI failure 
      2. customer issue / SD
      3. internal RedHat testing failure

       

      If it is an internal RedHat testing failure:

      • Please share a kubeconfig or creds to a live cluster for the assignee to debug/troubleshoot along with reproducer steps (specially if it's a telco use case like ICNI, secondary bridges or BM+kubevirt).

       

      If it is a CI failure:

       

      • Did it happen in different CI lanes? If so please provide links to multiple failures with the same error instance
      • Did it happen in both sdn and ovn jobs? If so please provide links to multiple failures with the same error instance
      • Did it happen in other platforms (e.g. aws, azure, gcp, baremetal etc) ? If so please provide links to multiple failures with the same error instance
      • When did the failure start happening? Please provide the UTC timestamp of the networking outage window from a sample failure run
      • If it's a connectivity issue,
      • What is the srcNode, srcIP and srcNamespace and srcPodName?
      • What is the dstNode, dstIP and dstNamespace and dstPodName?
      • What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)

       

      If it is a customer / SD issue:

       

      • Provide enough information in the bug description that Engineering doesn’t need to read the entire case history.
      • Don’t presume that Engineering has access to Salesforce.
      • Please provide must-gather and sos-report with an exact link to the comment in the support case with the attachment.  The format should be: https://access.redhat.com/support/cases/#/case/<case number>/discussion?attachmentId=<attachment id>
      • Describe what each attachment is intended to demonstrate (failed pods, log errors, OVS issues, etc).  
      • Referring to the attached must-gather, sosreport or other attachment, please provide the following details:
        • If the issue is in a customer namespace then provide a namespace inspect.
        • If it is a connectivity issue:
          • What is the srcNode, srcNamespace, srcPodName and srcPodIP?
          • What is the dstNode, dstNamespace, dstPodName and  dstPodIP?
          • What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
          • Please provide the UTC timestamp networking outage window from must-gather
          • Please provide tcpdump pcaps taken during the outage filtered based on the above provided src/dst IPs
        • If it is not a connectivity issue:
          • Describe the steps taken so far to analyze the logs from networking components (cluster-network-operator, OVNK, SDN, openvswitch, ovs-configure etc) and the actual component where the issue was seen based on the attached must-gather. Please attach snippets of relevant logs around the window when problem has happened if any.
      • For OCPBUGS in which the issue has been identified, label with “sbr-triaged”
      • For OCPBUGS in which the issue has not been identified and needs Engineering help for root cause, labels with “sbr-untriaged”
      • Note: bugs that do not meet these minimum standards will be closed with label “SDN-Jira-template”

      Attachments

        Issue Links

          Activity

            People

              jcaamano@redhat.com Jaime Caamaño Ruiz
              rhn-support-sreber Simon Reber
              Arti Sood Arti Sood
              Chris Fields, Jaime Caamaño Ruiz
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: