Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14966

Route 53 DNS Record are taking a long time to propagate to CI clusters

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • 4.13, 4.12, 4.11, 4.14, 4.15, 4.16, 4.17
    • Networking / DNS
    • None
    • Moderate
    • No
    • 1
    • Sprint 243, NE Sprint 258, NE Sprint 259
    • 3
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Through debugging https://issues.redhat.com/browse/OCPBUGS-13810, we discovered that DNS records are not propagating to CI Cluster upstream DNS Servers (or even Google 8.8.8.8 for the matter) within our originally specified time of 5 minutes.
      
      - Increasing the wait time for DNS to resolve to 15-20 minutes also fixed the issue, giving it enough time to resolve.
      - The ELB Hostname as specified by the Service-type Loadbalancer, resolves in a reliable manner, the problem seems to be with Route53 DNS records
      - Directly querying Google's 8.8.8.8 DNS Server also produced unreliable results (on & off resolving until 15 minutes in or so).
      More details can be found in the analysis I wrote up here: https://issues.redhat.com/browse/OCPBUGS-13810?focusedId=22419881&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-22419881
      
      

      Version-Release number of selected component (if applicable):

      4.11 - 4.17

      How reproducible:

      50% ish

      Steps to Reproduce:

      Option 1:
      1. Run an old version of TestAWSELBConnectionIdleTimeout (without https://github.com/openshift/cluster-ingress-operator/pull/944 checked in) in CI
      2. Try until it fails. This PR https://github.com/openshift/cluster-ingress-operator/pull/940 can help with debugging.
      
      Option 2:
      1. Get access to a CI cluster
      2. Create a Route53 DNS record in another AWS account/cluster
      3. See if DNS record resolves inside of the CI Cluster within 5 minutes

      Actual results:

      Route 53 DNS Records sometimes take 15-20 minutes to resolve 

      Expected results:

      Route 53 DNS Records should resolve in other AWS clusters/account within 5 minutes

      Additional info:

       

              gspence@redhat.com Grant Spence
              gspence@redhat.com Grant Spence
              Melvin Joseph Melvin Joseph
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: