Description of problem:

We noticed an increase in DNS failures for SNO upgrades, this seems to be a regression since the same error rate was not present in 4.17 and 4.16

We are now passing at 92-93% on SNO in 4.18

4.18 Test Pass Rate 92~93

4.17 Pass Rate 98~99

Version-Release number of selected component (if applicable):

How reproducible:

    This error is happening frequently enough in our CI micro runs.

Ex:
CI Search Link: https://search.dptools.openshift.org/?search=Verify+DNS+availability+during+and+after+upgrade+success&maxAge=48h&context=1&type=bug%2Bissue%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Job Failure Sample: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-origin-29136-ci-4.18-e2e-aws-upgrade-ovn-single-node/1842063899902349312

Steps to Reproduce:

    1.
    2.
    3.
    

Actual results:

Expected results:

Additional info:

Loki Query for Sample Run: https://grafana-loki.ci.openshift.org/explore?orgId=1&left=%7B%22datasource%22:%22PCEB727DF2F34084E%22,%22queries%22:%5B%7B%22expr%22:%22%7Binvoker%3D%5C%22openshift-internal-ci%2Fopenshift-origin-29136-ci-4.18-e2e-aws-upgrade-ovn-single-node%2F1842063899902349312%5C%22%7D%20%7C%20unpack%20%7C%20pod%3D%5C%22dns-test-8ea409cd-2658-473e-9215-5579a4716412-cg47f%5C%22%20or%20namespace%3D%5C%22openshift-dns%5C%22%22,%22refId%22:%22A%22,%22editorMode%22:%22code%22,%22queryType%22:%22range%22%7D%5D,%22range%22:%7B%22from%22:%221727988043189%22,%22to%22:%221728024043191%22%7D%7D

Note that we do see logs intermixed in Loki if you look at the timestamps of when Loki received them and when the pod logs the time, this seems to be due to them being ingested later, this seems to be a side effect of running log collection on SNO during upgrades.