Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-15368

Service DNS resolutions fails in Windows Pods on converted OVN hybrid clusters

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Undefined
    • None
    • 4.13.z, 4.12.z, 4.14.0
    • None
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem

      The service DNS resolution test with Windows workloads has been been observed to fail on ovn-kubernetes-master-e2e-aws-ovn-windows which uses the ipi-aws-ovn-hybrid-toggle workflow. This test does not fail when the ipi-aws-ovn-hybrid-toggle workflow is not used.

      How reproducible: 80-90% of the time
      

      Steps to Reproduce

      1. Bring up an OCP 4.14 cluster with OVN networking on AWS
      2. Convert it to use OVN hybrid networking
      3. Install WMCO from source
      a. Check out the WMCO repo and run:
      b. OPERATOR_IMAGE=quay.io/aravindh/wmco:latest hack/olm.sh -k $private-kye.pem
      c. Bring up a Windows node: hack/machineset apply
      d. Bring up a Windows workload. Please ensure host and container image matches.
      e. Exec into a Windows Pod and try to curl the service DNS enpoint.

      Actual results:

      Endpoint is not reachable
      

      Expected results:

      Endpoint should be reachable
      

      this error shows up in CI as

      === RUN   TestWMCO/network/East_West_Networking/ip-10-0-182-245.ec2.internal/service_DNS_resolution
      W0517 18:22:22.520679   14795 warnings.go:70] would violate PodSecurity "restricted:latest": runAsNonRoot != true (pod or container "win-curler-ec2amaz-4127ogk-dns-test" must set securityContext.runAsNonRoot=true)
          network_test.go:162: 
              	Error Trace:	/go/src/github.com/openshift/windows-machine-config-operator/test/e2e/network_test.go:162
              	Error:      	Received unexpected error:
              	            	job &Job{ObjectMeta:{win-curler-ec2amaz-4127ogk-dns-test-job-c88hn win-curler-ec2amaz-4127ogk-dns-test-job- wmco-test  40749324-1534-4d79-b2de-111566cd5afe 51274 1 2023-05-17 18:22:22 +0000 UTC <nil> <nil> map[controller-uid:40749324-1534-4d79-b2de-111566cd5afe job-name:win-curler-ec2amaz-4127ogk-dns-test-job-c88hn] map[batch.kubernetes.io/job-tracking:] [] [] [{e2e.test Update batch/v1 2023-05-17 18:22:22 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{}},"f:spec":{"f:backoffLimit":{},"f:completionMode":{},"f:completions":{},"f:parallelism":{},"f:suspend":{},"f:template":{"f:spec":{"f:affinity":{".":{},"f:nodeAffinity":{".":{},"f:requiredDuringSchedulingIgnoredDuringExecution":{}}},"f:containers":{"k:{\"name\":\"win-curler-ec2amaz-4127ogk-dns-test\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:nodeSelector":{},"f:os":{".":{},"f:name":{}},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{},"f:tolerations":{}}}}} } {kube-controller-manager Update batch/v1 2023-05-17 18:22:26 +0000 UTC FieldsV1 {"f:status":{"f:active":{},"f:ready":{},"f:startTime":{},"f:uncountedTerminatedPods":{}}} status}]},Spec:JobSpec{Parallelism:*1,Completions:*1,ActiveDeadlineSeconds:nil,Selector:&v1.LabelSelector{MatchLabels:map[string]string{controller-uid: 40749324-1534-4d79-b2de-111566cd5afe,},MatchExpressions:[]LabelSelectorRequirement{},},ManualSelector:nil,Template:{{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[controller-uid:40749324-1534-4d79-b2de-111566cd5afe job-name:win-curler-ec2amaz-4127ogk-dns-test-job-c88hn] map[] [] [] []} {[] [] [{win-curler-ec2amaz-4127ogk-dns-test mcr.microsoft.com/powershell:lts-nanoserver-1809 [pwsh.exe -command for (($i =0), ($j = 0); $i -lt 60; $i++) { $response = Invoke-Webrequest -UseBasicParsing -Uri win-webserver-ec2amaz-4127ogk-deployment-7mfs7-qxb8h.wmco-test.svc.cluster.local; $code = $response.StatusCode; echo "GET returned code $code";If ($code -eq 200) {exit 0}; Start-Sleep -s 10;}; exit 1] []  [] [] [] {map[] map[] []} [] [] nil nil nil nil /dev/termination-log File IfNotPresent nil false false false}] [] Never 0xc0008323b8 <nil> ClusterFirst map[kubernetes.io/os:windows]   <nil>  false false false <nil> PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} []   &Affinity{NodeAffinity:&NodeAffinity{RequiredDuringSchedulingIgnoredDuringExecution:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{},MatchFields:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:metadata.name,Operator:In,Values:[ip-10-0-182-245.ec2.internal],},},},},},PreferredDuringSchedulingIgnoredDuringExecution:[]PreferredSchedulingTerm{},},PodAffinity:nil,PodAntiAffinity:nil,} default-scheduler [{os  Windows NoSchedule <nil>}] []  <nil> nil [] <nil> <nil> <nil> map[] [] <nil> &PodOS{Name:windows,} <nil> [] []}},BackoffLimit:*6,TTLSecondsAfterFinished:nil,CompletionMode:*NonIndexed,Suspend:*false,PodFailurePolicy:nil,},Status:JobStatus{Conditions:[]JobCondition{},StartTime:2023-05-17 18:22:22 +0000 UTC,CompletionTime:<nil>,Active:1,Succeeded:0,Failed:0,CompletedIndexes:,UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],},Ready:*1,},} timed out: [{{ } {win-curler-ec2amaz-4127ogk-dns-test-job-c88hn-nfjdr.1760015908cc78ff  wmco-test  7e8adb22-db7f-4370-99c0-e355abdf4bd6 51256 0 2023-05-17 18:22:22 +0000 UTC <nil> <nil> map[] map[] [] [] [{kube-scheduler Update events.k8s.io/v1 2023-05-17 18:22:22 +0000 UTC FieldsV1 {"f:action":{},"f:eventTime":{},"f:note":{},"f:reason":{},"f:regarding":{},"f:reportingController":{},"f:reportingInstance":{},"f:type":{}} }]} {Pod wmco-test win-curler-ec2amaz-4127ogk-dns-test-job-c88hn-nfjdr 4bbd38a4-0a07-4f6b-b9f4-12b88030c60e v1 51251 } Scheduled Successfully assigned wmco-test/win-curler-ec2amaz-4127ogk-dns-test-job-c88hn-nfjdr to ip-10-0-182-245.ec2.internal { } 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0 Normal 2023-05-17 18:22:22.547892 +0000 UTC nil Binding nil default-scheduler default-scheduler-ip-10-0-164-150} {{ } {win-curler-ec2amaz-4127ogk-dns-test-job-c88hn-nfjdr.1760015964aad38c  wmco-test  ac895667-581b-437e-ab5b-e4d0206f8e8b 51263 0 2023-05-17 18:22:23 +0000 UTC <nil> <nil> map[] map[] [] [] [{kubelet.exe Update v1 2023-05-17 18:22:23 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}} }]} {Pod wmco-test win-curler-ec2amaz-4127ogk-dns-test-job-c88hn-nfjdr 4bbd38a4-0a07-4f6b-b9f4-12b88030c60e v1 51252 spec.containers{win-curler-ec2amaz-4127ogk-dns-test}} Pulled Container image "mcr.microsoft.com/powershell:lts-nanoserver-1809" already present on machine {kubelet ip-10-0-182-245.ec2.internal} 2023-05-17 18:22:24 +0000 UTC 2023-05-17 18:22:24 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {win-curler-ec2amaz-4127ogk-dns-test-job-c88hn-nfjdr.17600159681d7254  wmco-test  f9b86c3c-ccd4-4cb3-b009-4aeeabd09ec5 51264 0 2023-05-17 18:22:24 +0000 UTC <nil> <nil> map[] map[] [] [] [{kubelet.exe Update v1 2023-05-17 18:22:24 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}} }]} {Pod wmco-test win-curler-ec2amaz-4127ogk-dns-test-job-c88hn-nfjdr 4bbd38a4-0a07-4f6b-b9f4-12b88030c60e v1 51252 spec.containers{win-curler-ec2amaz-4127ogk-dns-test}} Created Created container win-curler-ec2amaz-4127ogk-dns-test {kubelet ip-10-0-182-245.ec2.internal} 2023-05-17 18:22:24 +0000 UTC 2023-05-17 18:22:24 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {win-curler-ec2amaz-4127ogk-dns-test-job-c88hn-nfjdr.1760015996a7abbc  wmco-test  a1adcbc0-7eed-4a30-b90d-8ff2baca87c9 51268 0 2023-05-17 18:22:24 +0000 UTC <nil> <nil> map[] map[] [] [] [{kubelet.exe Update v1 2023-05-17 18:22:24 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}} }]} {Pod wmco-test win-curler-ec2amaz-4127ogk-dns-test-job-c88hn-nfjdr 4bbd38a4-0a07-4f6b-b9f4-12b88030c60e v1 51252 spec.containers{win-curler-ec2amaz-4127ogk-dns-test}} Started Started container win-curler-ec2amaz-4127ogk-dns-test {kubelet ip-10-0-182-245.ec2.internal} 2023-05-17 18:22:24 +0000 UTC 2023-05-17 18:22:24 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  }]
      

      Attachments

        Issue Links

          Activity

            People

              jtanenba@redhat.com Jacob Tanenbaum
              paravindh Aravindh Puthiyaparambil
              Anurag Saxena Anurag Saxena
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: