Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-58354

4.19: primary UDN layer2 ingress live migration breaks tcp connections

XMLWordPrintable

    • Quality / Stability / Reliability
    • 5
    • False
    • Hide

      None

      Show
      None
    • True
    • CNV v4.99.0.rhel9-2279
    • Hide

      reproducer:

      create VM and UDN

      --
      apiVersion: v1
      kind: Namespace
      metadata:
        name: test12
        labels:
          k8s.ovn.org/primary-user-defined-network: ""
      ---
      apiVersion: k8s.ovn.org/v1
      kind: UserDefinedNetwork
      metadata:
        name: namespace-scoped
        namespace: test12
      spec:
        topology: Layer2
        layer2:
          role: Primary
          ipam:
            lifecycle: Persistent
          subnets:
            - 203.203.0.0/16
            - 2010:100:200::0/60
      ---
      apiVersion: kubevirt.io/v1
      kind: VirtualMachine
      metadata:
        labels:
          app: iperf-server
        name: vm-a
        namespace: test12
        annotations:
          k8s.ovn.org/open-default-ports: |
            - protocol: tcp
              port: 22
      spec:
        runStrategy: Always
        template:
          metadata:
            name: vm-a
            namespace: test12
            labels:
              app: iperf-server
            annotations:
              k8s.ovn.org/open-default-ports: |
                - protocol: tcp
                  port: 22
          spec:
            domain:
              devices:
                disks:
                - disk:
                    bus: virtio
                  name: containerdisk
                - disk:
                    bus: virtio
                  name: cloudinitdisk
                interfaces:
                - name: isolated-namespace
                  binding:
                    name: l2bridge
                rng: {}
              resources:
                requests:
                  memory: 2048M
            networks:
            - pod: {}
              name: isolated-namespace
            terminationGracePeriodSeconds: 0
            volumes:
            - containerDisk:
                image: quay.io/kubevirtci/fedora-with-test-tooling:v20241128-4d4c8fe
              name: containerdisk
            - cloudInitNoCloud:
                userData: |-
                  #cloud-config
                  password: fedora
                  chpasswd: { expire: False }
                  runcmd:
                    - ["iperf3", "-s", "-D", "--logfile", "/tmp/iperf3.log"]
                networkData: |-
                  version: 2
                  ethernets:
                    eth0:
                      dhcp4: true
                      dhcp6: true
              name: cloudinitdisk
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: iperf3-server-vm-a
        namespace: test12
      spec:
        ports:
        - port: 5201
        ipFamilyPolicy: PreferDualStack
        selector:
          app: iperf-server
        sessionAffinity: None
        type: NodePort 

      Start iperf3 client towards the nodeport service

      iperf3 -p $(kubectl get svc -n test12 iperf3-server-vm-a -o json |jq '.spec.ports[0].nodePort') -c [node ip] -t 0 

      Do a migration with 

      virtctl migrate -n test12 vm-a
      

      During tcp connections the iperf3 client output will be

      [  5]  31.00-32.00  sec  2.00 GBytes  17.1 Gbits/sec    0   2.72 MBytes       
      [  5]  32.00-33.00  sec  2.03 GBytes  17.4 Gbits/sec    0   2.72 MBytes       
      [  5]  33.00-34.00  sec  2.02 GBytes  17.3 Gbits/sec    0   3.16 MBytes       
      [  5]  34.00-35.00  sec  2.01 GBytes  17.3 Gbits/sec    0   3.16 MBytes       
      [  5]  35.00-36.00  sec  1.97 GBytes  16.9 Gbits/sec   47   3.16 MBytes       
      [  5]  36.00-37.00  sec  1.99 GBytes  17.1 Gbits/sec    0   3.16 MBytes       
      [  5]  37.00-38.00  sec  1.96 GBytes  16.8 Gbits/sec    0   3.16 MBytes       
      [  5]  38.00-39.00  sec  1.88 GBytes  16.2 Gbits/sec   60   3.16 MBytes       
      [  5]  39.00-40.00  sec  1.90 GBytes  16.4 Gbits/sec    0   3.16 MBytes       
      [  5]  40.00-41.00  sec  1.95 GBytes  16.8 Gbits/sec    0   3.16 MBytes       
      [  5]  41.00-42.00  sec  1.86 GBytes  16.0 Gbits/sec    0   3.16 MBytes       
      [  5]  42.00-43.00  sec  2.02 GBytes  17.3 Gbits/sec    0   3.16 MBytes       
      [  5]  43.00-44.00  sec  1.91 GBytes  16.4 Gbits/sec    0   3.16 MBytes       
      [  5]  44.00-45.00  sec  1.93 GBytes  16.6 Gbits/sec    1   3.16 MBytes       
      [  5]  45.00-46.00  sec  1.90 GBytes  16.3 Gbits/sec    0   3.16 MBytes       
      [  5]  46.00-47.00  sec  1.93 GBytes  16.6 Gbits/sec    0   3.16 MBytes       
      [  5]  47.00-48.00  sec  1.90 GBytes  16.3 Gbits/sec    0   3.16 MBytes       
      [  5]  48.00-49.00  sec   457 MBytes  3.83 Gbits/sec  264   1.32 KBytes       
      [  5]  49.00-50.00  sec   862 MBytes  7.23 Gbits/sec  1991   3.10 MBytes       
      [  5]  50.00-51.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  51.00-52.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  52.00-53.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  53.00-54.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  54.00-55.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  55.00-56.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  56.00-57.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  57.00-58.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  58.00-59.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  59.00-60.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  60.00-61.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  61.00-62.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  62.00-63.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  63.00-64.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes       
      [  5]  64.00-65.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes  

      That shows that tcp connection is broken

      Show
      reproducer: create VM and UDN -- apiVersion: v1 kind: Namespace metadata:   name: test12   labels:     k8s.ovn.org/primary-user-defined-network: "" --- apiVersion: k8s.ovn.org/v1 kind: UserDefinedNetwork metadata:   name: namespace-scoped   namespace: test12 spec:   topology: Layer2   layer2:     role: Primary     ipam:       lifecycle: Persistent     subnets:       - 203.203.0.0/16       - 2010:100:200::0/60 --- apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata:   labels:     app: iperf-server   name: vm-a   namespace: test12   annotations:     k8s.ovn.org/open- default -ports: |       - protocol: tcp         port: 22 spec:   runStrategy: Always   template:     metadata:       name: vm-a       namespace: test12       labels:         app: iperf-server       annotations:         k8s.ovn.org/open- default -ports: |           - protocol: tcp             port: 22     spec:       domain:         devices:           disks:           - disk:               bus: virtio             name: containerdisk           - disk:               bus: virtio             name: cloudinitdisk           interfaces:           - name: isolated-namespace             binding:               name: l2bridge           rng: {}         resources:           requests:             memory: 2048M       networks:       - pod: {}         name: isolated-namespace       terminationGracePeriodSeconds: 0       volumes:       - containerDisk:           image: quay.io/kubevirtci/fedora-with-test-tooling:v20241128-4d4c8fe         name: containerdisk       - cloudInitNoCloud:           userData: |-             #cloud-config             password: fedora             chpasswd: { expire: False }             runcmd:               - [ "iperf3" , "-s" , "-D" , "--logfile" , "/tmp/iperf3.log" ]           networkData: |-             version: 2             ethernets:               eth0:                 dhcp4: true                 dhcp6: true         name: cloudinitdisk --- apiVersion: v1 kind: Service metadata:   name: iperf3-server-vm-a   namespace: test12 spec:   ports:   - port: 5201   ipFamilyPolicy: PreferDualStack   selector:     app: iperf-server   sessionAffinity: None   type: NodePort Start iperf3 client towards the nodeport service iperf3 -p $(kubectl get svc -n test12 iperf3-server-vm-a -o json |jq '.spec.ports[0].nodePort' ) -c [node ip] -t 0 Do a migration with  virtctl migrate -n test12 vm-a During tcp connections the iperf3 client output will be [  5]  31.00-32.00  sec  2.00 GBytes  17.1 Gbits/sec    0   2.72 MBytes        [  5]  32.00-33.00  sec  2.03 GBytes  17.4 Gbits/sec    0   2.72 MBytes        [  5]  33.00-34.00  sec  2.02 GBytes  17.3 Gbits/sec    0   3.16 MBytes        [  5]  34.00-35.00  sec  2.01 GBytes  17.3 Gbits/sec    0   3.16 MBytes        [  5]  35.00-36.00  sec  1.97 GBytes  16.9 Gbits/sec   47   3.16 MBytes        [  5]  36.00-37.00  sec  1.99 GBytes  17.1 Gbits/sec    0   3.16 MBytes        [  5]  37.00-38.00  sec  1.96 GBytes  16.8 Gbits/sec    0   3.16 MBytes        [  5]  38.00-39.00  sec  1.88 GBytes  16.2 Gbits/sec   60   3.16 MBytes        [  5]  39.00-40.00  sec  1.90 GBytes  16.4 Gbits/sec    0   3.16 MBytes        [  5]  40.00-41.00  sec  1.95 GBytes  16.8 Gbits/sec    0   3.16 MBytes        [  5]  41.00-42.00  sec  1.86 GBytes  16.0 Gbits/sec    0   3.16 MBytes        [  5]  42.00-43.00  sec  2.02 GBytes  17.3 Gbits/sec    0   3.16 MBytes        [  5]  43.00-44.00  sec  1.91 GBytes  16.4 Gbits/sec    0   3.16 MBytes        [  5]  44.00-45.00  sec  1.93 GBytes  16.6 Gbits/sec    1   3.16 MBytes        [  5]  45.00-46.00  sec  1.90 GBytes  16.3 Gbits/sec    0   3.16 MBytes        [  5]  46.00-47.00  sec  1.93 GBytes  16.6 Gbits/sec    0   3.16 MBytes        [  5]  47.00-48.00  sec  1.90 GBytes  16.3 Gbits/sec    0   3.16 MBytes        [  5]  48.00-49.00  sec   457 MBytes  3.83 Gbits/sec  264   1.32 KBytes        [  5]  49.00-50.00  sec   862 MBytes  7.23 Gbits/sec  1991   3.10 MBytes        [  5]  50.00-51.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  51.00-52.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  52.00-53.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  53.00-54.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  54.00-55.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  55.00-56.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  56.00-57.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  57.00-58.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  58.00-59.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  59.00-60.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  60.00-61.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  61.00-62.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  62.00-63.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  63.00-64.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes        [  5]  64.00-65.00  sec  0.00 Bytes  0.00 bits/sec    0   3.10 MBytes  That shows that tcp connection is broken
    • Yes

      If a virtual machine is created using primary UDN layer2 topology and a tcp service on it is exposed on it tcp connections from clients are broken during live migration

              ellorent Felix Enrique Llorente Pastora
              ellorent Felix Enrique Llorente Pastora
              Yossi Segev Yossi Segev
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: