Uploaded image for project: 'Migration Toolkit for Virtualization'
  1. Migration Toolkit for Virtualization
  2. MTV-1098

Migration fails if VM has large disks, pod is evicted due to excessive usage of ephemeral storage.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • 2.5.0
    • Controller
    • None
    • False
    • None
    • True

      A migration fails because the v2v pod uses too much space under /var/tmp.

      Source: vmware (using vddk)
      OCP 4.15.10
      MTV 2.6.0
      VM: Windows, with 2 disks (100G OS + 400G data)

      Once the pod starts, look at the / usage inside the pod (so the node /var/lib as this is ephemeral)

      $ df -h
      Filesystem                                                                      Size  Used Avail Use% Mounted on
      overlay                                                                         125G   41G   85G  33% /
      tmpfs                                                                            64M     0   64M   0% /dev
      tmpfs                                                                            16G     0   16G   0% /sys/fs/cgroup
      shm                                                                              64M     0   64M   0% /dev/shm
      tmpfs                                                                           6.3G   75M  6.2G   2% /etc/hostname
      /dev/nvme0n1p4                                                                  125G   41G   85G  33% /opt
      tmpfs                                                                            30G  8.0K   30G   1% /etc/secret
      synology.home.arpa:/volume5/openshift/pvc-6db49023-16ed-46df-a53a-0b9487a06888  960G   35G  926G   4% /mnt/disks/disk0
      synology.home.arpa:/volume5/openshift/pvc-29a477b5-7262-485b-802d-bc31d4c881d0  960G   35G  926G   4% /mnt/disks/disk1
      tmpfs                                                                            30G   20K   30G   1% /run/secrets/kubernetes.io/serviceaccount
      tmpfs                                                                            16G     0   16G   0% /proc/acpi
      tmpfs                                                                            16G     0   16G   0% /proc/scsi
      tmpfs                                                                            16G     0   16G   0% /sys/firmware
      
      

      Some time later:

      $ df -h
      Filesystem                                                                      Size  Used Avail Use% Mounted on
      overlay                                                                         125G   82G   44G  66% /
      tmpfs                                                                            64M     0   64M   0% /dev
      tmpfs                                                                            16G     0   16G   0% /sys/fs/cgroup
      shm                                                                              64M     0   64M   0% /dev/shm
      tmpfs                                                                           6.3G   75M  6.2G   2% /etc/hostname
      /dev/nvme0n1p4                                                                  125G   82G   44G  66% /opt
      tmpfs                                                                            30G  8.0K   30G   1% /etc/secret
      synology.home.arpa:/volume5/openshift/pvc-6db49023-16ed-46df-a53a-0b9487a06888  960G   86G  875G   9% /mnt/disks/disk0
      synology.home.arpa:/volume5/openshift/pvc-29a477b5-7262-485b-802d-bc31d4c881d0  960G   86G  875G   9% /mnt/disks/disk1
      tmpfs                                                                            30G   20K   30G   1% /run/secrets/kubernetes.io/serviceaccount
      tmpfs                                                                            16G     0   16G   0% /proc/acpi
      tmpfs                                                                            16G     0   16G   0% /proc/scsi
      tmpfs                                                                            16G     0   16G   0% /sys/firmware
      

      Because of this 400G file (second disk) sitting on the sysroot, which keeps growing:

      $ ls -lh /var/tmp/v2v/
      total 50G
      lrwxrwxrwx. 1 qemu qemu   25 Apr 30 01:12 windows-sda -> /mnt/disks/disk0/disk.img
      lrwxrwxrwx. 1 qemu qemu   25 Apr 30 01:12 windows-sdab -> /mnt/disks/disk1/disk.img
      -rw-r--r--. 1 qemu qemu 400G Apr 30 01:45 windows-sdb
      
      sh-5.1$ du /var/tmp/v2v/windows-sdb 
      59G	/var/tmp/v2v/windows-sdb
      ...
      sh-5.1$ du /var/tmp/v2v/windows-sdb 
      62G	/var/tmp/v2v/windows-sdb
      

      It won't take a disk too big to fail a migration if the node doesn't have all this free space.

              mnecas@redhat.com Martin Necas
              rhn-support-gveitmic Germano Veit Michel
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: