-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
2.11.0
-
Quality / Stability / Reliability
-
False
-
-
False
-
-
Description: We observed a significant performance degradation in MTV 2.11 compared to previous versions (2.9 and 2.10) during a 10 VM cold migration scenario. The duration has increased by approximately 70% (from ~15 mins to ~25 mins).
This test uses AIO buffer enable with the default hard-coded values (16/4) on the provider side.
bash-5.2$ cat /mnt/vddk-conf/vddk-config-file VixDiskLib.nfcAio.Session.BufSizeIn64KB=16 VixDiskLib.nfcAio.Session.BufCount=4
spec:
secret:
name: vsphere-7.0.3-1-secret
namespace: openshift-mtv
settings:
sdkEndpoint: vcenter
useVddkAioOptimization: 'true'
vddkInitImage: quay.io/qiyuan1/test7
To rule out storage hardware issues, we reproduced the test on both our standard NetApp backend and our new IBM storage backend. The results were consistent, indicating the issue lies within the MTV version/configuration rather than the underlying storage.
| MTV Version | Storage Backend | Duration | Status |
| 2.9 | NetApp (Original) | 00:14:47 | Baseline (Good) |
| 2.10 | NetApp (Original) | 00:15:25 | Acceptable (+4% vs Baseline) |
| 2.11.28 | NetApp (Original) | 00:26:41 | REGRESSION (+80% vs Baseline) |
| 2.11.30 | NetApp (Original) | 00:25:36 | REGRESSION (+73% vs Baseline) |
| 2.11.30 | IBM Storage (New) | 00:24:40 | Confirms issue persists on new hardware |
Key Analysis for the Report:
- Significant Regression: The move from 2.10 to the 2.11 stream resulted in a performance drop of over 70%.
- Hardware Agnostic: The issue is reproducible on both the original NetApp storage and the new IBM storage environment (00:24:40), confirming this is a software regression in MTV 2.11 and not related to specific storage hardware performance.
Migration Breakdown example :
MTV 2.11 :
Report Date: 2026-01-15 , 21:42:04
MTV Version: mtv-operator.v2.11.0 , IIB:
MigrationStartTime: 2026-01-15 21:14:48 , MigrationEndTime: 2026-01-15 21:40:24
MigrationName: 10vm-aio-buffer-1disk-50gb-35usage-cold-tc3-0 , Total VMs: 10 , Total Duration: 00:25:36
TargetNamespace: 10vm-aio-buffer-1disk-50gb-35usage-cold-tc3-0 , Total VMs: 10
MigrationType: COLD
MigrationStatus: SucceededVM MigrationTime Initialize DiskAllocation ImageConversion DiskTransferV2v VirtualMachineCreation
-- ------------- ---------- -------------- --------------- --------------- ----------------------
rhel79-50gb-70usage-vm-1 00:25:36 00:00:07 00:00:22 00:01:29 00:23:36 00:00:02
rhel79-50gb-70usage-vm-2 00:25:34 00:00:07 00:00:11 00:01:33 00:23:40 00:00:03
rhel79-50gb-70usage-vm-3 00:25:35 00:00:07 00:00:22 00:01:33 00:23:29 00:00:04
rhel79-50gb-70usage-vm-4 00:25:35 00:00:07 00:00:22 00:01:33 00:23:29 00:00:04
rhel79-50gb-70usage-vm-5 00:24:56 00:00:07 00:00:11 00:01:26 00:23:09 00:00:03
rhel79-50gb-70usage-vm-6 00:25:20 00:00:08 00:00:10 00:01:30 00:23:28 00:00:04
rhel79-50gb-70usage-vm-7 00:25:27 00:00:08 00:00:17 00:01:26 00:23:35 00:00:01
rhel79-50gb-70usage-vm-8 00:25:17 00:00:08 00:00:17 00:01:23 00:23:28 00:00:01
rhel79-50gb-70usage-vm-9 00:25:36 00:00:08 00:00:17 00:01:30 00:23:40 00:00:01
rhel79-50gb-70usage-vm-10 00:25:24 00:00:08 00:00:10 00:01:30 00:23:32 00:00:04
avg 00:25:26 00:00:07 00:00:15 00:01:29 00:23:30 00:00:02
min 00:24:56 00:00:07 00:00:10 00:01:23 00:23:09 00:00:01
max 00:25:36 00:00:08 00:00:22 00:01:33 00:23:40 00:00:04
MTV 2.10 GOOD results :
Report Date: 2025-11-25 , 18:55:39
MTV Version: mtv-operator.v2.10.0 , IIB:
MigrationStartTime: 2025-11-25 18:37:57 , MigrationEndTime: 2025-11-25 18:54:34
MigrationName: 10vm-aio-buffer-1disk-50gb-35usage-cold-tc3-0 , Total VMs: 10 , Total Duration: 00:16:37
TargetNamespace: 10vm-aio-buffer-1disk-50gb-35usage-cold-tc3-0 , Total VMs: 10
MigrationType: COLD
MigrationStatus: SucceededVM MigrationTime Initialize DiskAllocation ImageConversion DiskTransferV2v VirtualMachineCreation
-- ------------- ---------- -------------- --------------- --------------- ----------------------
rhel79-50gb-70usage-vm-1 00:15:45 00:00:09 00:00:11 00:01:15 00:13:56 00:00:00
rhel79-50gb-70usage-vm-2 00:15:38 00:00:09 00:00:11 00:01:11 00:13:52 00:00:01
rhel79-50gb-70usage-vm-3 00:15:39 00:00:09 00:00:11 00:01:11 00:13:56 00:00:00
rhel79-50gb-70usage-vm-4 00:15:24 00:00:09 00:00:07 00:01:08 00:13:46 00:00:00
rhel79-50gb-70usage-vm-5 00:15:38 00:00:09 00:00:11 00:01:14 00:13:49 00:00:00
rhel79-50gb-70usage-vm-6 00:16:38 00:00:10 00:00:15 00:01:16 00:14:46 00:00:01
rhel79-50gb-70usage-vm-7 00:15:39 00:00:10 00:00:11 00:01:04 00:14:03 00:00:00
rhel79-50gb-70usage-vm-8 00:15:39 00:00:10 00:00:15 00:01:09 00:13:56 00:00:00
rhel79-50gb-70usage-vm-9 00:15:39 00:00:10 00:00:11 00:01:14 00:13:53 00:00:00
rhel79-50gb-70usage-vm-10 00:15:40 00:00:10 00:00:15 00:01:09 00:13:56 00:00:01
avg 00:15:43 00:00:09 00:00:11 00:01:11 00:13:59 00:00:00
min 00:15:24 00:00:09 00:00:07 00:01:04 00:13:46 00:00:00
max 00:16:38 00:00:10 00:00:15 00:01:16 00:14:46 00:00:01
MTV 2.9.6 GOOD results :
Report Date: 2025-11-18 , 22:32:22
MTV Version: mtv-operator.v2.9.6 , IIB:
MigrationStartTime: 2025-11-18 22:14:14 , MigrationEndTime: 2025-11-18 22:31:24
MigrationName: 10vm-aio-buffer-1disk-50gb-35usage-cold-tc3-0 , Total VMs: 10 , Total Duration: 00:17:10
TargetNamespace: 10vm-aio-buffer-1disk-50gb-35usage-cold-tc3-0 , Total VMs: 10
MigrationType: COLD
MigrationStatus: SucceededVM MigrationTime Initialize DiskAllocation ImageConversion DiskTransferV2v VirtualMachineCreation
-- ------------- ---------- -------------- --------------- --------------- ----------------------
rhel79-50gb-70usage-vm-1 00:16:26 00:00:11 00:00:09 00:01:13 00:14:44 00:00:00
rhel79-50gb-70usage-vm-2 00:17:11 00:00:11 00:00:15 00:01:18 00:15:13 00:00:00
rhel79-50gb-70usage-vm-3 00:17:06 00:00:11 00:00:09 00:01:20 00:15:16 00:00:00
rhel79-50gb-70usage-vm-4 00:17:11 00:00:11 00:00:15 00:01:18 00:15:14 00:00:00
rhel79-50gb-70usage-vm-5 00:17:03 00:00:11 00:00:09 00:01:13 00:15:20 00:00:00
rhel79-50gb-70usage-vm-6 00:17:07 00:00:11 00:00:16 00:01:07 00:15:20 00:00:00
rhel79-50gb-70usage-vm-7 00:17:07 00:00:11 00:00:19 00:01:17 00:15:07 00:00:00
rhel79-50gb-70usage-vm-8 00:17:03 00:00:12 00:00:09 00:01:13 00:15:20 00:00:00
rhel79-50gb-70usage-vm-9 00:17:07 00:00:12 00:00:16 00:01:18 00:15:10 00:00:00
rhel79-50gb-70usage-vm-10 00:17:07 00:00:12 00:00:16 00:01:21 00:15:07 00:00:00
avg 00:17:02 00:00:11 00:00:13 00:01:15 00:15:11 00:00:00
min 00:16:26 00:00:11 00:00:09 00:01:07 00:14:44 00:00:00
max 00:17:11 00:00:12 00:00:19 00:01:21 00:15:20 00:00:00
- relates to
-
MTV-4077 AIO Buffer Regression for COLD Migration
-
- Closed
-
