GuestVm high disk response times compare to Hyper-V host 2019

Regular Visitor

Hi,

 

Recently, we had a migration from our old Hyper-V Cluster WS 2012R2 to our new hosts (Dell PowerEdge R740) and the new 2019 Hyper-V cluster. We migrated the VMs with live compute+storage migration. Everything went without problems. After the migration, our customers started to report, that the VMs are very slow after the migration. Our initial analysis showed high disk response times (1000 - 2500ms) monitored from the VM guest. The host is not showing high response times on the CSV disks. All drivers, firmware, and windows updates are updated. We also raised the machine version level and integration services are up to date. The storage array is All-flash and connected through SAN. 

 

We did a disk performance test with diskspd, between the host, and one virtual machine. All machine virtual disks are located on the same CSV disk. Here are the results:

 

diskspd -b64K -d30 -o4 -t8 -Su -r -w25 -L -Z1G -c20G

 

Host:

Results for timespan 1:
*******************************************************************************

actual test time: 30.00s
thread count: 8
proc count: 48

CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 24.11%| 0.83%| 23.28%| 75.89%
1| 22.60%| 0.63%| 21.98%| 77.40%
2| 22.50%| 0.68%| 21.82%| 77.50%
3| 22.71%| 0.89%| 21.82%| 77.29%
4| 22.40%| 0.68%| 21.72%| 77.60%
5| 24.43%| 0.78%| 23.65%| 75.57%
6| 24.17%| 1.20%| 22.97%| 75.83%
7| 22.19%| 0.68%| 21.51%| 77.81%
8| 0.83%| 0.21%| 0.63%| 99.17%
9| 0.26%| 0.21%| 0.05%| 99.74%
10| 0.31%| 0.05%| 0.26%| 99.69%
11| 0.31%| 0.00%| 0.31%| 99.69%
12| 0.16%| 0.10%| 0.05%| 99.84%
13| 0.26%| 0.16%| 0.10%| 99.74%
14| 0.73%| 0.10%| 0.63%| 99.27%
15| 0.68%| 0.42%| 0.26%| 99.32%
16| 1.25%| 0.05%| 1.20%| 98.75%
17| 0.05%| 0.00%| 0.05%| 99.95%
18| 0.99%| 0.16%| 0.83%| 99.01%
19| 0.10%| 0.00%| 0.10%| 99.90%
20| 1.30%| 0.21%| 1.09%| 98.70%
21| 0.83%| 0.00%| 0.83%| 99.17%
22| 0.78%| 0.16%| 0.63%| 99.22%
23| 0.99%| 0.21%| 0.78%| 99.01%
24| 31.04%| 0.73%| 30.31%| 68.96%
25| 0.47%| 0.21%| 0.26%| 99.53%
26| 7.60%| 0.68%| 6.93%| 92.40%
27| 1.09%| 0.26%| 0.83%| 98.91%
28| 4.11%| 0.52%| 3.59%| 95.89%
29| 0.42%| 0.26%| 0.16%| 99.58%
30| 4.06%| 0.52%| 3.54%| 95.94%
31| 0.42%| 0.16%| 0.26%| 99.58%
32| 1.35%| 0.31%| 1.04%| 98.65%
33| 0.73%| 0.16%| 0.57%| 99.27%
34| 3.39%| 0.57%| 2.81%| 96.61%
35| 2.81%| 0.21%| 2.60%| 97.19%
36| 1.46%| 0.57%| 0.89%| 98.54%
37| 0.73%| 0.05%| 0.68%| 99.27%
38| 0.89%| 0.57%| 0.31%| 99.11%
39| 1.20%| 0.78%| 0.42%| 98.80%
40| 3.80%| 0.16%| 3.65%| 96.20%
41| 1.72%| 0.73%| 0.99%| 98.28%
42| 6.51%| 0.42%| 6.09%| 93.49%
43| 0.26%| 0.05%| 0.21%| 99.74%
44| 20.73%| 0.16%| 20.57%| 79.27%
45| 2.81%| 0.31%| 2.50%| 97.19%
46| 7.92%| 2.03%| 5.89%| 92.08%
47| 1.20%| 0.26%| 0.94%| 98.80%
-------------------------------------------
avg.| 6.28%| 0.40%| 5.89%| 93.72%

Total IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 8321761280 | 126980 | 264.54 | 4232.63 | 0.944 | 0.650 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
1 | 8333426688 | 127158 | 264.91 | 4238.57 | 0.943 | 0.621 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
2 | 8330346496 | 127111 | 264.81 | 4237.00 | 0.943 | 0.626 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
3 | 8335917056 | 127196 | 264.99 | 4239.83 | 0.943 | 0.652 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
4 | 8300331008 | 126653 | 263.86 | 4221.73 | 0.947 | 0.690 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
5 | 8350793728 | 127423 | 265.46 | 4247.40 | 0.941 | 0.680 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
6 | 8323858432 | 127012 | 264.61 | 4233.70 | 0.944 | 0.647 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
7 | 8354398208 | 127478 | 265.58 | 4249.23 | 0.940 | 0.652 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
-----------------------------------------------------------------------------------------------------
total: 66650832896 | 1017011 | 2118.76 | 33900.10 | 0.943 | 0.653

Read IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 6231621632 | 95087 | 198.10 | 3169.54 | 0.905 | 0.672 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
1 | 6258753536 | 95501 | 198.96 | 3183.34 | 0.905 | 0.644 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
2 | 6234439680 | 95130 | 198.19 | 3170.98 | 0.904 | 0.642 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
3 | 6275465216 | 95756 | 199.49 | 3191.84 | 0.905 | 0.682 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
4 | 6228082688 | 95033 | 197.98 | 3167.74 | 0.908 | 0.716 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
5 | 6266224640 | 95615 | 199.20 | 3187.14 | 0.903 | 0.718 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
6 | 6245777408 | 95303 | 198.55 | 3176.74 | 0.903 | 0.661 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
7 | 6278217728 | 95798 | 199.58 | 3193.24 | 0.902 | 0.649 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
-----------------------------------------------------------------------------------------------------
total: 50018582528 | 763223 | 1590.04 | 25440.57 | 0.904 | 0.673

Write IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 2090139648 | 31893 | 66.44 | 1063.09 | 1.061 | 0.566 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
1 | 2074673152 | 31657 | 65.95 | 1055.23 | 1.056 | 0.531 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
2 | 2095906816 | 31981 | 66.63 | 1066.03 | 1.060 | 0.559 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
3 | 2060451840 | 31440 | 65.50 | 1047.99 | 1.058 | 0.536 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
4 | 2072248320 | 31620 | 65.87 | 1053.99 | 1.062 | 0.589 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
5 | 2084569088 | 31808 | 66.27 | 1060.26 | 1.056 | 0.536 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
6 | 2078081024 | 31709 | 66.06 | 1056.96 | 1.067 | 0.586 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
7 | 2076180480 | 31680 | 66.00 | 1055.99 | 1.056 | 0.651 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)
-----------------------------------------------------------------------------------------------------
total: 16632250368 | 253788 | 528.72 | 8459.53 | 1.060 | 0.570

 

GuestVM (Windows server 2016):

 

Results for timespan 1:
*******************************************************************************

actual test time: 30.00s
thread count: 8
proc count: 16

CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 1.61%| 0.63%| 0.99%| 98.39%
1| 2.03%| 1.77%| 0.26%| 97.97%
2| 1.72%| 1.30%| 0.42%| 98.28%
3| 2.19%| 0.83%| 1.35%| 97.81%
4| 0.57%| 0.47%| 0.10%| 99.43%
5| 0.31%| 0.26%| 0.05%| 99.69%
6| 0.47%| 0.26%| 0.21%| 99.53%
7| 0.47%| 0.36%| 0.10%| 99.53%
8| 2.29%| 1.46%| 0.83%| 97.71%
9| 0.21%| 0.10%| 0.10%| 99.79%
10| 1.98%| 0.78%| 1.20%| 98.02%
11| 1.72%| 1.67%| 0.05%| 98.28%
12| 1.20%| 0.52%| 0.68%| 98.80%
13| 1.35%| 1.30%| 0.05%| 98.65%
14| 3.91%| 2.50%| 1.41%| 96.09%
15| 0.16%| 0.00%| 0.16%| 99.84%
-------------------------------------------
avg.| 1.39%| 0.89%| 0.50%| 98.61%

Total IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 11206656 | 171 | 0.36 | 5.70 | 688.358 | 945.686 | E:\iotest.dat (20GiB)
1 | 9764864 | 149 | 0.31 | 4.97 | 783.187 | 952.255 | E:\iotest.dat (20GiB)
2 | 9895936 | 151 | 0.31 | 5.03 | 772.906 | 921.165 | E:\iotest.dat (20GiB)
3 | 8519680 | 130 | 0.27 | 4.33 | 898.271 | 939.896 | E:\iotest.dat (20GiB)
4 | 11075584 | 169 | 0.35 | 5.63 | 699.540 | 896.609 | E:\iotest.dat (20GiB)
5 | 9830400 | 150 | 0.31 | 5.00 | 782.480 | 933.612 | E:\iotest.dat (20GiB)
6 | 10289152 | 157 | 0.33 | 5.23 | 738.098 | 929.830 | E:\iotest.dat (20GiB)
7 | 11862016 | 181 | 0.38 | 6.03 | 630.648 | 852.503 | E:\iotest.dat (20GiB)
-----------------------------------------------------------------------------------------------------
total: 82444288 | 1258 | 2.62 | 41.93 | 742.060 | 923.016

Read IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 7929856 | 121 | 0.25 | 4.03 | 389.060 | 723.862 | E:\iotest.dat (20GiB)
1 | 7208960 | 110 | 0.23 | 3.67 | 477.191 | 741.434 | E:\iotest.dat (20GiB)
2 | 7602176 | 116 | 0.24 | 3.87 | 579.250 | 849.748 | E:\iotest.dat (20GiB)
3 | 6225920 | 95 | 0.20 | 3.17 | 708.906 | 894.934 | E:\iotest.dat (20GiB)
4 | 8454144 | 129 | 0.27 | 4.30 | 486.887 | 771.755 | E:\iotest.dat (20GiB)
5 | 7340032 | 112 | 0.23 | 3.73 | 601.953 | 845.077 | E:\iotest.dat (20GiB)
6 | 7798784 | 119 | 0.25 | 3.97 | 600.108 | 840.095 | E:\iotest.dat (20GiB)
7 | 9306112 | 142 | 0.30 | 4.73 | 537.521 | 846.196 | E:\iotest.dat (20GiB)
-----------------------------------------------------------------------------------------------------
total: 61865984 | 944 | 1.97 | 31.47 | 542.452 | 819.194

Write IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 3276800 | 50 | 0.10 | 1.67 | 1412.658 | 1024.290 | E:\iotest.dat (20GiB)
1 | 2555904 | 39 | 0.08 | 1.30 | 1646.254 | 951.272 | E:\iotest.dat (20GiB)
2 | 2293760 | 35 | 0.07 | 1.17 | 1414.739 | 855.258 | E:\iotest.dat (20GiB)
3 | 2293760 | 35 | 0.07 | 1.17 | 1412.261 | 863.605 | E:\iotest.dat (20GiB)
4 | 2621440 | 40 | 0.08 | 1.33 | 1385.345 | 927.101 | E:\iotest.dat (20GiB)
5 | 2490368 | 38 | 0.08 | 1.27 | 1314.560 | 978.064 | E:\iotest.dat (20GiB)
6 | 2490368 | 38 | 0.08 | 1.27 | 1170.222 | 1056.216 | E:\iotest.dat (20GiB)
7 | 2555904 | 39 | 0.08 | 1.30 | 969.727 | 786.905 | E:\iotest.dat (20GiB)
-----------------------------------------------------------------------------------------------------
total: 20578304 | 314 | 0.65 | 10.47 | 1342.155 | 957.000

 

We cannot find the problem. Every comment, help, and solution is welcome.

 

BR,

Ivan

 

0 Replies