First published on TECHNET on Mar 27, 2017
Hello, Claus here again. By now, you have probably seen some of my blogs and demos on Storage Spaces Direct performance. One of Storage Spaces Direct’s advantages is
networking support that lowers latency and reduces CPU consumption. I often get the question “Is RDMA
for Storage Spaces Direct”. The answer to this question is:
. We support plain-old
as long as it’s 10GbE or better. But let’s look a bit deeper.
Recently we did a performance investigation on new hardware, comparing it with an in-market offering (more about that in another post). We ran the tests with RDMA enabled and RDMA disabled (Ethernet mode), which provided the data for this post. For this investigation, we used
with the following configuration:
10 threads, each thread at queue depth 4 (40 total)
A 10GiB file per thread (“a modest VHDX”) for a total of 100GiB
We used the following hardware configuration:
4 node cluster
Intel S2600WT Platform
2x E5-2699v4 CPU (22c44t 2.2Ghz)
128GiB DDR4 DRAM
4x Intel P3700 NVMe per node
Mellanox CX3 Pro 40Gb, dual port connected, RoCE v2
C States disabled, OS High Performance, BIOS Performance Plan, Turbo/HT on
Windows Server 2016 with January roll-up package
No cache drive configuration
3-copy mirror volume
We are by no means driving this system hard, which is on purpose since we want to show the delta between RDMA and non-RDMA under a reasonable workload and not at the edge of what the system can do.
40,000 additional IOPS with the same workload.
3,500 additional IOPS per percent CPU consumed.
90th percentile write latency
90th percentile read latency
I think there are two key take-away’s from this data:
Use RDMA if you want the absolute best performance. RDMA significantly boosts performance. In this test, it shows 28% more IOPS. This is realized by the reduced IO latency provided by RDMA. It also shows that RDMA is more CPU efficient (27%), leaving CPU to run more VMs.
TCP/IP is no slouch, and absolutely a viable deployment option. While not quite as fast and efficient as RDMA, TCP/IP provides solid performance and is well suited for organizations without the expertise needed for RDMA.