Brodde the source code for ntttcp (Windows and Linux) and ctsTraffic is freely available online in GitHub. The links are in the article. You are welcome to scrutinize the code, build your own executables, and test whether the Microsoft numbers and methodologies are accurate or not.

Also, all synthetic network benchmark tools measure theoretical performance. Even iPerf3. These tools measure network stack isolated performance by removing reliance on things like storage, processing time, and other overheads associated with production applications and processes.

What about iperf3 on WSL/WSL2? Do you also not recommend it?

Great question, baskapteijn and Christopher Demicoli. iPerf3 will run natively inside of WSL2. There are no emulation issues there.

But... the WSL2 network traffic will traverse either the NAT or mirrored network subsystem inside of Windows. These are software network layers, which means additional latency and possible performance limitations will be incurred on the WSL2 iPerf3 traffic.

So please keep in mind that the iPerf3 results from within WSL2 will not be the same as Windows native ntttcp/ctsTraffic results. Using iPerf3 in WSL2 would measure WSL2 network performance, not Windows native performance.

WSL1 is Linux syscall emulation on Windows. iPerf3 will kind of run natively, but all the syscalls are emulated. iPerf3 should work, but I have not personally tested it.

I can see why a company would like you to use their own measuring device to do performance testing. Wow this car can do 400mph according to the speedometer!

But seriously, with iPerf I am usually interested in finding the cause for performance issues (usually detecting the worst cases). If that is a bad emulation layer in Windows, that is what I am expecting to find. If MS would like all applications to use some other OS-calls or libraries or complete tool chains, they should make them easily available, portable and compatible. If this has been arranged already, it should not be any problem for MS to just create a pull request for iPerf3 and fix the problem.

This would also fix the problem with all the other applications that are using the same network libraries. I would assume MS have fixed a native version of Git by now, that is not relying on MinGW. But I am sure a lot of other software still do. So for these iPerf3 will tell the truth. While ctsTraffic etc. will give you some theoretical number.

Brodde You are, of course, welcome to recommend whatever you prefer. Just as Microsoft is welcome to recommend whatever we prefer.

There are other ways you can test without viewing source code. It's not like Microsoft can quantum teleport packets between Windows and Linux (could you even imagine how much wealthier we'd be if we could?).

Setup a port mirror (SPAN in Cisco terms), switch logging, and/or a sniffer appliance/tool of your choice between two systems of your choice. Run an ntttcp trace between said systems.  Measure the throughput on the wire and see how it measures against reported results. Or use ntttcp on Linux plus whatever trusted tools and metrics in Linux you prefer to measure generic throughput. It would be rather trivial to certify that the results are accurate since the traffic must traverse mediums that Windows and Microsoft does not control. If you do find a reporting error, because errors do happen, please submit an issue on GitHub, let me know, and I will engage people who can address any issues.

But, please, do not make unfounded accusations here. 

FYI, the iperf.fr webpage has been fixed, so reason 2 can be safely removed 🙂

There is some confusion here. First, some information on iperf 2 which is not the predecessor to iperf 3. Iperf 2 is the continuation of the original iperf that was abandoned in 2010 or so. We in Wi-Fi chips took over maintenance in 2014. It's mostly been rewritten since then though adhering to the major design patterns, e.g. threads and separation of user i/o from traffic i/o. More about the differences here: https://iperf2.sourceforge.io/IperfCompare.html

To your reasons (which don't necessarily apply to iperf 2.)  Also note, that tools really need broad statistical verifications before they can be used broadly and across platforms. This is non trivial and goes beyond code writing.

Reason 1: There really is minimal support by ESNet for any release. They are a research group. The most testing & support, including statistical verification from release to release, comes with iperf 2 because Wi-Fi mfgs have something like 14B devices in the field. We also do test 100G+ networks as chips (NICs and merchant forwarding silicon) support that too.

Reason 2: MingW support does allow for posix calls. Some key things are:

• Some use iperf for network measurements
• Some use iperf to characterize applications that use TCP/UDP or end to end which includes host stacks and app level writes/reads.

The CPU, and hence emulation, should not be the bottleneck for network measurement. If it is, then iperf is not a network test tool anymore. It becomes a syscall limited tool. Second, interactions with the stack can impact interoperability but that usually shows up as latency of syscalls that get seen by the network message passing. Again, not a network issue, but a stack/app issue. So, from a network capacity test perspective, the use of posix emulation shouldn't matter. Latency measurement or CPU constrained systems need more analysis and posix emulation could have impacts.

Reason 3: Yes, iperf.fr is woefully out of date and spreads misinformation. A suggestion is to go the sites for authoritative information. There is much misinformation found on the net, lots and lots of misunderstandings.

Also, your article conflates two different metrics, throughput and latency. Iperf 2 has had a focus on latency, including one way delay, traffic patterns & measurements for years now. Most don't understand latency and those impacts on user experience. More capacity, or a capacity measurement, shouldn't be confused with latency or responsiveness.

Finally, IPerf 2 https://sourceforge.net/projects/iperf2/ is different from the iperf3 found at https://github.com/esnet/iperf Each can be used to measure network performance, however, they DO NOT interoperate. They are completely independent implementations with different strengths, different options, and different capabilities. Both are under active development.

Thanks for starting the discussion.

PS. A localhost test can provide CPU & syscall measurements, e.g. iperf -c localhost -i 1 -e and iperf -s 

JamesKehrYeah, it's a common misconception that a bigger number means better. In iperf's case, the number indicates different development teams with different goals. Iperf 3 is a misnomer as it's a not a follow on to the original iperf.  Iperf 2 is based from the original code.

I'd suggest trying iperf 2 on Windows. We're doing some 2.2.1 work around 64b binaries that should be released soon. Some engineers at Meta are helping. That might be worth evaluating when it's released within a few weeks.

Also, something you might want to mention is that only iperf 2 has statistical versification. We test across lots of test beds and multiple days before doing a release. The tool needs to come up with the same numbers otherwise it's broken. We use control charts to make sure the tool performs the same. My guess is most other tools don't do this level of qualifications of the tool itself.

Finally, it might be good if Windows systems added support for latency testing. This is non trivial but very important. All the features can be found in the man page. You can reach out to me on sourceforge if Windows developers want to help take this on.

Hi James and everyone else in this thread – Bruce Mah here, one of the developers of iperf3 at ESnet. We’ve heard from a lot of people about this blog post and been following the back and forth. Thank you for all the updates to the original post, to iperf.fr for removing the old Windows iPerf3 binaries, and to rjmcmahon for jumping in to clarify the iperf2/iperf3 differences and the “bigger number ≠ better” issue. I have a bit more context to add to this thread. 

A little background on ESnet and iperf3
ESnet (www.es.net) provides scientific networking services to support the U.S. Department of Energy and its national laboratories, user facilities, and scientific instruments complex. We developed iperf3 as a rewrite of iperf2 in order to be able to test the end-to-end performance of networks doing large transfers of scientific data. The primary consumer of iperf3 is the perfSONAR measurement system (https://www.perfsonar.net/), which is widely used in the research and education (R&E) networking community. iperf3 is of course also usable as a standalone tool, which is one of the reasons it’s been released separately on GitHub. Work on iperf3 started in 2009 (the first commit was an import of the iperf-2.0 sources), with the first public release in 2013.

The commit history (and the original iperf2 project maintainer) will confirm that iperf3 was intended essentially as an iperf2 replacement. Thus there was a time during which iperf2 was basically abandonware. We’re happy to see that rjmcmahon has assumed the maintainership of this code base and is actively developing for it.

Linux vs. other operating systems

Most of the high-performance networking that we see in the R&E networking space comes from Linux hosts, so it was natural that this is the main supported platform. Supporting iperf3 on FreeBSD and macOS has ensured some level of cross-platform support, at least to the extent of other UNIX and UNIX-like systems. While we have had many requests to make iperf3 work under Windows, we didn’t have the developer skills or resources to support that — and we still don’t. The fact that iperf3 works on Windows at all is a result of code contributions from the community, which we gratefully acknowledge.

There are many facets to end-to-end application network performance. These include of course routers, switches, NICs, and network links, but also the end host operating system, runtime libraries, and the application itself. To that extent iperf3 does characterize the performance of a certain set of applications designed for UNIX but trying to run (with some emulation or adaptation) in a Windows environment. We completely agree that this may not provide the highest throughput numbers on Windows, compared to a program that uses native APIs.

Iperf3 and Windows

We’re happy to see that iperf.fr has removed the old, obsolete binaries from their Web site. This is a problem that can affect any open-source project, not just iperf3.

As mentioned earlier, we’ve generally accepted patches for iperf3 to run on Windows (or other not-officially-supported operating systems such as Android, iOS, or various commercial UNIXes). These changes have allowed Windows hosts to run iperf3 tests (apparently with sub-optimal performance) against any other instance of iperf3, regardless of operating system.  

If there’s interest on the part of Microsoft in making a more-Windows-friendly version of iperf3, we’d welcome a conversation on that topic. Feel free to reach out to me anytime (https://www.es.net/bruce-mah/). 

Bruce.

Hi Bruce,

Thanks for your informative post. I'm skeptical that posix emulation is the bottleneck for these capacity tests. One can run iperf3 -c localhost and iperf3 -s to check. CPUs tend to run sub nanosecond and from internal cache so this should be fine and off board network i/o is going to be many orders of magnitude slower.

I also don't see a technical reason to go to MSFT tools here. We've been using iperf 2 on Windows systems over a decade now for our Wi-Fi related testing. So have our industry partners. We see no issue, at least with iperf2, from a capacity test perspective. 

I do agree the MSFT could help better support industry standard tools. The gist of this blog post is not to use industry standard tools. I'd be skeptical as a customer or user to follow this suggestion.