LEDBAT is the background data transfer product built into the Windows networking stack and recommended by the Windows Data Transport team for moving bulk data without interfering with foreground traffic. LEDBAT has a couple of advantages that make it our choice for the Windows networking stack. LEDBAT automatically leverages any unused bandwidth in the network and in the blink of an eye relinquishes the bandwidth to any foreground traffic that is present in the network. Also, LEDBAT is what we call a pluggable congestion controller. That means that it is fully integrated into the Windows networking stack at the TCP layer and gives you all of the goodness of Windows TCP in addition to LEDBAT’s background characteristics. Also, LEDBAT is a mature and well tested product. It is already in production use in many Windows products including Windows Updates, System Center Configuration Manager, Windows Error Reporting and OneDrive. (Please check with the individual product to find if LEDBAT is used in a default on, opt in or any other configuration).
Let’s dive in a little deeper on this. (or you can jump down to the TL;DR section at the bottom)
LEDBAT is automatic: This means that when you are moving background traffic across the network and there is no foreground traffic, LEDBAT will leverage all of the bandwidth. When a foreground data flow begins to use the network LEDBAT will sense the foreground data flow in about 60 milliseconds (blink of an eye) and yield the bandwidth to the foreground data flow. When the foreground data flow is complete then LEDBAT will again leverage the full network bandwidth. No interference.
LEDBAT doesn’t require coordination between the server, the network, or the client: This allows configuration where you only control one side of the communication. Consider Figure 1 where Alice is downloading background traffic from Server 1. In the Figure, Alice has LEDBAT configured on her client. This means that the background data flow is LEDBATed. The server doesn’t need to know anything about it. Alternatively, Server 1 could have LEDBAT configured, and the background data flow would be LEDBATed without Alice’s client needed to know anything about it. Or both Alice’s client and Server 1 could have LEDBAT configured, and the result would be the same. Also, LEDBAT requires no support from the network. The routers, switches and other network hardware do not need to know anything about the LEDBAT configuration.
Figure 1 -- LEDBAT in Action
LEDBAT is holistic: This means that LEDBAT senses the network path not the NICs on the local computers. Consider Figure 1 again. Alice is downloading background traffic as before. While she is doing this Bob starts up some foreground traffic which crosses the network path of Alice’s background traffic. LEDBAT senses Bob’s foreground traffic and yields the network bandwidth to Bob. Bob gets to use the network and Alice gets to download her background data.
The secret is in LEDBAT’s quick reaction time. When Bob needs the network bandwidth, Alice’s LEDBATed data flow senses this condition in about 60 milliseconds. Before Bob can blink his eye, the bandwidth is available to him. For instance, if Bob is downloading web pages, he will have the full network bandwidth available for him to receive the web pages and while he is reading the web pages (not using the bandwidth) Alice’s background download will continue until Bob needs bandwidth again.
Make sure that you only LEDBAT Background Data
LEDBAT is for background data connections and only for background data connections. LEDBATing a foreground data connection will lead to unsatisfactory user experience. For instance, if an RDP connection or an SSH is LEDBATed then mouse movements/clicks, and keystrokes will yield to foreground data flows. This is the opposite of what we want to do.
Can I Set Priorities for Data Flows with LEDBAT?
Nope, LEDBAT is not QoS. Connections either use LEDBAT or they don’t. The reason for this is that LEDBAT is an integral part of TCP and the algorithm is carefully balanced. Adding priorities to LEDBAT would require manipulating that balance and add complexity that we don’t want. Also, in order to do priorities right (IMO) in-network participation is required. That means that all the routers and switches in the data path would have to participate. This is a requirement that we don’t want in Windows LEDBAT.
LEDBAT, LEDBAT++ and rLEDBAT
Yes, this is very confusing. To simplify things, we will just call it LEDBAT for Windows or simply LEDBAT. How this plethora of names came about is a longer story. LEDBAT is experimental: RFC 6817: Low Extra Delay Background Transport (LEDBAT) (rfc-editor.org). The original LEDBAT has some issues that prevented us from bringing it to production, so we fixed them. The whole thing with the fixes we called LEDBAT++ (LEDBAT plus our fixes).
Now what about rLEDBAT? Like we said above Windows LEDBAT is an integral part of Windows TCP. Therefore, it has the same characteristics as TCP. Refer to Figure 2. Each side of a TCP connection is both a sender and receiver. Alice’s sender is connected to Server 1’s receiver and Server 1’s receiver is connected to Alice’s receiver. Which all makes perfect sense.
Figure 2 -- TCP Senders and Receivers
However, this makes things complicated for an algorithm like LEDBAT which controls the sending rate in order to yield the network bandwidth. Inside of Windows TCP is a thing called a pluggable Congestion Controller which controls the sending rate. LEDBAT++ is a pluggable Congestion Controller which we built and deployed in Windows Server 2016. This is well and good for applications like SCCM and Windows Error Reporting which control the sending side of the connection. Just plug in LEDBAT++ and start sending. But, what about client-side apps that download stuff from servers that they don’t control or that are very difficult to change? For these apps we need a “side” LEDBAT.
This is where rLEDBAT comes into play. We built rLEDBAT in the Windows Server 2022 era and added it to the Windows LEDBAT family. So, when we say “Windows LEDBAT” we are saying LEDBAT++ and rLEDBAT together which means that all data flow into and out of the LEDBATed connection will behave with LEDBAT style politeness yielding bandwidth to foreground connections.
So, why didn’t we just make both of them together at the same time and avoid the LEDBAT++/rLEDBAT naming complexity? Well, there were many reasons. is that rLEDBAT is not a Congestion Controller like LEDBAT++ because there is no such thing as a receive side CC. Instead, we had to use the TCP flow control mechanism (receive window) which you can read about in my other blog. Also, the LEDBAT algorithm (which you can read about in RFC 6817) has to know the connection Round Trip Time or RTT which measures how long it takes to send data from the TCP sender to the TCP receiver. RTT is automatically measured by TCP for the sender side, however, there is no such measurement for the receive side. To make rLEDBAT we had to use TCP’s flow control for Congestion Control and use TCP timestamps for RTT measurement. All this added complexity made it take longer to develop rLEDBAT and productize it for a fleet as large as Windows.
rLEDBAT and TCP Timestamps
As described above rLEDBAT requires that the server supports TCP timestamps. All modern Operating Systems (AFAIK) support TCP timestamps. However, this does not mean that the admin of a server has to have TCP timestamps enabled. If you intend to use rLEDBAT then you must check to see that the sender does indeed support TCP timestamps and has them turned on. If the sender refuses to use TCP timestamps then rLEDBAT will simply revert to the normal TCP foreground mode. rLEBAT status can be queried as shown in the code snippet below.
How do I configure Windows LEDBAT?
LEDBAT for Windows is configured either with Admin templates or programmatically with a TCP socket option.
Using TCP Templates and Transport Filters
Set-NetTCPSetting -SettingName InternetCustom -CongestionProvider LEDBAT
New-NetTransportFilter -SettingName InternetCustom -Protocol TCP -LocalPortStart 4445 -LocalPortEnd 4445 -RemotePortStart 0 -RemotePortEnd 65535
This will configure a server to LEDBAT any client that connects to it on port 4445. Unfortunately, TCP templates are not available on Client.
Using SIO_PRIORITY_HINT
PRIORITY_STATUS rledbat_status;
SOCKADDR_STORAGE conn_addr = { 0 };
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
DWORD bytes;
int opt_int = SocketPriorityHintVeryLow;WSAIoctl(
clientsock._s,
SIO_PRIORITY_HINT,
(LPVOID)&opt_int,
sizeof(opt_int),
NULL, 0,
&bytes, NULL,
NULL);
if (connect(clientsock, &conn_addr, sizeof(conn_addr) == 0) {
// Successfully connected to the peer.
WSAIoctl(
clientsock._s,
SIO_PRIORITY_HINT,
NULL, 0,
&rledbat_status, sizeof(rledbat_status),
&bytes, NULL, NULL);
// If rLEDBAT is in use, rledbat_status.Receiver == SocketPriorityHintVeryLow
. // Note that LEDBAT++ is successful unconditionally.}
Note that SIO_PRIORITY_HINT must be set before you call accept/connect().
In order to make rLEDBAT more acceptable, we have made changes to the TCP timestamps configuration in Windows Server 2022. Now, the knob is a tri-state with a default value being passive on, which means if the incoming connection request (SYN) attempts to enable TCP timestamps, timestamps will be accepted and used. For outgoing connections, it's still not enabled by default. Also, when we enable low priority on a particular connection, we enable timestamps automatically for the connection. So, the default timestamps configuration should work for both incoming and outgoing connections out of box. For servers running Server 2019, timestamps option is default off even for incoming connection requests unless it's specified by socket option, SIO_PRIORITY_HINT or TCP_TIMESTAMPS.
TL;DR
- LEDBAT enables moving bulk data in the background while not interrupting foreground traffic
- LEDBAT is in use by large services, such as Windows Update and SCCM
- LEDBAT can be configured using an admin template or a socket option
- LEDBAT can be used on either client or server without any configuration needed on the other side (or intermediate network equipment).
- LEDBAT for Windows consists of 2 products. LEDBAT++ which enables send side LEDBAT and rLEDBAT which enables receive side LEDBAT.
- With Windows Server 2016 LEDBAT++ (send side) is supported in both server and client
- With Windows Server 2022 both rLEDBAT and LEDBAT++ are supported in both server and client.
Microsoft
