Mastering Configuration Manager Bandwidth limitations for VPN connected Clients

Published 04-03-2020 02:40 PM 31.7K Views
Microsoft

Hi, Jonas, Roland and Stefan here!

We are Microsoft Premier Field Engineers (PFEs) based in Germany focused on Microsoft Endpoint Manager related topics. Due to the current situation we get a lot of questions from our customers around Configuration Manger traffic for VPN connected Clients.

 

 

TL;DR:

Providing Content to VPN connected Endpoints can be challenging. While the ideal solution would be VPN split tunneling and content downloads directly from the Internet, the implementation may currently not be possible. Reducing the bandwidth of Distribution Points (DPs) and follow some guidelines can help you to successful distribute patches without overloading your VPN infrastructure until you are able to configure VPN split tunnel.

If you have VPN split tunnel in place just scroll to scenario 1 and follow one of the mentioned links.

 

The challenges:

With the current pandemic outbreak of COVID-19 many of us are working from home forced or voluntarily. In addition to the unknown duration of this pandemic outbreak, this also has the potential to change the way we work in the future. Many organizations might offer their teams additional work from home (WFH) opportunities after the crisis.

Managing your Devices (especially security updates and software installations) is necessary and will become challenging as the majority of your work force will be connected to the corporate network via VPN. Depending on the number of clients even a couple of 100MB security updates will quickly add up to several Gigabyte or Terabyte that needs to be pushed out over your VPN network. Without further consideration you can quickly overload your VPN connection causing other applications to degrade in performance or to completely fail. Trust us, you don't want ConfigMgr to be the reason for network overload.

 

What is a VPN split tunnel configuration?

A VPN split tunnel configuration allows for some traffic to go through the VPN tunnel and for all the other traffic to go directly to the internet. That is extremely useful in a work from home scenario were maybe hundreds of users connect to your VPN gateway and you want to avoid every traffic to go through the VPN, into your on-premises infrastructure and out to the internet again.

A great example is Office 365 with Teams, Outlook and SharePoint traffic. Read about the setup for Office 365 here: How to quickly optimize Office 365 traffic for remote staff & reduce the load on your infrastructure...

 

A simplified VPN split tunnel setup could look like this: 

 

1_split-tunnel-001.PNG

 

In the example picture above the on-premises infrastructure is using a subnet of 10.0.0.0/24 and a VPN subnet of 10.200.2.0/24 for your VPN-Clients.The client in the example is connected your on-premises infrastructure via VPN. Locally on the client you would end up with the LAN or Wi-Fi Adapter connected to your home router and having the IP 192.168.2.52 for example and a second adapter connected to the corporate on-premises VPN gateway having the IP 10.200.2.234 as shown in the picture below:

 

2_Ipconfig.PNG

 

The clients local route table could then look like this:

 

3_route-table.png

 

Everything send to 10.0.0.0/24 will be send to your VPN adapter 10.200.2.234 and therefore to your on-premises infrastructure (second line in the picture) and everything else 0.0.0.0 will be send to your local home router 192.168.2.1 and therefore directly to the internet (first line in the picture). So, the traffic is split and either send to the on-premises infrastructure or the internet directly, hence the name VPN split tunnel configuration.

 

If you are using local outbound firewall rules, you might need to adjust them, to allow traffic to internet resources according to your split tunnel configuration. 

 

If you are using a web proxy for your clients, change your proxy configuration to bypass the download URLs which should go directly to the internet. Otherwise the traffic might be routed through your VPN to the proxy and from the proxy to the internet and is not using the split tunnel bypassing VPN as shown in the picture below:

 

4_split-tunnel-002.PNG

 

Considerations / Preparations

  • Talk to your network team how much bandwidth ConfigMgr is allowed to utilize over VPN
  • We recommend to always use dedicated boundary groups for VPN Addresses to:
    • redirect VPN traffic to specific DPs if you have dedicated DPs for VPN clients
    • be able to report download volume on a per VPN boundary group basis
    • disable Peer Cache for VPN Clients (via Boundary Group Options)
    • use IP range boundaries, because the VPN connection might have a 255.255.255.255 subnet mask - with IP subnet boundaries you would end up having one boundary per client
  • Deadline and start time should not be the same time, to avoid many parallel downloads, because in that case the download will start almost immediately after policy arrival. But if you set the deadline to be in the near future, the download start time will be slightly randomized
  • Plan your deployment strategy based on overall download time and maybe stretch the deployment times
    • Download example:
      • You need to distribute 500MB to 5000 clients over VPN
      • You would need to transfer 500MB * 5000 Clients = ~2,5TB of data through the VPN gateway
      • Assuming you have 200Mbit/s available on the VPN gateway, all clients downloading at the same time would take around 24h
        • Keep in mind that there is also some protocol overhead and the actual download time could be higher
      • So, the VPN would not be available for any other traffic for around 24h, assuming no quality of service (QoS) is in place
    • Consider enabling BranchCache on your DPs and Clients. This will help to reduce the amount of data that needs to be downloaded over VPN, by leveraging the Data Deduplication feature of BranchCache. You can find more technical details in the Speed-up Content Distribution with PullDPs and BranchCache Blog, as the technique behind the feature applies to this VPN scenario as well.
    • Don’t forget the proxy configuration as mentioned in the split tunnel section above

 

Scenario #1: With VPN split tunnel or no VPN at all

If your clients are directly connected to the internet or if you have a VPN split tunnel configuration in place, you can make use of the modern content delivery methods described in the below articles:

 

Managing ConfigMgr clients on the Internet:

Managing remote machines with cloud management gateway in Microsoft Endpoint Configuration Manager

 

Patching ConfigMgr clients on the Internet:

Managing Patch Tuesday with Configuration Manager in a remote work world

 

For Office 365 traffic:

How to quickly optimize Office 365 traffic for remote staff & reduce the load on your infrastructure...

 

Office 365 VPN testing:

Office 365 Network Onboarding tool POC updated with VPN testing

 

 

Scenario #2: Without VPN split tunnel and with dedicated DPs for VPN subnets

You are not able to configure VPN split tunnel yet and you have dedicated DPs for all your VPN clients available.

As mentioned in the considerations section, always use dedicated boundary groups for VPN clients and set your dedicated DPs as possible sources.

In that scenario we really need to make sure that the VPN connection will not be stressed with all your clients downloading content and clogging up the VPN connection (as shown in the left picture below), so that business critical traffic can still flow as needed.

There are different ways to achieve this, but from a ConfigMgr admin perspective, the easiest way to achieve this is by setting a maximum bandwidth limit on the IIS of the DPs or set a QoS policy which we describe in scenario 3.

Let's assume we have 200Mbit/s on the VPN gateway overall and we need 100Mbit/s for business-critical data.

If we limit the bandwidth on our DPs to a combined maximum of 100Mbit/s, we will still have the business critical 100Mbit/s left on the VPN gateway. (as shown on the right picture below)

 

5_split-tunnel-003.PNG

 

To configure the IIS bandwidth limitation, just open the IIS Manager on your DP,

select the Default Web Site => Advanced Settings => Limits => Maximum Bandwidth

The technical reference for IIS Limits can be found here.

 

6_IIS_Limits.jpg

 

Please keep in mind that the limit is configured in Bytes per Second. To help you with the conversion from Megabit to Bytes, you can use the Windows 10 Calculator by choosing Data as shown in the picture below:

 

7_Calculator.jpg

 

Scenario #3: Without split tunnel, but NO dedicated DPs for VPN subnets:

The challenges are the same as in scenario 2 and we need to keep business critical data flowing, but you don't have dedicated DPs just for VPN Clients. Instead the DPs are also used by clients on your on-premises infrastructure and a bandwidth limit as in scenario 2 might not be the right choice, because that would also limit the on-premises bandwidth.

Instead we could use local QoS policies directly on the DPs and just limit the bandwidth for every subnet for VPN clients.

 

8_split-tunnel-004.PNG

 

To configure a local QoS Policy, go to Start => Run => gpedit.msc

In the Local Group Policy Editor, expand Computer Configuration => Windows Settings => Policy-based QoS

 

9_QoS-Policy1.jpg

 

Do a right click and create a new policy.

Give it a name and specify the Outbound Throttle Rate:

 

10_QoS-Policy2.jpg

 

Please keep in mind that the limit is configured in Kilobytes or Megabytes per second and not Kilobit or Megabit per second.

 

Click Next, select All applications and Next

Configure the destination IP Range of your VPN Clients. Only this Range will be affected by the throttling:

 

11_QoS-Policy3.jpg

 

 

Click Next and Finish to complete the configuration. The throttling limit will be in place right away.

 

12_QoS-Policy4.jpg

 

If you have multiple DPs or VPN ranges, you need to configure this accordingly on each DP and for each subnet.

The technical reference for QoS Policies can be found here.

 

Other configuration options:

BITS Throttling:

There is the possibility of using BITS Throttling via Client Settings or GPO, but we would advise against it. First, even if you limit the download speed of each client to 1MBit/s, you can still overload your VPN gateway. Second, all downloads via BITS would be throttled, not only ConfigMgr Traffic.

LEDBAT:

LEDBAT can be an easy solution to get your Traffic concerns out of the way with a single checkbox. While this is a strength of this solution, it is also the only configuration option you have - you can only turn it on or off. When it works it is great, but if your VPN gateway is under high load, VPN connected clients might not be able to download important security updates at all. Therefore, if you want to use LEDBAT, monitor that your clients are still able to download content. More technical Details about LEDBAT can be found here.

Express Updates:

Before Windows 10 1803 you had the possibility to use Express Updates to reduce the amount of data that needs to be downloaded from a client. If most or all your clients are above 1803, don´t consider enabling this feature. It will add a lot of overhead to your infrastructure and clients.

 

Conclusion:

The best case would be to have a VPN split tunnel configuration or a direct internet connection and consume as much as possible directly from internet resources to minimize pressure on the VPN gateway and to have the most flexibility when using modern delivery technologies.

But since sometimes configuring VPN split tunnel can be challenging for some customers, one of the above solutions can help you on the way to a modern managed client.

 

Stay safe and split the tunnel 13_Vulcan.jpg

 

Roland Spindeler

Premier Field Engineer - Microsoft Germany

 

Jonas Ohmsen

Premier Field Engineer - Microsoft Germany

 

Stefan Röll

Premier Field Engineer - Microsoft Germany

 

 

Disclaimer:

This posting is provided "AS IS" with no warranties, and confers no rights

 

Changelog:

V1.0 - 04/03/2020 - initial release

V1.1 - 04/06/2020 - added local firewall rules, IP Range for BG, Peer Cache BG Options and Express Updates section

V1.2 - 04/21/2020 - added Office 365 VPN testing link

 

References:

Speed-up Content Distribution with PullDPs and BranchCache

https://techcommunity.microsoft.com/t5/premier-field-engineering/speed-up-content-distribution-with-...

Limits for a Web Site

https://docs.microsoft.com/en-us/iis/configuration/system.applicationhost/sites/site/limits

Quality of Service (QoS) Policy

https://docs.microsoft.com/en-us/windows-server/networking/technologies/qos/qos-policy-top

How to quickly optimize Office 365 traffic for remote staff & reduce the load on your infrastructure https://techcommunity.microsoft.com/t5/office-365-blog/how-to-quickly-optimize-office-365-traffic-fo...

Office 365 Network Onboarding tool POC updated with VPN testing:

https://techcommunity.microsoft.com/t5/office-365-networking/office-365-network-onboarding-tool-poc-...

Managing remote machines with cloud management gateway in Microsoft Endpoint Configuration Manager

https://techcommunity.microsoft.com/t5/configuration-manager-blog/managing-remote-machines-with-clou...

Managing Patch Tuesday with Configuration Manager in a remote work world

https://techcommunity.microsoft.com/t5/configuration-manager-blog/managing-patch-tuesday-with-config...

Top 10 Networking Features in Windows Server 2019: #9 LEDBAT - Latency Optimized Background Transport

https://techcommunity.microsoft.com/t5/networking-blog/top-10-networking-features-in-windows-server-...

12 Comments
New Contributor

Nice document 

Thank you for Sharing with the Community, Awesome blogpost :cool:

Visitor

This is amazingly explained :thumbs_up:

Senior Member

Nice article at the right time 

Senior Member

Hi Roland, 

 

well done, nice article! We are facing this problem exactly and using CDP to solve this issue :)

 

best regards,

Min

Senior Member

Hi Guys,

Great Article for Modern client Management.

Regular Visitor
This is really very help full info. Thanks team appreciated your efforts...
New Contributor

This is a great read. What about VPN clients in a split-tunnel configuration that are pulling updates from CMG / MS Windows Update servers, how can we throttle the bandwidth used for patches being pulled from these sources so it does not overload the clients local internet speed (laptop users working remotely from their home).

Microsoft

Hi Mirza,

downloads use BITS to get the content. BITS is designed to avoid overload from your local internet connection by design:

https://docs.microsoft.com/en-us/windows/win32/bits/background-intelligent-transfer-service-portal

Therefore, you don´t need to configure anything for your use case.

Cheers,

Stefan

Excellent article guys.

 

Could you elaborate more on this bit, please?

 

Express Updates:

Before Windows 10 1803 you had the possibility to use Express Updates to reduce the amount of data that needs to be downloaded from a client. If most or all your clients are above 1803, don´t consider enabling this feature. It will add a lot of overhead to your infrastructure and clients.

 

Cheers.

Microsoft

Express Updates should only be enabled with a planning and testing phase. 

In the current situation most customers we are talking to, don´t have the time for that and need quick solutions. 

Please keep in mind that every Win10 Update (before 1803) will be about 5GB in size with Express Updates enabled. If you have a lot of DPs this can cause some bandwidth challenges. 

The effort is better invested in bringing your clients above 1803, as cumulative updates are only about 200-300MB in newer Windows 10 versions.

Cheers,

Stefan

Thanks Stefan.  Sorry I'm more confused than ever now.

 

What you're saying is what I assumed; older versions of W10 will have much larger CUs, due to their age and thus large amount of CUs rolled into the latest ones.

 

The newer OS versions have much smaller CUs.

 

But then does this mean:

 

Express Updates:

Before Windows 10 1803 you had the possibility to use Express Updates to reduce the amount of data that needs to be downloaded from a client. If most or all your clients are above 1803, don´t consider enabling this feature. It will add a lot of overhead to your infrastructure and clients.

 

... that because of the smaller size of the CUs (1803 onwards), it's simply not worth implementing Express Updates?

 

I was reading it as; there is a larger compute, storage and bandwidth cost related to the primary DP to additional DPs data transfer - for 1803 onwards.

Version history
Last update:
‎Apr 21 2020 02:44 AM
Updated by: