Managing remote machines with cloud management gateway in Microsoft Endpoint Configuration Manager
Published Mar 18 2020 01:17 PM 146K Views
Microsoft

In light of the global situation that has escalated over the past weeks regarding COVID-19 and the coronavirus; there has been a significant increase in the number people working from home. Indeed, myself and the rest of the Microsoft Endpoint Manager team are among 100,000+ Redmond based Microsoft employees who are entering our third week of remote work.

 

This increase in the global workforce working from home is unsurprisingly putting an added focus from organizations on remote functionality and management. Not to mention an increased load and strain on services that were implemented to accommodate lower concurrent numbers of remote working employees.

 

Naturally we have seen an increase in the number of queries, questions and tweets around the tools and features Microsoft Endpoint Manager can offer in the way of remote management of the workforce. One of the most common topics I have had to field enquiries is around the use of cloud management gateway (CMG), usually in conjunction with keeping traffic off the VPN.

 

Firstly, let’s clarify some terms….

 

Internet-based client management is a longstanding concept in Configuration Manager whereby servers are placed in the DMZ and published to the Internet to allow clients to continue to be managed when roaming on the Internet.

 

Cloud management gateway, or as I shall refer to it in the rest of the blog, CMG for short, is a cloud service hosted in Azure that acts as a proxy for clients. It greatly simplifies the configuration required to manage clients on the Internet.

 

The final concept is cloud distribution point, also a cloud service hosted in Azure, that allows clients to retrieve content. For the purposes of simplicity, and because cloud distribution point has been deprecated in favor of enabling content distribution from a CMG, I will use the term “CMG” to refer to a content-enabled cloud management gateway for the remainder of this blog

 

Secondly, let’s talk about why clients will potentially still communicate over the VPN when a CMG is deployed. Essentially, the Configuration Manager client has logic that looks at several factors, including being able to resolve a management point and the internal domain. When these factors are not met, the client will evaluate as IsInternet=1 and will communicate with resources published to the Internet.  When a client is connected to a VPN it is likely that the client will meet enough criteria to consider itself IsInternet=0 which is why client traffic will go over the VPN and not the Internet even if split tunneling is configured to allow direct Internet traffic.

 

NOTE: Everything in this blog will require a split-tunnel VPN. If all the traffic is directed back to the corporate network by the VPN client, then even if the Configuration Manager client is ultimately going out to cloud services, it won’t be alleviating VPN traffic.

 

The good news is that there are a couple of configuration options that you can take to move traffic away from the VPN and directly to Internet sources. These options should hopefully free up some bandwidth for line of business traffic whilst ensuring clients remain managed and up to date.

 

When the VPN has a known IP range

 

If your VPN clients are sat neatly in a known IP range or ranges, then firstly you need to create boundaries in Configuration Manager to cover the VPN ranges:

 
 

Rob York_6-1584492420485.png

 

and then add them to a boundary group:

 

Rob York_1-1584492331636.png

 

Then you need to configure that boundary group to use cloud services. You do this on the references tab, to explicitly accommodate the CMG with the boundary group:

 

Rob York_2-1584492331659.png

 

And also on the options tab select  Prefer cloud based sources over on-premise sources

 

Rob York_3-1584492331671.png

 

This option will apply even if you don’t have a CMG, so can offer some respite to your VPN by directing clients to Microsoft Update for content.

 

When the VPN doesn’t have a known IP range

 

Admittedly this complicates matters, but we added the concept of default site boundary group in version 1610 as a replacement to the concept of fallback content location. This behavior means that if your VPN clients do not fall into a known boundary group, they can fallback to communicate with referenced site systems from the default site boundary group.

 

Again, add the CMG to the references tab

 

Rob York_4-1584492331682.png

 

NOTE: This will result in clients in the corporate network, but not in a known boundary, to connect to the CMG.

 

Force the client to Always Internet mode

 

If networking or boundary configuration makes either of the first two options unviable, you can always force the client to always consider itself IsInternet=1, effectively overriding the logic I talked about earlier. Toggling the client back and forth from explicitly Always Internet is not possible, hence why we make the previous options available. If needed, as a matter of last resort, you could (re)deploy the client using the CCMALWAYSINF parameter to ensure your remote clients are always managed by the CMG.

 

Finally, I wanted to call out an implementation within the Configuration Manager client when it comes to Microsoft Updates. You do not need to deploy your Microsoft software updates packages to the CMG: If a client is on the Internet communicating to a CMG, it will instead retrieve updates from Microsoft Updates. As long as the client can download directly from Microsoft Updates it will never download Microsoft updates from a CMG. Although, a good practice is to not deploy updates packages to a CMG that contain Microsoft Updates.

 

We had previously blocked the deploying of update packages to CMG and CDP for this very reason, but we relaxed the restriction in order to facilitate third party updates.

 

 

To allow clients to use cloud sources for Microsoft Update content, ensure you select the “If software updates are not available on distribution point in current, neighbor or site boundary groups, download content from Microsoft Updates” check box on the updates deployment:

 

Rob York_5-1584492331712.png

 

Rob York

@robdotyork

Program Manager

Microsoft Endpoint Manager

70 Comments
Copper Contributor

Is it possible to just manage Windows Updates through these methods? Is there a way to manage standard content via on-prem and Winodws Updates via CMG / Internet?

Brass Contributor

This is a cool way only if the computer are not under AlwaysOnVpn device force-tunnel mode. I always say this to my customers first by listing the pros and cons between aovpn device and CMG. They generally choose aovpn for better mgmt and fully netlogon approach into the DC. 
No more errors in trust relationship between workstations domain for "fully away" users ;)


Cheers

Copper Contributor

We have still Windows 10 1709, I now we are late! Also with the cloud distribution point it's hard to upgrade all devices until April 14. Still 2000 devices left. Employee can't go back to work during the quarantine time to change their devices (a few devices need to be replaced). 

Microsoft can you please postpone the end-of- life for this build during the COVID-19 days. Two more months security updates would help a lot.

 

 

Edit:

Oh great news! Thank you guys :hearteyes:

https://techcommunity.microsoft.com/t5/windows-it-pro-blog/revised-end-of-service-date-for-windows-1...

Copper Contributor

For those of us without CMG, if you create the VPN boundary group and configure it to prefer cloud resources do you need to associate site system servers with it or can that be left blank since it prefers the cloud anyways.

Copper Contributor

You might want to turn off P2P for that boundary group too if using Peer Cache ;)

Copper Contributor

Where can I find the IP addresses of the Windows updates servers to include in the split tunneling rules (can only find URL's or the whole MS IP address space) !

Microsoft

@Doogle2006 there is no list available with IPs addresses for WU. The WU endpoints are distributed across the world with different CDNs and there is no possibility to provide/maintain a list of the IPs.

Copper Contributor

Anything to add for clients who are on Direct Access? Gotcha's when it comes to ADRs?

Brass Contributor

@robdotyork We've been implementing CMG (using Enhanced HTTP + Azure AD) and are happy to see already quite some traffic from the Cloud DP's.
However, we run into an issue where clients using the CMG as management point, don't see user-targeted applications in their Software Center, and in the SCClient logs it shows: 

 

Using endpoint Url: https://*********.CLOUDAPP.NET/CCM_Proxy_MutualAuth/72057594037927951:443/CMUserService_WindowsAuth, Windows authentication (Microsoft.SoftwareCenter.Client.Data.ACDataSource+<>c at <RefreshLocalSettingsAsync>b__16_0) SCClient 3/26/2020 12:33:19 PM 5 (0x0005)

GetApplicationsAsync: The HTTP request was forbidden with client authentication scheme 'Negotiate'.. Unable to fetch user categories, unknown communication problem.      (Microsoft.SoftwareCenter.Client.ViewModels.SoftwareListViewModel+<LoadAppCatalogApplicationsAsync>d__164 at MoveNext)
 
Any ideas on what I'm missing? All the rest seems to work fine. 
Copper Contributor

We have the same issue with user targeted apps and the 'Negotiate' error.  It seems since the client thinks it is on the intranet with a split tunnel VPN instead of the internet that it tries to authenticate to the CMG with some method other than PKI which fails.  Disconnecting the VPN to force the client into internet mode shows proper PKI authentication and user apps work fine.  Any suggestions to resolve would be appreciated, we are working with premier support, but not making any progress.

Brass Contributor

@Greg Neveau Well at least there will be 2 cases with premier support then, I'm opening one this morning. Perhaps with more cases it will get more attention :) 

Copper Contributor

We can use subnets instead of of IP ranges right? 

Brass Contributor

@Greg Neveau and @Andy D'Hollander i think we have the same issue.  You make any headway on it?

Copper Contributor

No headway for us, we are working with support on getting updates to work via the CMG when the client is in intranet mode and then have a case waiting with support to work on the negotiate error.  

Microsoft

@Chris Calaf  yes. i just chose ranges for the purposes of screenshot

Microsoft

@Greg Neveau @Nick Wiley @Andy D'Hollander we're investigating if you have a case open get your support person to email me the ccm\logs folder from your client.

Brass Contributor

@Rob York I opened up a case with Premier Support this morning but still have heard anything... I can zip the client logs I backed up yesterday and attach them to the case, and let you know the case number if that helps :)  

Copper Contributor

@Rob York , we will have our TAM loop you in on the cases.  We have two open, the first dealing with software updates failing and the second for intranet clients and authentication.  Thank You.

Copper Contributor

@Rob York  @Andy D'Hollander @Greg Neveau we have the same Problem. That the Internet Clients (with no VPN) only reach the Device Software or installed software before. But the Available User Software not showing up in the Internet. In the "Intranet" Modus with VPN Connection the User Software aviable is showing up normaly. We have testet it with Hybrid Join Device an the right clients setting with our partner from switzerland ITNETX had we correctly set. Should we open a case too?

Copper Contributor

@romanmensch, I think you are seeing the opposite of us where our clients work on the internet and not on the intranet.

 

We are noticing that when the client is in intranet mode ( on VPN ), we see in our SCClient logs that the configuration manager client is trying to use windows authentication to the CMG which fails.

Using endpoint Url: https://XXXXXXXX.CLOUDAPP.NET/CCM_Proxy_MutualAuth/XXXXXXXX:443/CMUserService_WindowsAuth, Windows authentication (Microsoft.SoftwareCenter.Client.Data.ACDataSource+<>c at <RefreshLocalSettingsAsync>b__16_0)

 

When in Internet mode, we see the configuration manager client using AAD auth to the CMG which succeeds.

Using endpoint Url: https://XXXXXXXX.CLOUDAPP.NET/CCM_Proxy_ServerAuth/XXXXXXXX/CMUserService, AAD authentication (Microsoft.SoftwareCenter.Client.Data.ACDataSource+<>c at <RefreshLocalSettingsAsync>b__16_0)

 

We are still working with support on this issue.

Brass Contributor

@Greg Neveau @romanmensch Indeed, we have the same issue as Greg :) Actually on a support call with Microsoft at the moment. If it leads to anything I’ll let you know. 

Brass Contributor

@Greg Neveau @Nick Wiley @romanmensch 
Here it goes! 
Basically, when a client is able to reach an on-premise domain controller and considered to be on the "intranet", it needs to receive the client policies from an on-premise Management Point, not a CMG. So the only option is to add an on-premise MP in the boundary group(s) you have configured, and enable the checkbox to have the client prefer cloud sources over on-premise sources. 
Which is indeed how we had set it up initially, but unfortunately that checkbox only applies to applications, not software updates. 
So in order to have VPN clients download update content from Microsoft Update instead of the local DP (which in our case is on the MP we had to add back in the boundary group), we'll have to split up our deployments and work with the download settings to prevent it from downloading from the local DP, and fallback to MS Update for content on the deployments targeting VPN connected devices... 


@Rob York I can feel some UserVoice requests in the air :)  
And that also means that this item on Microsoft Docs needs some more details: https://docs.microsoft.com/en-us/configmgr/core/servers/deploy/configure/boundary-groups#bkmk_bgopti... 

Copper Contributor

@Andy D'Hollander+ others: Please post a new comment if you find a solution or workaround. We have the same problem...

Microsoft

@Andy D'Hollander i cover the implementation logic around IsInternet=1 at the beginning of the blog. but it is not correct to say that "the only option is to add an on-premise MP in the boundary group"

have you added the CMG to the Boundary group?

 

Rob York_2-1584492331659.png

 
 

 

 

Microsoft

@Greg Neveau i responded on email but replying here for broader benefit

 

 

 

If the client is in a known boundary then SUP needs to be configured to be in the client’s boundary group https://docs.microsoft.com/en-us/configmgr/sum/plan-design/plan-for-software-updates#BKMK_SUPSwitchi..., OR in the fallback chain from the current boundary group https://docs.microsoft.com/en-us/configmgr/core/servers/deploy/configure/boundary-groups#fallback

Brass Contributor

@Rob York Yes we did add only the CMG in the VPN boundary group and tried that again with the support engineer yesterday, but in that case the user-targeted app deployments don't show up in the Software Center. For that to work, the engineer said that when a device is on intranet, it needs to receive the policy from an on-premise MP. And in our case the MP also hosts the SUP/DP role, and then clients don't pull the content from Microsoft Update but use the on-premise content, unless we split up our patch deployment collections and use different download settings for the VPN clients (which is going to be complex to manage). 

 

We'll have another look at it today with the fallback chain but we had already tried that last week. 

Copper Contributor

Thanks for the reply @Rob York and @Andy D'Hollander.   We have configured both our SUP and a stand alone MP into the VPN boundary group with the CMG and our application deploys and software updates are now working.  In this configuration, the management traffic traverses the VPN connection, but we are seeing the the content downloads falling back to the CMG or Microsoft update so the largest portion of our traffic is offloaded from the VPN.

Copper Contributor

@Rob York @romanmensch we're seeing the same thing (users not being able to download content for user-targeted apps that are "required") and believe it to be an issue with how our AD is connected to Azure. We're investigating using our Premier DSE for #MEMCM but believe that it may be because user-targeted apps that are required need to be authenticated via Azure and not via on-prem AD. I don't believe all of our users are being sync'd fully into Azure such that a domain\user auth = user@domain.com ... we're still investigating tho so I will report back when we see a solution in sight.

Microsoft

@James Lewis yes, in order to leverage user policy over CMG you need to enable Azure AD User Discovery https://docs.microsoft.com/en-us/configmgr/core/servers/deploy/configure/azure-services-wizard

Copper Contributor

Glad to see we're not the only ones with the issue;  User Apps not appearing in Software Center when utilising CMG + EHTTP + VPN.

 

I'm getting no where with my PremSupport case.  @Rob York this realllly feels like a bug..  Are you able to confirm that when client is on Intranet (via VPN), with CMG as it's sole Site Server in boundary, that when it contacts the CMG upon opening Software Center, it should use Windows Authentication, as opposed to AAD Authentication (which works when on Internet) as per the below lines:

 

Intranet Software Center:

Using endpoint Url: https://FQDN-OF-CMG/CCM_Proxy_MutualAuth/XXXXXXXX:443/CMUserService_WindowsAuth, Windows authentication (Microsoft.SoftwareCenter.Client.Data.ACDataSource+<>c at <RefreshLocalSettingsAsync>b__16_0)

 

Internet Software Center:

Using endpoint Url: https://FQDN-OF-CMG/CCM_Proxy_ServerAuth/XXXXXXXX/CMUserService, AAD authentication (Microsoft.SoftwareCenter.Client.Data.ACDataSource+<>c at <RefreshLocalSettingsAsync>b__16_0)

 

The issue here is also that because it fails with Windows Authentication, it takes 2 minutes of 403 returns (confirmed by iis on CMG) until Software Center actually loads.

 

It really feels like someone has just forgotten that the CMG being a sole Site System on Intranet was a possible outcome, and the 'Intranet Only' switch in the sms agent instantly sets it to Windows Auth be damned.

Microsoft
It does look like client on intranet talking to CMG wont use AAD auth. We're investigating. Workaround is to make an MP available to the VPN boundary
 
Copper Contributor

@Rob York thanks for the follow up, we also have a case open and haven't been able to make any progress. Will be watching closely for updates :)

Copper Contributor

Is anyone seeing that when they add the internal management point to the VPN boundary group, some clients still prefer the CMG over the internal management point and fail authentication?

Copper Contributor

@Rob York what is the effect of overlapping boundaries?  If we have a boundary for an AD site of which the VPN IP range is a part, do we need to remove the AD site boundary and replace it with IP ranges/subnets within that site? Or can we set up a new boundary for the VPN IP range and put it in its own boundary group and configure the appropriate site systems and settings for the VPN boundary?

Copper Contributor

@Greg Neveau @Rob York , we opened a case with MS this week, saw this thread, and have since added an internal MP to the VPN boundary group.  The MS case SE told us to use an ARM CMG to resolve this issue.  Has anyone seen VPN clients not downloading from ARM CMG, or knowing the classic ASM CMG working for them?  btw, non-VPN clients can download from our ASM CMG just fine.

Microsoft

@eschloss Overlapping boundaries are supported for content but you would probably still some some(?) clients going to on prem sources. Best option is to get the AD site split out

Copper Contributor

Great article!

However, I've got stuck with the problem of VPN IP Address registering in SCCM.

All of the computers that I've checked on the VPN IP address isn't published in the WMI Class Win32_NetworkAdapterConfiguration and as a consequence the VPN address isn't registered in SCCM. Seems this is a bug https://support.microsoft.com/ru-ru/help/2549091/the-win32-networkadapterconfiguration-class-is-unab...

Has anyone dealt with the issue?

Bronze Contributor

Under the current circumstances, with many users working from home who would otherwise be on-prem only, is there any reason not to redeploy the client as always internet now, and then re-deploy with that option off in a month or two when they're back on site for good? 

Copper Contributor

Short update from me 24.04.20: @Rob York  We also make an MS call. Unfortunately, we have a solution yet. However, we also found a very hidden user settings in configmgr that allowed cloud policies. We have to say yes its was no. Now in Production it's works!

It is important that both apps (Client / Server APP) are available in AD Azure and the CMG Analyzer is completely green an the Clients are Hybrid Joined.

 

User Setting in Client Setting and deploy it on active users: 

image.png

Copper Contributor

@Rob York We have a CMG setup. However I am confused on setting up my VPN Boundary group. If want to setup our VPN clients to only go to Microsoft for updates but still want to serve up content (applications) from our on-premise DP how would we configure that in the boundary group? I've added both our CMG and SUP to the site system servers, but from what I understand the checkbox "prefer cloud based sources over on-premise sources" only applies to applications not updates?

Copper Contributor

@coreypullman your VPN boundary group (BG) does not control clients going to get updates from Microsoft updates, but your Software Update (SU) deployment should. In your deployment properties, you could check the box that says something like "If SU are not available on DP , download from MS Updates".  My understanding is that your CMG has to be an ARM CMG for this to work, and that your on-premises DP should within that VPN boundary should not have any SU content.  

Copper Contributor

@FintanSoUnderstood. However my issue is that I only have one DP in my site, so I still need to serve up the SU content to my other on-premise clients. I get that I could remove the DP from the boundary group and just have the CMG under site servers, but then I wouldn't be able to deploy any non-update content to these VPN clients.

Copper Contributor

@coreypullman Not sure if I understand.  If you have just your ARM CMG with App content in your VPN boundary group, why won't you be able to deploy app content from the ARM CMG to your VPN clients, and have the same VPN clients get SU content from MS updates? Your only on-premises DP can serve all contents to your on-premises clients and leave it out of your VPN BG.

Copper Contributor

@FintanSoSorry if I'm not being more clear. So the way I understand it, to configure how you're describing it wouldn't I have to upload the app content to a cloud DP and then pay for the egress traffic? That is how I understood it, which is why I was trying to avoid doing that since pushing the clients to Microsoft for updates would avoid any extra costs.

Copper Contributor

@coreypullman What do you use your ARM CMG for at the moment if you don't populate it with app content?  Yeah, you will need to compare if have a second on-premises DP in the VPN BG with just apps and no SU content would cost less than the egress traffic.

 

Copper Contributor

We have the exact same issue.  I have been on a Premier Case since Monday April 6.  I have multiple engineers teamed into my CMG server on the intranet and working a laptop on my personal internet.  Applications that are available/deployed to computer groups work fine getting there deployments from Azure.  User available/deployed packages do not show as available

Copper Contributor

Read this thread and are having a similar problem although not exactly as it is mentioned. Clients are detecting when not on VPN that they are internet clients and checking into the CMG and reporting back. However the software center is not available to install device targeted apps. We have removed the deprecated application catalog website and application catalog web service roles from the site system (client settings have been updated) but it seems that the client on the internet is still trying to reach the Site server by its internal DNS name which is not resolvable externally. It still lists the following "GetCategoryValuesAsync: There was no endpoint listening at http://Internalservername/CMApplicationCatalog/applicationviewservice.asmx that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.. Unable to fetch user categories, no endpoint found." as if it is trying to connect directly to the app catalog webservice role.

 

Followed by "GetCategoryValuesAsync: Object reference not set to an instance of an object.. Unable to fetch user categories, application catalog role is probably not installed. (Microsoft.SoftwareCenter.Client.ViewModels.SoftwareListViewModel+<UpdateCategoriesAsync>d__126 at MoveNext)"

 

Any ideas?

 

 

Microsoft

Repeating my response from earlier. If a client is reporting as intranet and talking to CMG it wont use AAD auth. We're investigating. Workaround is to make an MP available to the VPN boundary

Copper Contributor

@Rob Yorkit looks like we might be seeing issues with clients reporting as intranet not using AAD auth (at least, as far as I can tell). We're seeing issues with Software Updates coming down to computers when on VPN. You say we should make an MP available to the VPN boundary, but we have a single server SCCM configuration, so our MP is also our distribution point on prem. Won't making this available cause VPN connected machines to get content from that on prem server over VPN instead of the CMG? Do we need to set up another MP somewhere that is NOT also a DP?

Version history
Last update:
‎Mar 18 2020 02:33 PM
Updated by: