Office 365 ProPlus Deployment and Proxy Server Guidance

[Dave Guenthner]

By far, the most important prerequisite for successful Office 365 ProPlus deployment is network configuration. 

Unlike older versions of Office, Office 365 ProPlus was designed from the ground up to work with cloud services such as Microsoft Content Delivery Network (CDN).  Microsoft recommends IT Pros “Bypass or white list endpoints on network devices and services that perform traffic interception, SSL decryption, deep packet inspection and content filtering” when accessing Microsoft Office 365 service endpoints.  

We often find customers apply “legacy” network configurations for on-premises only products to Office 365 ProPlus which can lead to slower product adoption, poor product performance, and higher cost of ownership.  The network requirements are documented in Office 365 URLs and IP address ranges document.

The goal of this blog is to clarify how IT Pros can optimize Office 365 ProPlus deployments with a proxy server in order to leverage a new concept called Hybrid or “Lean Installs”.

In terms of Office 365 ProPlus general deployment, we have several broad approaches. We’re going to focus on the 3^rd option, "SCCM with Office CDN fallback" or "Lean Install".

• On-premises only – download and mirror all content from CDN on-premises. Enterprise customers have a variety of install packages (Base Office, Visio, Project, Visio + Project, second installs for languages).  All Office 365 ProPlus builds are cumulative and are updated monthly which can make this cumbersome and difficult to maintain as each permutation requires refreshed content.
• Cloud only – installations from the Office portal and update workflow occur using CDN. End users in the enterprise are normally not Administrators so self-service installations from portal.office.com are blocked.  Further, installation from CDN doesn’t currently support custom configuration files to exclude applications and so forth.
• SCCM with Office CDN fallback or “Lean Install” – IT Pros use SCCM (which has elevated permissions and allows custom configuration.xml files) to deploy Office 365 ProPlus but can either omit all or portions of the installation source and use CDN content.

SCCM is not a requirement to adopt “Lean Install” approach. If you are using 3rd party deployment tool, identify user context of process using process monitor and adopt proxy strategy below.

Lean install examples:

1^st Install

SCCM package contains all Office 365 ProPlus content and only subset of languages. You support 12 languages but only include two primary languages in the application source to minimize content and include AllowCdnFallback as Enabled within configuration.xml.  During the Office 365 ProPlus installation process, the Office Deployment Tool (ODT) looks first for source files in local working directory. If the language pack files required aren’t available in local source location and the AllowCdnFallback setting within configuration.xml is set to True, then the ODT will leverage the Office CDN for the missing ones.

2^nd Install

Office may need to be reconfigured to make changes to Office deployments without changing the version, like adding a language or Project/V.... In this case, we only want the required bits to perform the change and nothing else.

All example scenarios above depend on the CDN to fetch content when embracing these new “Lean Install” approaches.  The primary reason we want to lean on the CDN is because it allows Office 365 ProPlus to only download the bits it requires for the change request resulting in the smallest network payload possible. 

Exploring 2^nd Install Scenario in detail in terms of content size:

IT Pro wants to perform a 2^nd install to add Project to an existing Office 365 ProPlus installation on one machine. 

If we use SCCM on-premises only strategy:

SCCM will download full Office content from CDN ~2GB.  SCCM will then copy this content to all distribution points to support scenario.  Let’s assume an enterprise customer has 50 distribution points, 2 GB * 50 = 100 GB per month every month (build needs to be up to date as to not downgrade client introducing security concerns).  Office 365 ProPlus builds are cumulative, irrespective of channel, so this content changes each month.

If we use SCCM with Office CDN fallback:

SCCM calls ODT Setup.exe /configure to add Project, only ~41 MB will be downloaded from CDN instead of ~2GB (without language packs). 

Make sure to use MatchInstalled parameter in your XML. Ensure Delivery Optimization is working for further efficiency.

We expect most customers will download and cache all Office 365 ProPlus content one time to existing machines to perform an upgrade to Office 365 ProPlus but once installed we recommend to leverage the lean technique going forward.

Having “lean” applications in SCCM also means they rarely need to be updated. An occasional refresh of the Office Deployment Tool (ODT) is good idea. (Less than 10 MB)

Tip: There are several ODT features which can benefit from approach (FallbacktoCDN, MatchPreviousMSI, MatchInstalled, MatchOS)

Proxy Challenge

It seems there are an infinite number of ways customers secure egress traffic to internet for security and compliance.  Some don't configure a proxy within Windows and some do.  The Office team is always looking for opportunities to reduce network friction where possible.  One example is product change in terms of transport used from BITS to Delivery Optimization (DO) with fallback when downloading files during product installation and update workflow.  Delivery Optimization provides user impersonation which will allow connectivity to internet when an interactive user is logged on with WinINET internet access even if System account has no internet access.  By default, Delivery Optimization as a technology was designed to download portions of the content from peers in their network.  Customers who have investments in SCCM can enable Microsoft Connected Cache in Configuration Manager which can be a powerful combination of these technologies.  In this way, Office can leverage Windows network stack feature Delivery Optimization and combine that with Microsoft Endpoint Configuration Manager for increased network efficiency.  Customers can use Delivery Optimization in Update Compliance dashboard to see metrics about Office peer sharing efficiency.  SCCM IT Pros can reduce operational costs by eliminating need to manually download and distribute static content to Distribution Points and allow Office to dynamically populate cache based on their needs regardless of channel or version and at the same time remain in control of what versions of Office are deployed to their clients.  The transport change to Delivery Optimization for Office downloads is currently rolling out now and should be completed by July 2020 in Semi-Annual Channel.  IT Pros don't need to do anything intentional to take advantage of these enhancements.

Many customers we visit in the field prohibit computers from accessing the internet directly.  Typically, only Users can access the internet through a proxy server or via PAC file.  These User settings are defined as WinINET proxy setting you’ll find in Internet Explorer. 

Standard proxy configuration in Internet Explorer

So, what about the local SYSTEM account needed by SCCM?  If customers allow users and computers access to Office 365 endpoints, everything “just works”.  However, often we find customers only configure network proxy for Users and therefore the “lean install” or update scenarios driven by Office Automatic Updates 2.0 scheduled task may fail when user is not interactively logged on as user impersonation is not possible. (Example: Installation will hang as Office Deployment Tool running as System process will fail when attempting to access Office CDN)

OK, what can we do to solve problem?  Configure network perimeter device to white list Office CDN\URLs or configure additional proxy settings within Windows.

Alternatives:

1. Configure WinHTTP proxy (if no interactive user) as DO runs as Network Service
2. If DO fails, attempt to fallback to BITS will respect WinINET for System account

*Installs and Updates can still succeed without System proxy via User Impersonation (leverages DO transport to provide feature) through interactive logged on user session who does have internet access but this can limit ability to perform deployments after hours when no user session is present or ensuring update compliance if computers are not logged on regularly.

In my lab I use PSEXEC.EXE to accelerate testing.

From elevated command prompt, run PSEXEC.EXE -s -i cmd.exe.  This will launch cmd.exe process in the SYSTEM context to simulate SCCM package etc.  Type whoami from command line to verify.

C:\Windows\System32>whoami
nt authority\system

Sample commands to set WinINET and import into WinHTTP:

C:\Windows\System32>bitsadmin /util /setieproxy localsystem MANUAL_PROXY proxy.contoso.com:8080 ";*.contoso.com"
C:\Windows\System32>netsh.exe winhttp import proxy source=ie

Sample commands to reset:

C:\windows\system32>bitsadmin /util /setieproxy localsystem RESET
C:\windows\system32>netsh winhttp reset proxy

The proxy server\network team should only allow computer access to internet URLs as defined by Office 365 URLs and IP address ranges document as well as any other URLs that they want to explicitly allow the Computer account to access.  A number of network appliance vendors provide seamless Office 365 integration as a feature by calling Microsoft Graph API to dynamically populate network access list.

In summary, using a “lean” Office 365 ProPlus deployment strategy will greatly reduce complexity and cost of ownership to operate Office 365 ProPlus. 

Additional Reference Documentation and tools for proxy configuration for Windows

Office 365 Network Onboarding tool

Use Group Policy to apply WinHTTP proxy settings to Windows clients

bitsadmin util and setieproxy

[Lucas Girardin]

Thank for this clear blog, it's a frequent issue on the field.

[DM51673]

Thanks for this article, but I need to challenge you on it I'm afraid.  I'm struggling to understand why this advice for the Microsoft Office Click-to-Run Service deviates from previous best practice advice gathered by @Eric_Lawrence for "well-written" WinHTTP apps running as SYSTEM. Use user impersonation. Most other Microsoft PGs and 3rd parties, e.g, those based on the Omaha opensource project, like the Google Update Service, or your own Microsoft Edge Update Service accomplishes this just fine. 

I'd really love some rationale please. Is this a general change in tack across Microsoft or just an Office Click to Run directional/workaround steer only? 

Scalability in larger global enterprises of such a solution aside, from a security perspective you can't possibly expect an enterprise to directly proxy the WinHTTP service on all our clients in order to permit the Microsoft Office Click-to-Run Service to be able to reach the Office CDN?  I'm sorry, in my view, user impersonation is broke with the Microsoft Office Click-to-Run Service.  Proxied CDN downloads still fall flat for us, and it needs to be fixed please. Lean installs of ProPlus (which we'd love to adopt) and fallback to CDN for those missing bits we don't have on-prem cannot be for us I'm afraid until this is addressed.

Many thanks, I look forward to your response and perhaps some input from @Eric_Lawrence and others!

[Dave Guenthner]

@DM51673 Thank you for your post.  Starting with Office 365 ProPlus Semi-Annual Channel from July 2019 Version 1902 (Build 11328.20368), several enhancements were added to help address this type of scenario. First, Office has been updated to use API WINHTTP_ACCESS_TYPE_AUTOMATIC_PROXY. This allows us to use system proxy (winhttp) and per-user proxy settings (IE). When using per-user IE proxy, it uses the IE proxy of current user. If the process is running under System (SCCM scenario), it can use the IE proxy for System. Therefore, our first recommendation is to "Configure WinINET Proxy for SYSTEM". In terms of impersonation, if the process is running under System but System account is blocked to access network and doesn't have any proxy settings, our Office Click-to-Run processes attempt impersonation under current logged in user. This also means if no user is logged on, impersonation is not possible which can be common scenario when SCCM deployments occur during off peak hours such as maintenance windows. If you have System proxy configured for WinINET or WinHTTP and are not gaining access to network as expected during Office installs etc., you may want to reach out to Microsoft Customer Service and Support to investigate with you further to scope your specific environmental needs. I hope this explanation helps, thank you for your comments.

[DM51673]

@Dave Guenthner  thank you for your response. Today we rely 100% on impersonation of the logged on user for connectivity to the Office CDN. We appreciate the limitation in that we can't update Office without a user being logged in. For security reasons we will not proxy the SYSTEM account

Unfortunately even with impersonation, CDN downloads fail at the vast majority of our sites, and we don't know why.

We use Zscaler and one click for O365 is enabled, so it should not be that.

[Dave Guenthner]

@Duncan I re-validated this morning in my lab using Office Automatic Updates 2.0 scheduled task to update behind proxy (Squid).  The only proxy configuration exists in WinINET for the logged on end User, therefore System processes have no access to internet at all.  I verified that Office processes running as System log "IdentityContext::TryGetImpersonationToken - Successfully retrieving token from User process" in Office logs and then is able update from Version 1902 (Build 11328.20492) -> Version 1908 (Build 11929.20562).  I can confirm in generic lab scenario\repro, this does work.

[DM51673]

Thanks @Dave Guenthner. We will need to bottom out what's wrong on our end then.  We have no issues with normal web browsing, or Chrome or Edge (Chromium) autoupdates (which both rely on impersonation). And as mentioned we use Zscaler One-Click, so the constant additions and changes to the Office 365 URLs and IP address ranges are all catered for.  Are there any other special networking requirements we need to be aware of perhaps? 

[Dave Guenthner]

@DM51673 I performed additional testing and did find a pattern leading to unexpected results [ERR].  I will investigate\escalate on our side, will take some time and update thread in coming weeks.  Stay tuned.

[null null]

@Dave Guenthner do you have an update from the product team as per your last thread ? 

[Dave Guenthner]

Yes, issue\pattern reference above is resolved starting in Monthly Feb 25, SAC-T tomorrow and SAC in July.  For those customers who update from CDN, while blocking all System access to Office CDN and rely on User Impersonation when interactively logged on require Delivery Optimization transport to function. (must meet base DO requirements) This is the transport which provides this feature.