Exchange Management Shell and Mailbox Anchoring
Published Dec 15 2015 07:00 AM 49.1K Views

Update: behavior described in this blog post has changed in later Cumulative Update; please see Remote PowerShell Proxying Behavior in Exchange 2013 CU12 and Exchange 2016.

Coming to the next CUs for Exchange 2013 and Exchange 2016 there are some changes to how the Exchange Management Shell (EMS) connects to Exchange. In previous versions, we have seen that customers who were reliant on a load balanced solutions for third party apps and scripts may get routed to a non-Exchange 2016 server. This would lead customers to some broken administrative experiences based on the reliance of Exchange 2016 cmdlets and features. Today we will dive deeper into these changes to help the Exchange Administrator understand how these changes will affect their Exchange Organization. In Exchange 2010 we used session affinity to provide a long-standing association between a particular client and a particular Client Access Server. With Exchange 2013 and continuing on into Exchange 2016 we have removed the requirement for session affinity. Also with the release of Exchange 2016 we added an ability to proxy between Exchange 2013 and Exchange 2016. This allowed us to have an Exchange 2013 server public facing and proxy traffic back to an Exchange 2016 server. So in order to ensure that when a connection to the Exchange Management Shell (EMS) always goes to an Exchange 2016 server we have introduced mailbox anchoring into Exchange 2013 CU11 and Exchange 2016 CU1. In mailbox anchoring, CAS locates where the mailbox resides by querying Active Directory for the user's mailbox GUID. As soon as we obtain the GUID, HTTP Proxy refers to Active Manager to locate the database containing the user's mailbox. Once CAS knows where the user's mailbox is located, the protocol request is then proxied to the appropriate mailbox server. If that server goes down and the database is a part of a DAG, the Client Access Proxy will proxy the session to the new active mailbox server for the corresponding database. This process is utilized currently for most client protocols, such as OWA, ECP, and EWS with the exception of Remote PowerShell. This is great for the client but what about the Exchange Management Shell (EMS)? In Exchange 2013 RTM through CU10 and Exchange 2016 (RTM), we do not use the mailbox anchoring logic to proxy the PowerShell session we just connect to the local server or proxy to another available server in the Exchange Organization. This means if I log into an Exchange 2013 Server I cannot manage any new cmdlets that became available in Exchange 2016. For that I would have to logon directly to an Exchange 2016 Server. Starting with Exchange Server 2013 CU11 and Exchange Server 2016 CU1, the Exchange Management Shell (EMS) session will also be using mailbox anchoring. If the aforementioned environment has all servers upgraded to Exchange 2013 CU11 and Exchange 2016 CU1 the behavior of Exchange Management Shell changes. Now when a user logs on to the Exchange Server and open up EMS, the session will be proxied to the server where the user’s mailbox is located. If the user does not have a mailbox, we will utilize the arbitration mailboxes for the mailbox anchoring logic. Wherever the arbitration mailbox is mounted is where the EMS session will be proxied. It is important to understand when you upgrade your existing Exchange Server 2010/Exchange Server 2013 organization to Exchange Server 2013/Exchange 2016, you must move the arbitration mailboxes to a database on highest version of Exchange Mailbox server. Same applies to administrators that have mailbox associated. You should move the mailbox after installing and verifying Exchange was working properly. If you don’t move the arbitration mailbox to higher version of Exchange, you may run into issues like not getting version appropriate tasks or tasks not being saved to the administrator audit log when you run the Search-AdminAuditLog cmdlet. We primarily use the Arbitration mailbox “SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}” however if that mailbox is unavailable we can use the other arbitration mailboxes to make our connection. So let’s cover how to handle a couple of scenarios that you may run into.

Installing Exchange 2016 into an Exchange 2013 Organization

After you complete the installation of your first Exchange 2016 server and verify that everything is working properly you need to move all arbitration mailboxes to Exchange 2016. To do so you can use the Exchange admin center or Exchange Management Shell and move all of the Microsoft Exchange Migration, Federation, and System Mailboxes. To move all the arbitration mailboxes via Exchange Management Shell we would run:

[PS] C:\PowerShell> Get-Mailbox -Arbitration | New-MoveRequest -TargetDatabase <Exchange_2016_DataBase>

To move all the arbitration mailboxes via Exchange Admin Center we would run:
1. In the EAC, go to Recipients > Migration. 2. Click New Add Icon, and then click Move to a different database. 3. On the New local mailbox move page, click Select the users that you want to move, and then click Add Icon. 4. On the Select Mailbox page, search for Microsoft Exchange add all the mailboxes that are in the list below:
image
5. Click OK, and then click Next. 6. On the Move configuration page, type the name of the migration batch, and then click Browse next to the Target database box. 7. On the Select Mailbox Database page, add the mailbox database indicating where you want to move the system mailbox. Verify that the version of the mailbox database that you select is Version 15.1, which indicates that the database is located on an Exchange 2016 server. 8. Click OK, and then click Next. 9. On the Start the batch page, select the options to automatically start and complete the migration request, and then click New.

How do you know which server you are connected to?

The Exchange Management Shell console will always display the name of CAS server that received the initial connection. However, the connection may be proxied to a different mailbox server in the background. You can use following trick to identify the mailbox server to which the connection is being proxied to. Open up Exchange Management Shell, and run a simple query to determine which Exchange Server you are connected to: image As you can see from the above EMS Session even though I was on the host E1501 when I ran the cmdlet Get-ExchangeCertificate it returned the certificate for server E1503. If you want to dig deeper you can go into the logging directory for the HTTPProxy PowerShell and see what server, you proxied to. (Before you can search in this path you will need to open the path in File Explorer to take ownership) image As you see above we proxied the connection to e1503.contoso.com. Since your PowerShell connection is actually running on the server e1503, therefore all cmdlets run will be against that server. If the cmdlet you are using supports the -server switch, this can be utilized to ensure which server the cmdlet is run against. For example, if I run Get-PowerShellVirtualDirectory I can specify which server I want to communicate with. image

Fail Overs

If you have an admin mailbox and the database that hosts the mailbox or DAG that the mailbox is in is down, we will then proxy to the database that has the arbitration mailbox (which is why we recommend moving this mailbox as noted before). This way cmdlets can always be run against the highest version Exchange server in the organization.

AD Site Proxying

In a multisite or geographically dispersed datacenters we will proxy the EMS session back to where the admin mailbox is mounted. In most scenarios this is fine, however there may be some tasks that would require us to have an administrator in the local site. In order to accomplish this, you will need to have either the arbitration mailboxes in the site and logon to the server with an administrator account that has no mailbox or you can create an admin account with a mailbox in the corresponding site.

Critical Failures

If you open EMS and are not able to connect to any Exchange Servers due to loss of communication, then you will need to add the PSSnapin for Exchange to your local PowerShell Session. Rob Whaley Beta Engineer for Exchange and Office 365
12 Comments
Not applicable
"If you open EMS and are not able to connect to any Exchange Servers due to loss of communication, then you will need to add the PSSnapin for Exchange to your local PowerShell Session."


Can you provide documentation in the article for how this is performed? Some environments will encounter this during failure or disaster recovery scenarios and may not know how this action is performed.

Not applicable
Instructions to load PSSnapin for Exchange in the local PowerShell session are already in the Exchange 2013 documentation:

https://technet.microsoft.com/en-us/library/jj150489%28v=exchg.150%29.aspx The instructions are right below this caution: "Loading the Microsoft.Exchange.Management.PowerShell.SnapIn Windows PowerShell snap-in and running cmdlets other than the *-TransportAgent

cmdlets is not supported and may result in irreparable damage to your Exchange deployment." I guess the documentation needs more updating.


My biggest concern is about the effect of this change in multi-site environments and the impact on performance.


"In previous versions, we have seen that customers who were reliant on a load balanced solutions for third party apps and scripts may get routed to a non-Exchange 2016 server."


At this point I'm not sure if I understand why this change was necessary. It introduces quite a few new interesting challenges to fix an issue that could've solved with one page of documentation on how to configure your environment with load balanced PS access.

Not applicable
Why not have it work like EWS and give the user the option of specifying the anchor mailbox (X-AnchorMailbox)? It could be added as a parameter to the Connect-ExchangeServer cmdlet then all will be well with the world.
Not applicable
To add to Andrew and Jetze's comments. using PSSnapin completely bypasses RBAC framework and relies on AD permissions of user running it from PowerShell. Ugh. I've intentionally avoided discussing this publicly until now but since it's being recommended

on this blog, I must also warn of the pitfalls of using that method even though it is last resort.

Not applicable
This solves one problem (which could be engineered differently, eg explicitly start and connect to specific host was already possible), and creates 2-3 new (potential) problems with additional workarounds. And fully agree with Bhargav, your 'admin' account

may have limited AD permissions, limiting recover/rehome/etc. options.

Not applicable
+1 to all the comments above. This is a solution in search of a problem and creates more issues that it solves.
Not applicable
Thanks Rob! for detailed explanation.. was thinking what's wrong with EMS after CU11 update :)
Not applicable
Having seen this in action, it is bound to confuse customers who create their first Exchange server (in a recent client case of mine an Exchange 2013 server in an Exchange 2010 environment). When they launch the Start Menu shortcut to the Exchange Management

Shell, it shows the session connected to the 2013 server, however as we know from your above change, they are proxied to the mailbox server where the admin is, which is 2010. Now the same admin tries to find the new Exchange 2013 server in PowerShell and will

not able to do so. Unless they know about the local PowerShell (PSSnappin) or find the other shortcut in the Bin directory of the Exchange install, they will be sorely confused. Seems this will actually cause more issues than it solves. +1 to all the above

comments as well

Not applicable
+1 to DJ Grijalva's suggestion. We need to have an option to connect to the desired mailbox server without getting automatically proxyed. As already said by others, this is going to create a lot of confusion.


I like to use my account to manage all the sites, not an account each for each site.


Rather than changing the behavior, it could have been a additional switch allowing this functionalitybehavior.

Not applicable
My concern with this 'solution' has to do with Hybrid servers and Office 365. If Exchange 2013/2016 is introduced into the environment as the Hybrid server for managing Free/Busy and mailbox moves. It doesn't inherently, and isn't licensed for, mailboxes

to be homed on that server.


This requires mailboxes to be moved to the Hybrid server, which puts the customer out of licensing compliance and requires the user account to be associated to an arbitration mailbox, as their personal/admin mailbox may/will be moved to O365.



Is there a way to turn this feature off so it doesn't cause huge issues for O365 migration deployments?

Not applicable
It does create issues. I ran into problems in two separate forests upgrading 2013CU10 to 11 in coexistence with Exchange 2010Sp3RU11. One leaving me to use only the local powershell adding the pssnapin manually to access the server again. The other I needed

to create new admin accounts.

The following post by ASHigginbotham pretty much sums it up (thanks for posting)

https://exchangemaster.wordpress.com/2016/01/06/mailbox-anchoring-affecting-new-deployments-upgrades/


Haven't tried clearing the listed cookies mentioned.

Not applicable
@Satyajit321 and others- I have a quick and dirty method for solve your administration problem.

To resolve this issue

Open IIS on the CAS server

Navigate to the PowerShell virtual directory

Select "Advanced Settings" from the Actions pane on the right.

Change the Physical Path settings from"C:ExchangeServerV15FrontEndHttpProxyPowerShell" to "C:ExchangeServerV15ClientAccessPowerShell"


Run IISRESET

And of course, I don't advise you to repeat it at home, boys and girls!

I recommend to wait for correction of the problem in future releases.

Regards, Dima.

Version history
Last update:
‎Jul 01 2019 04:25 PM
Updated by: