Exchange 2013/2016 Monitoring Mailboxes
Published Mar 20 2015 06:53 AM 288K Views

 

Note 10/2022: This article also applies to Exchange Server 2019.

Introduction

Exchange Server 2013 introduced a new feature called Managed Availability, which is a built-in monitoring system with self-recovery capabilities. Managed Availability performs continuous tests (probes) that simulate end-user actions, to detect possible problems with Exchange components or their dependencies. If probes are failing, it performs gradual simple recovery actions to bring the affected component in healthy state. It uses special type of mailboxes, called Monitoring Mailboxes or health mailboxes, to simulate end-user kinds of tests. The life cycle of monitoring mailboxes is taken care entirely by Managed Availability components. In this post, we’ll see how Managed Availability takes care of monitoring mailboxes, what best practices to keep monitoring mailboxes happy are and some related troubleshooting.

Monitoring Mailboxes

Functioning of Managed Availability is implemented by Microsoft Exchange Health Manager Service running on each Exchange Server 2013 role. The Microsoft Exchange Health Manager Service is responsible for creating and maintaining monitoring mailboxes Let’s take a look at how the Health Manager creates them!

How do we create monitoring mailboxes?

The MS Exchange Health Manager Service runs in two processes, MSExchangeHMHost.exe and MSExchangeHMWorker.exe (let’s call it the HM worker). HM worker process, at the time of startup, checks for availability of monitoring mailboxes and creates monitoring mailboxes as needed. Starting with Exchange Server 2013 Cumulative Update 1, accounts for monitoring mailboxes are created under following container in the domain Exchange server resides:

<ADdomain>\Microsoft Exchange System Objects\Monitoring Mailboxes

Example: image The logic HM worker process uses to detect and create monitoring mailboxes depends on Exchange Cumulative Update (CU) installed, Exchange role installed on the box and mailbox databases present. The Following logic was used to create monitoring mailboxes for Exchange Server 2013 servers between RTM to Cumulative Update 5:

  • One monitoring mailbox per mailbox database copy, plus one for all of CAS servers.

Here’s an example of monitoring mailboxes created on Exchange Server 2013 SP1 server that hosts both CAS and mailbox role:

[PS] C:\>Get-Mailbox -Monitoring | ft displayname,whencreated
DisplayName                                         WhenCreated
-----------                                         -----------
HealthMailboxb285a119be6649b3a89574f078e947f5      11/10/2014 9:07:29 AM
HealthMailbox60d8a8d1285e41bfa5ce1ef1fb93d14e      11/10/2014 9:07:36 AM

The display name of the monitoring mailbox created for database copy contained the GUID of the mailbox database for which it was created. Example: image The display name of the monitoring mailbox created for CAS server contained the GUID of the Exchange server for which it was created. Example: image The following logic is used to create monitoring mailbox for Exchange Server 2013 Cumulative Update 6 and onwards:

  • One monitoring mailbox for each mailbox database copy hosted on mailbox role, plus ten monitoring mailboxes for each CAS server.

The following naming convention is used for the display name of the monitoring mailbox created for a database:

“HealthMailbox-“ + host name of server + “-“ + database name.

At the startup, the HM worker checks for name of databases present on the server and then checks for presence of monitoring mailboxes with display name as explained above. If it doesn’t find a monitoring mailbox for a specific database, it creates a new monitoring mailbox. That means, if you rename DB1 to DB2, HM worker will create new monitoring mailbox for DB2 on next startup. And the following naming convention is used for display name of the monitoring mailbox created for a CAS server:

“HealthMailbox-“ + host name of the CAS server + “-001” through “010.”

We attempt to distribute the monitoring mailboxes created for CAS servers across the mailbox databases available. Let’s make this real: Imagine that you have only one Exchange server running Exchange 2013 CU6 or newer in the org (server is named EXCH1) that hosts both CAS and Mailbox role and has only one mailbox database, 11 monitoring mailboxes will be created. Example: image The Health Manager Service will create more mailboxes according to the logic explained above as you go on adding more databases or server roles.

Password resets

HM Worker is responsible for maintaining the password for Monitoring mailboxes. HM worker uses a complex algorithm to generate password to be used for monitoring mailbox. The password for monitoring mailbox is reset under the following conditions:

  • A new health mailbox is being created
  • Each time HM Worker process starts and is not able to retrieve the existing password for the monitoring mailbox
  • Any other scenario where HM Worker is not able to get hold of existing password for the monitoring mailbox

Best practices

Here are some best practices regarding management of user accounts associated with monitoring mailboxes as well as mailboxes themselves:

  • Do not apply third party customized password policies to user accounts of monitoring mailboxes
  • Exclude monitoring mailboxes from user account lockout policies
  • Do not move the user accounts out from the Monitoring Mailboxes container
  • Do not change the user account properties, like restricting password change etc.
  • Do not change AD permission inheritance
  • Since HM worker handles password resets for monitoring mailboxes, in a large environment, it is normal to see increased password reset traffic for monitoring mailbox accounts; note that doing one of the things above might increase the frequency of those resets
  • Do not move the monitoring mailboxes between mailbox databases
  • Do not apply mailbox quotas to monitoring mailboxes
  • If applying a retention policy, ensure the data within the monitoring mailbox is retained for a minimum of 30 days before being deleted

If you see a mailbox size increasing significantly for specific monitoring mailbox; you can simply disable the specific mailbox. The HM worker will create a new mailbox at next startup.

Common tasks with monitoring mailboxes

How to list monitoring mailboxes

The Get-Mailbox command is provided with special parameter “-Monitoring” to list only the monitoring mailboxes. Here are some examples:

  • To list all monitoring mailboxes present in the organization: Get-Mailbox –Monitoring
  • To list monitoring mailboxes present on a database: Get-Mailbox –Monitoring -Database <database name>

However, the mailboxes listed above may not be associated with the server on which database is hosted. As explained in creation logic, the name of Client Access Server or the mailbox database is important in searching for monitoring mailbox associated and in creation of monitoring mailbox.

  • Use the following command to list the monitoring mailboxes associated with a specific server:

Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-<servername>-*"}

Example:

Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-exch2-*"} | ft name,displayname,database

  • Use the following command to list the monitoring mailbox associated with a specific database:

Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-<name of database>"}

Example:

Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-db2exch2"} | ft name,displayname,database,servername

Troubleshooting tips

Here are some troubleshooting methods for monitoring mailboxes.

How to re-create monitoring mailboxes (NOT considered regular maintenance!)

The mailbox database removal doesn’t cleanup the AD user account associated with monitoring mailboxes. This can then result in orphaned AD user accounts. This happens because of deny permission inherited on Monitoring Mailboxes container. KB article 3046530has details on this, as well as the workaround to resolve it. If there are too many orphaned monitoring mailbox accounts, you may want to re-create them. Steps: 1) Make sure “Monitoring Mailboxes” container is present

  • Open Active Directory Users & Computers
  • Click on View and select “Advanced Features”
  • The Browse to Microsoft Exchange System Objects
  • Verify the presence of the “Monitoring Mailboxes” container.

Example: image If the Monitoring Mailboxes container is missing:

  • Make sure you have Exchange Server 2013 CU1 or above installed.
  • Perform PrepareAD with the Exchange Server 2013 version installed.

2) Stop the “Microsoft Exchange Server Health Manager” service on all Exchange Server 2013 servers. 3) Open Exchange Management Shell and use following command to disable existing health mailboxes:

Get-Mailbox -Monitoring | Disable-Mailbox

4) Go back to Active Directory users & computers, right click on domain and search for “HealthMailbox” image image 5) Delete the health mailbox user accounts. 6) Wait for AD replication or force AD replication. 7) Start the “Microsoft Exchange Server Health Manager” on all Exchange Server 2013 servers.

Bhalchandra Atre

Updates

  • 6/2/15: Updated best practices to include information on aging out data via retention policies.
17 Comments
Not applicable
Thanks!
Not applicable
Great article. What about if you have more than the expected number? For instance, I should have 11, but have 14. Looking at each one, I see that three of them still have the GUID in the Display Name, and 11 (which is the number I'm supposed to have post

CU6) follow the new naming convention.


Do I delete the 3 with the GUID in the display name? I should note that I'm not having any issues.

Not applicable
[tag:billroland]

You can leave alone the other mailboxes, won't cause harm.

Not applicable
Thanks for the insight. Straight forward and resourceful article.
Not applicable
Great Article... bonus points for posting on my BDay! [tag:Iam]MEC
Not applicable
Great guidance. I was in the process of writing a blog about some perceived issues I was having with the monitoring mailboxes when this article posted. Ended up incorporating into my post.


https://oddytee.wordpress.com/2015/03/21/this-page-cant-be-displayed-when-accessing-protocolhealthch...


Thanks.



Todd

Not applicable
Great article,thanks.
Not applicable
Great article, It finally explains why there are so many Monitoring Mailboxes after CU6.
Not applicable
A Great Exchange Server 2013 (On-Premises) Article :) Thanks.
Not applicable
Can someone explain why the MSExchangeHM workers use a CONSTRUCTED UserPrincipalName when attempting to logon to Monitoring Mailboxes. Typically, the constructed UPN is in form Alias@DefaultAcceptedDomain.


Since not everyone uses AD Forest Name = Default SMTP Domain (an there's probably a minority of people doing this indeed), then this causes the Managed Availability components to fail login on when the default Accepted Domain is NOT the same as the Forest's

Root Domain's Fqdn.


E.g. in an Exchange 2007 pre-existing environment:


Default Accepted Domain = company.com

Forest Root DNS = company.local


In this case, Monitoring Mailboxes' UPN (as created in AD) is HealthMailboxXXXXXXXXXX@company.local - we observed Login Failures in Security Event log on all Exchange servers (Unknown User Name or Bad Password). After a few ticks, all "proxy-kind" Server Components

States show Inactive (Requester: HealthApi).


We therefore created a new Accepted Domain, named "company.local" and configured it as Default. Restarted the MSExchangeHM service on ALL Exchange 2013 servers and this fixed it.


After rolling out CU8 in our qual lab, we found that the HealthApi bought back all proxy-kind Server Component States to Inactive. Back to Security event logs, issue reappeared.


Reason:

company.com Accepted Domain was named "Default Domain". For a reason, Exchange setup forced it to default again (indeed, as the name says but why the hell is it touching an existing object ????).


Final resolution was to:


1. Rename "Default Domain" to "company.com"

2. Rename "company.local" to "Default Domain" and make it Default again. Expecting that next setup will not break things again (at least, "Default Domain" is now Default, so shouldn't).


In the end, the question is:


Why on hell, developers did prefer constructing a UserPrincipalName rather than doing something really stupid (???) as using the UserPrincipalName property ?


Dear Microsoft, not everyone is using contoso.com everywhere... :)

Not applicable
This is one stop shop for Monitoring Mailboxes. Thanks for putting these together Bhalchandra!
Not applicable
Doesn't at all address how much space these mailboxes use, or how to clean them up. We regularly blow these mailboxes away because we found that they were using more than 2TB of storage in our environment. It's all well and good that you guys just blow

away your servers in 365 every time you patch them, but it sure as hell makes you guys sloppy. Won't be long before you have to reinstall a Windows server every month or two.

Not applicable
A helpful article, I now understand why there are so many Monitoring Mailboxes, however, it would be nice if someone from Microsoft would publish an article to explain the procedure for cleaning up these mailboxes.

I deleted and let Exchange re-create all my Health Mailboxes 5 weeks ago, and already some hold 50k items plus, totaling 300MB.

You state that you can simply disable a large growing Mailbox and Exchange will create another, but this does not solve the issue, and it is not recommended to delete them as regular maintenance.

Has anyone any advice?

Not applicable
I get a lot of errors in the Application Log, all about HealthMailboxes "


The user "domain.net/Microsoft Exchange System Objects/Monitoring Mailboxes/HealthMailbox5738c5948af14a7587a89d81ffc7a33d" isn't assigned to any management roles."


I tried to delete them all and let Exchange recreate them, but same error.

Not applicable
If we are not supposed to apply retention policies on the mailboxes how do they get cleared out?


Thanks

Clint

Copper Contributor

What a great article itis really useful for me

Brass Contributor

Just discovered that setting AutoDagExcludeFromMonitoring to $true also prevents the Health Mailbox creation in the database.

Co-Authors
Version history
Last update:
‎Oct 07 2022 12:16 PM
Updated by: