Introduction to Public Folder Hierarchy Sync

Introduction

The purpose of this article is to provide high level overview of hierarchy synchronization process in public folders and provide some troubleshooting steps to monitor and troubleshoot hierarchy related issues. The information presented here applies to public folder deployment in Exchange on-premises as well as Exchange Online, with any differences called out. Before we get started, these are some of the terms used in public folder world:

Term	Definition
Modern public folders	Public folder architecture and deployment in Exchange Online and Exchange on-premises versions at or newer than Exchange Server 2013. In this architecture, public folders are stored in specialized mailboxes, called public folder mailboxes.
Public folder hierarchy	The logical structure/skeleton of public folders and associated properties as well as permissions.
Public folder mailbox	A special type of mailbox that is used to store public folder content and public folder hierarchy for modern public folders.
Public folder content	The actual data stored within public folders.
Primary hierarchy public folder mailbox	The public folder mailbox that hosts writable copy of the public folder hierarchy. The first public folder mailbox created in an Exchange Organization is the primary hierarchy mailbox. There is currently no supported way to transfer this functionality to another mailbox.
Secondary hierarchy public folder mailbox	All other public folder mailboxes in an Exchange organization, except the primary hierarchy, which store read-only copy of the public folder hierarchy.
Hierarchy synchronization	The process of copying the public folder hierarchy between public folder mailboxes. The hierarchy replication is always from primary hierarchy public folder mailbox (which contains the only writeable copy of the hierarchy) to secondary hierarchy mailboxes.
Public folder content mailbox	The public folder mailbox storing the actual content of a public folder. In modern public folders, there is only one copy of content.
Hierarchy mailbox	Any public folder mailbox that is enabled to serve the hierarchy information to clients is referred as a hierarchy mailbox.
Full synchronization	The process of synchronizing entire hierarchy to secondary public folder mailboxes. Note that this is a hierarchy only synchronization.
Incremental synchronization	The process in which only changes made in hierarchy, after last sync, are synchronized to secondary hierarchy PF mailboxes.

Public folder hierarchy sync

In simple terms, the public folder hierarchy is the skeleton or logical structure in which the permissions and other properties of public folders are arranged. The hierarchy makes it possible for clients to determine which PF mailbox stores the actual content of the public folder. The hierarchy is synchronized across all public folder mailboxes, to ensure a single PF mailbox is not burdened with serving the hierarchy to all clients. Frequent synchronization of the hierarchy is critical to ensure end users always get an up to date view of the hierarchy. Here’s what can happen when the hierarchy is not in sync. Public folder mailbox EPF1 is the primary hierarchy mailbox and has two folders in the hierarchy. A new PF mailbox EPF2 is created. EPF2 won’t show the complete public folder structure until the hierarchy is synced: Let’s discuss how the public folder synchronization works and the components involved.

Pull mode

In pull mode, hierarchy replication is driven by secondary hierarchy mailboxes. At regular interval, the secondary hierarchy mailbox connects to the primary hierarchy mailbox to pull the changes in hierarchy. This process is accomplished with the help of the ServiceHost process (running on each server) and the MRS proxy service (running on the server hosting the primary mailbox). The frequency of connection depends on whether clients were connected to the mailbox or not. This workflow is executed with the following frequencies:

Mailbox Type	Frequency	Notes
Primary	Never	Primary is the master of the hierarchy; it does not pull from anywhere else
Secondary with users connected	15 minutes	At least once every 15 minutes as long as there are client connections to the mailbox
Secondary with no user connections	24 hours	All secondary mailboxes are synced once daily.

Additionally, the following situations also trigger sync operations between a secondary mailbox and primary:

1. A user connected to a secondary mailbox performs a hierarchy update / change: The operation is performed in the primary hierarchy mailbox and then immediately replicated to the PF mailbox that is serving hierarchy for the user who made the change.
2. An update is done in a folder whose content is hosted in a mailbox other than the primary mailbox: The operation is first applied in primary (all write hierarchy client operations get proxied there) and then replicated immediately to the mailbox hosting the affected folder.
3. An admin user invokes the Update-PublicFolderMailbox with the InvokeSynchronizer parameter.
4. A new mailbox is created by Migration or AutoSplit (in Exchange Online only): These processes will cause a full hierarchy sync to be executed in the new mailbox.

The process described above was the default model for full as well as incremental hierarchy synchronization in the early days of modern public folders and served well as long as the number of public folder mailboxes was not high. However, there were some limitations with this model:

• Secondary mailboxes had no way to know if there had been a change in hierarchy. In other words, secondary mailboxes would keep connecting to the primary at defined intervals, even if there was no change in hierarchy.
• More secondary mailboxes meant more sync jobs, thus more connections competing for the limited resources in the primary mailbox.
• Larger hierarchies mean more data during sync (larger sync states and more folders to transfer) which meant longer execution times and higher resource consumption (memory, IO, CPU and network).
• As the size of the user base increases it means more hierarchy mailboxes (and consequently a higher number and frequency of sync jobs) and this usually leads to more hierarchy churn (thus requiring longer sync jobs as more data is processed).

To address these problems, a new “push mode” of hierarchy sync was introduced.

Push mode

The push mode of hierarchy sync is driven by primary PF mailbox, mainly for the incremental synchronization of hierarchy. This change was introduced in Exchange Server 2016 CU2 on-premises and in Exchange Online around the same time. In push mode, the primary PF mailbox proactively:

• Notifies secondary mailboxes when there have been changes.
• Sends any data the secondary mailbox may need to apply the recent changes, limiting the need for it to contact primary.

Pull mode is still used for:

• Initial sync of newly created PF mailboxes
• Full sync requests triggered by an administrator
• Fallback mechanism in case of push notification being lost, or when mailboxes become out of sync

In push mode, the primary PF mailbox drives the incremental hierarchy sync jobs, using transport to send email messages to a hidden distribution list containing the different public folder mailboxes present in the hierarchy. Some key properties of this distribution list:

• It is a hidden Dynamic DL.
• It has ShowInAddressList set to false.
• Its identity will be persisted in the organization object so mailboxes can refer to it.
• You should normally find only one such dynamic distribution group created in an Exchange environment.

The hidden DL created for sending push notification messages cannot be seen in Exchange Online but you can use the following command to view the hidden DL in Exchange on-premises:

Get-DynamicDistributionGroup -IncludeSystemObjects pub*

Secondary mailboxes will use both modes of sync, pull mode for initial full sync and push mode for incremental hierarchy sync, depending on the sync state they are in. The first sync is performed using pull mode, after that, they will not contact primary mailbox repeatedly for incremental syncs. Instead, they wait for the hierarchy sync notification email and apply the hierarchy changes once the email is received. If the secondary hierarchy mailbox doesn’t receive a push notification message within 10 minutes, it will fall back to pull mode and perform the incremental sync by contacting the Primary mailbox. Secondary mailboxes will also use the pull mode if an administrator triggers hierarchy sync.

Hierarchy mailbox assignment

Exchange by default automatically assigns a default public folder mailbox on all user mailboxes, load balancing between all available and assignable public folder mailboxes. A public folder mailbox having following properties is considered available for automatic assignment. In the following example, both public folder mailboxes are considered available for automatic assignment: An admin can exclude specific PF mailbox from being automatically assigned by configuring IsExcludedFromServingHierarchy to $True Example:

Set-Mailbox -PublicFolder epf1 -IsExcludedFromServingHierarchy $True

An admin can override the system assignment by using this command:

Set-Mailbox <username> -DefaultPublicFolderMailbox <PFMailboxName>

Example:

Set-Mailbox cloud1 -DefaultPublicFolderMailbox epf1

The public folder mailbox serving hierarchy for the user can be found using this command:

Get-Mailbox |ft name,DefaultPublicFolderMailbox,EffectivePublicFolderMailbox

The admin assigned mailboxes appear under “DefaultPublicFolderMailbox”; whereas the system assigned PF mailbox will be displayed under “EffectivePublicFolderMailbox” and DefaultPublicFolderMailbox will be blank. For larger deployments, avoid manually assigning DefaultPublicFolderMailbox on users, as it may overload the assigned PF mailbox with lots of concurrent connections. As we learned earlier, the system assigns DefaultPublicFolderMailbox on users in such a way that connections are load balanced between public folder mailboxes available for serving hierarchy. Follow the best practices guidelines here for public folder mailbox placement and assignment.

Making sure hierarchy is healthy

Here are some tips and tricks to help keep your public folder hierarchy in top shape.

Full hierarchy sync

Admins can use the following command to ensure that the public folder mailbox has received the initial full sync of hierarchy and confirm that the PF mailbox is ready to serve hierarchy. The value of True returned here indicates the mailbox has received its first full sync of hierarchy

Get-Mailbox -PublicFolder |ft Name,IsHierarchyReady

Diagnosing hierarchy sync issues

If you suspect a specific PF mailbox is not receiving hierarchy (perhaps users tell you they can’t see a folder they know to exist), you can use the following command to get diagnostic information about hierarchy:

Get-PublicFolderMailboxDiagnostics <pfmailboxname_notreceiving_hierarchy> -IncludeHierarchyInfo

Here are some other ways to view data; To compare the hierarchy between PF mailboxes

$P=Get-PublicFolderMailboxDiagnostics <Primary_pfmailboxname> -IncludeHierarchyInfo
$S=Get-PublicFolderMailboxDiagnostics <pfmailboxname_notreceiving_hierarchy> -IncludeHierarchyInfo

Then, you can check the output of “HierarchyInfo” from both the mailboxes and compare:

$p.HierarchyInfo
$s.HierarchyInfo

If you find the hierarchy information is not the same, you can use following command to view the time of the last sync:

$S.AssistantInfo.LastAttemptedSyncTime.LocalTime

This command tells you the last time sync failed; a gibberish value (that’s a technical term now!) indicates sync has never failed:

$s.AssistantInfo.LastFailedSyncTime.LocalTime

The following command will give you a detailed failure message from the last sync failure, a blank output indicates sync has never failed:

$s.AssistantInfo.LastSyncFailure

You can explore the other values of AssistantInfo, SyncInfo & HierarchyInfo blocks. In case the need arises to contact Microsoft Support for assistance, you should export the report to XML format and send it along.

Get-PublicFolderMailboxDiagnostics <pf mailbox failing to sync> -IncludeHierarchyInfo |Export-Clixml epf2.xml

Other useful information

The following are some things to know related to the hierarchy sync of public folders:

Push notification emails and journaling

Since push notifications are an email message delivered using transport they will be included in journaling if enabled in an organization. In such a scenario, you can use scoped journaling and exclude the SMTP address of primary public folder mailbox from the journaling rule (as that is a sender of push notification messages).

Public folder mailbox is a top sender

You may observe that the public folder mailbox shows in the “Top Sender and recipient report” in the Security & Compliance Center. This is currently normal and not a sign of a problem, because the primary public folder mailbox sends a large number of push notification emails to secondary mailboxes via transport. The next version of the report will exclude the public folder hierarchy sync emails.

Multiple dynamic DL’s created

As discussed earlier, push notification emails are sent to an Exchange-managed dynamic distribution group. Normally, only one dynamic distribution group is enough for the push notification emails. However, in environments that have multiple domain controllers within a single AD site with AD replication issues between the DCs, multiple dynamic distribution groups may get created unexpectedly. The issue happens because of a race condition and time lag between AD replication between DC’s in same site. Please follow steps in this KB to resolve this.

Conclusion

We hope this gives you better understanding of modern public folder hierarchy and some tools you can use to perform the troubleshooting. We would love to hear any feedback you may have on the topic. The Public Folder Team