Create bookingStaffMember also requires Add-MailboxPermission
I created an Entra application with the required Graph permissions to manage Bookings scheduling mailboxes so that I could update the staffing for a Bookings page that I'm not a staff member of. Using that application to request a Microsoft Graph access token, meant I had the required permissions to add a user account (that already had a Bookings license) as an "administrator" (role) for a Bookings page (Bookings business), to replace another person who was the previous Bookings page administrator, but who had since left. I used https://learn.microsoft.com/en-us/graph/api/bookingbusiness-post-staffmembers?view=graph-rest-1.0&tabs=powershell to do this. For the record, I was not able to do this using https://learn.microsoft.com/en-us/powershell/module/microsoft.graph.bookings/new-mgbookingbusinessstaffmember?view=graph-powershell-1.0 using Microsoft.Graph.Bookings module version 2.30.0, presumably because I am not a staff member with the Bookings "administrator" role for that particular Bookings page. However, the newly added user could not access that Bookings page at https://outlook.office.com/bookings/homepage until a further step had been completed (Error: "You don't have access to this booking page" shows instead). To ensure the new Bookings page administrator could access the Bookings page, I needed to add mailbox permissions, which I used https://learn.microsoft.com/en-us/powershell/module/exchangepowershell/add-mailboxpermission?view=exchange-ps for. I found that giving the new Bookings page administrator full access to the scheduling mailbox ensured they could access the Bookings page. Example using ExchangeOnlineManagement PowerShell module: Add-MailboxPermission -Identity email address removed for privacy reasons -User email address removed for privacy reasons -AccessRights FullAccess If there is a more efficient way for an system admin to add new staff to a Bookings page, please let me know - otherwise, hopefully this helps someone who gets stuck 😊Outlook clients receive error 0x8004010f when downloading the Offline Address Book
EDIT 10/01/2008: To read the updated (and more comprehensive) guidance for troubleshooting errors 0x8004010f, please go here. There are a multiple reasons for why an Outlook client can receive the 0x8004010f sync error. Unfortunately, 0x8004010f is just a generic MAPI error and will show up for a variety of problems. Here is what the error looks like under err.exe (Microsoft Exchange Server Error Code Look-up Tool): C:\WXP\system32>err 0x8004010f # for hex 0x8004010f / decimal -2147221233 ecNotFound ec.h ecAttachNotFound ec.h ecUnknownRecip ec.h ecPropNotExistent ec.h MAPI_E_NOT_FOUND mapicode.h # 5 matches found for "0x8004010f" This is what the error looks like from the sync log from within Outlook: 12:45:53 Synchronizing Mailbox <dgoldman> 12:45:53 Done 12:45:54 Microsoft Exchange offline address book 12:45:54 0x8004010f Some of the most common reasons for Outlook clients to receive the 0x8004010f error with regards to downloading the OAB are listed below. Most of these are documented and I have linked articles to each of these to help everybody out. Please note that these solutions can change a small bit depending on unknown factors in a company's environment. An administrator decommissioned the last Exchange server in a site and never pushes replicas to another Exchange server. A new OAB is created in the active directory and the information store never reads the active directory during its maintenance schedule. This will result in the OAB files never being generated and the Outlook client will fail to download anything. The information store has an invalid EntryID that points to the legacy EX:/ folders. Again there is nothing for the client to download. An outlook client logging in from one domain to another domain and attempting to log in to another users mailbox. The OAB was never generated or some OAB folders are missing from the public folder store. Multiple OAB Version folders exist of the same type. Clients are attempting to download the OAB files from a public folder store that have not received the replicated updates. The offline address book list object has a missing address list. The offline address book list object has an incorrect address list. Send/As changes in the store affect users accounts with no mailbox full rights to another mailbox. If you are seeing this error on an Exchange 2007 server and your OAB is generated by an Exchange 2007 server, please make sure of the following: 1. Make sure that you have added the replicas of OAB to the Exchange 2007 server 2. Make sure public folder replication is working. 3. Make sure the OAB is public folder enabled and you have OAB Version 2, OAB Version 3 and OAB Version 4 checked off so your legacy clients can download the OAB files from the public folder store. 4. Make sure that if you are using an Outlook 2007 client, your OAB is Web Distribution enabled and the OAB files have been replicated over to the Client Access Server. For more information on this process please see this blog: http://blogs.msdn.com/dgoldman/archive/2006/10/23/outlook-client-fails-to-download-the-oab-with-error-0x8004011b.aspx and http://blogs.msdn.com/dgoldman/archive/2006/08/25/How-Exchange-2007-OAB-Files-are-replicated-to-a-Client-Access-Server-for-download.aspx 5. If you are removing your last Exchange 2003 server from the org, please make sure that you follow our documentation on this process. - Dave GoldmanSearch-UnifiedAuditlogs For Mailbox - Problems
Introduction Like many, I have been faced with an audit search problem on mailboxes. I finally found a solution by searching deeply into the web. In this post I will provide you with Microsoft's documentation, I have tested everything, and it finally works. I also have comments to Microsoft, directly to the product group (with a case Microsoft) but also by the technet article feedback feature. Technical Content We assume that you have all necessary permissions and role to run audit logs search. For Regular mailboxes: if you have no results via GUI, It is possible that in the time interval there is no result. It may happen that the audit is blocked on the mailbox despite the fact that the feature is active. You may use the command http://Search-UnifiedAuditLog documenation]https://learn.microsoft.com/fr-fr/powershell/module/exchange/search-unifiedauditlog?view=exchange-ps with the following parameters: UsersIds : email address Operations : event to be search (https://learn.microsoft.com/en-us/purview/audit-log-activities#exchange-mailbox-activities) Search-UnifiedAuditLog -UserIds <MailboxIdentity> -Operations MoveToDeletedItems, SoftDelete, HardDelete -StartDate "01/01/2025" -EndDate "15/01/2025" Unfortunately, no results appear with powershell. https://learn.microsoft.com/en-us/purview/audit-troubleshooting-scenarios#search-for-mailbox-activities-performed-by-users-with-non-e5-licenses, you can find the documentation that describe the symptom and how to resolve it. Even when [mailbox auditing on by default](https://learn.microsoft.com/en-us/purview/audit-mailboxes) is turned on for your organization, you might notice that mailbox audit events for some users aren't found in audit log searches by using the Microsoft Purview portal or the compliance portal, the **Search-UnifiedAuditLog** cmdlet, or the Office 365 Management Activity API. The reason for this is that mailbox audit events is returned only for users with E5 licenses when you use one of the previous methods to search the unified audit log. You must run the following command within Exchange Online : Set-Mailbox -Identity <MailboxIdentity> -AuditEnabled $false And then : Set-Mailbox -Identity <MailboxIdentity> -AuditEnabled $true Now you can search within the GUI or with powershell and you will have some results. For Shared Mailboxes: To search audit logs for a SharedMailbox, you must use the following command, with the parameter *FreeText.* Search-UnifiedAuditLog -StartDate "08/01/2025" -EndDate "11/01/2025" -FreeText (Get-Mailbox -identity <MailboxIdentity>).ExchangeGuid -Operations MoveToDeletedItems` https://learn.microsoft.com/en-us/purview/audit-troubleshooting-scenarios#search-for-mailbox-activities-performed-in-a-specific-mailbox-including-shared-mailboxes you can find the article that describes the FreeText parameters, and also decscribes that GUI is not working for SharedMailboxes. Also, using the **User** dropdown list in the audit log search tool or the **Search-UnifiedAuditLog -UserIds** won't return results for activities performed in a shared mailbox. If there are no results and you are sure that there should be, then the same manipulation as described above will have to be done. Disable and then reactivate the audit on the mailbox: Set-Mailbox -Identity <SharedMailboxIdentity> -AuditEnabled:$false Set-Mailbox -Identity <SharedMailboxIdentity> -AuditEnabled:$true Run again the Search-UnifiedAuditLog command. Now you will find results. Conclusion I assume that the "Users" text box in the interface corresponds to the parameter "UserIds" in the cmdlet. And there is no match for the "FreeText" parameter. You can find other articles in my GitHub about Purview https://github.com/trisdev75/MicrosoftPurviewWant more control over Sent Items when using shared mailboxes?
Edit 6/15/2015: We are starting to make this feature available again in Office 365. Note that the new shared mailbox Sent Items behavior is disabled by default. If you want to enable it for your users, you can do so by using cmdlets mentioned below. Additionally, we are still on track to ship this to our on-premises customers in Exchange 2013 CU9, as mentioned before. Whether a mailbox is used by multiple users as a collaborative tool or a communication gateway to customers, retaining a record of emails sent from a shared mailbox remains an important business requirement. In Exchange 2010, there was a way to configure this behavior, but we did not have this feature starting with Exchange 2013. Our customers have told us that a shared mailbox should keep a copy of emails sent from the mailbox by all members of the mailbox in its own Sent Items folder. We have taken that feedback and decided to make some changes to how sent mails are handled for shared mailboxes. We are excited to announce that once this feature is enabled for you (see below), by default all shared mailboxes will retain a copy of emails sent from the mailbox. You will no longer have to figure out which mailbox member sent an email as the shared mailbox or on behalf of it. How does it work? Emails can be sent as the shared mailbox itself or on behalf of it by member(s) of the mailbox, assuming proper permissions have been granted. This feature is designed to retain a copy of an email sent from the shared mailbox in the Sent Items folder of the shared mailbox. The same behavior can be expected for emails sent on behalf of the shared mailbox, when configured to do so. A copy of the sent mail will also reside in the Sent Items folder of the member’s personal mailbox. Note: If the user has used the Outlook 2013 feature to change the folder that Sent Items are saved to, the messages will be copied to that folder instead of the user’s Sent Items folder. Users can reconfigure this by clicking the “Save Sent Items To” button on the Email Options tab. Administrators have control over this feature for either mails Sent As or Sent on Behalf of a shared mailbox. The table below summarizes where sent mails reside when members of a shared mailbox send mail from the shared mailbox. User Mailbox Shared Mailbox Sent Items Exchange 2010 Exchange 2010 Controlled through settings in KB2632409 Exchange 2010 Exchange 2013 (any version) Controlled through settings in KB2632409 Exchange 2013 CU9 (or newer) Office 365 Exchange 2010 The sent mail will be delivered to both the Inbox of the shared mailbox as an email attachment* and to the user mailbox' sent items Exchange 2013 CU9 (or newer) Office 365 Exchange 2013 CU9 (or newer) Office 365 The sent mail will be delivered to the Sent Items folder of shared mailbox and to the user mailbox' sent items *In a scenario where the user’s mailbox is on an Exchange 2013 CU9 server (or an Exchange 2013 without CU9 installed) and the shared mailbox is on an Exchange 2010 server, the shared mailbox will get an email message that looks like the following: This behavior will continue until the shared mailbox is moved to an Exchange 2013 CU9 server. Because this can happen even between Exchange 2013 servers (pre-CU9 and CU9), you might want to wait in turning the feature on for your on-premises deployment until you have completely rolled out Exchange 2013 CU9 to all of your servers. Who can use this feature? The feature is going to be available to all customers with shared mailboxes in Office 365 (phased rollout starting on 6/15/2015), as well as our on-premises customers (starting with Exchange 2013 CU9). Note: if after installation of Exchange 2013 CU9 in an on-premises environment you do not see the new CMDlet parameters, you should run /preparead explicitly with CU9 setup. How do I enable/disable this feature? In Office 365 and Exchange 2013 CU9, this feature is disabled by default. Enable the feature In Office 365, using the Admin portal, you can select your shared mailbox by going to Groups > Shared mailboxes first: Then Edit the Sent items copy status of the mailbox: Using PowerShell, for emails Sent As the shared mailbox: set-mailbox <mailbox name> -MessageCopyForSentAsEnabled $True For emails Sent On Behalf of the shared mailbox: set-mailbox <mailbox name> -MessageCopyForSendOnBehalfEnabled $True Disable the feature For messages Sent As the shared mailbox: set-mailbox <mailbox name> -MessageCopyForSentAsEnabled $False For emails Sent On Behalf of the shared mailbox: set-mailbox <mailbox name> -MessageCopyForSendOnBehalfEnabled $False What else do I need to know? The administrator for your organization has to create the shared mailbox and add you to it as an user, before you can use it. If you are an Office 365 Small Business administrator, see Create and use shared mailboxes. If you are an administrator for a different version of Office 365 or an on-premises Exchange administrator, see the TechNet article Create a Shared Mailbox. If you are currently using a client registry key to duplicate messages sent from a shared mailbox to a sent items folder of the sender, you should remember to remove client registry keys if you enable this feature server side. If you do not remove client registry keys, you could see duplicate messages in Sent Items folders (one created by client registry keys and one by the server). If one of the transport rules prevents messages larger than certain size and the message being sent is over that size, then the message will not be delivered to the its original recipient and will also not be copied to the sent items of the shared mailbox. Philip LohPrevent archiving of items in a default folder in Exchange 2010
In Exchange 2010, you can use Retention Policies to manage message retention. Retention Policies consist of delete tags, i.e. retention tags with either Delete and Allow Recovery or Permanently Delete actions, or archive tags, i.e. retention tags with the Move To Archive action, which move items to the user's archive mailbox. Depending on how they're applied to mailbox items, retention tags are categorized as the following three types: Default Policy Tags (DPTs), which apply to untagged items in the mailbox – untagged items being items that don't have a retention tag applied directly or by inheritance from parent folder. You can create three types of DPT s: an archive DPT, a delete DPT and a DPT for voicemail messages. Retention Policy Tags (RPTs), which are retention tags with a delete action, created for default folders such as Inbox and Deleted Items. Not all default folders are supported. You can find a table showing the default folders supported for RPT s in Understanding Retention Tags and Retention Policies. Notably, Calendar, Tasks and Contacts folders aren't supported 1 . Personal Tags, which are retention tags that users can apply to items and folders in Outlook 2010 and Outlook Web App. Personal tags can either be delete tags or archive tags. They're surfaced in Outlook 2010 and OWA as Retention policies and Archive policies. To deploy retention tags, you add them to a retention policy and apply the policy to mailbox users. In Exchange 2010 SP1, we added support for the Notes folder. In Exchange 2010 RTM, items in the Notes folder aren't processed. After you upgrade to SP1, if the user's retention policy doesn't have a RPT for the Notes folder, the DPT from the user's policy will apply to items in that folder. In existing deployments, your users may not be used to their notes being moved or deleted. To prevent the DPT from being applied to a default folder, you can create a disabled RPT for that folder (or disable any existing RPT for that folder). The Managed Folder Assistant, a mailbox assistant that processes mailbox items and applies retention policies, does not apply the retention action of a disabled tag. Since the item/folder still has a tag, it's not considered untagged and the DPT isn't applied to it. Figure 1: Create a disabled Retention Policy Tag for the Notes default folder to prevent the Default Policy Tag from being applied to items in that folder Note: You can create a disabled RPT for any supported default folder. Why are items in the Notes folder still archived? If you create a disabled RPT for the Notes folder, you'll see items in that folder are not deleted, but they do continue to be moved to the archive! Why does this happen? How do you prevent it? It's important to understand that: A retention policy can have a DPT to archive items (using the Move to Archive retention action) and a DPT to delete items (using the Delete and Allow Recovery or Permanently Delete retention actions). Both apply to untagged items. The move and delete actions are exclusive of each other. Mailbox folders and messages can have both types of tags applied - an archive tag and a delete tag. It's not an either/or proposition. If you create a disabled RPT for the Notes folder to not delete items, the archive DPT for the mailbox would still apply and move items. When it comes to archiving, there's only one archive policy that administrators can enforce – the DPT with 'Move to archive' action. You can't create a RPT with the 'Move to archive' action. This rules out using the disabled RPT approach to prevent items from being moved. How do you prevent items in a default folder from being archived? There's no admin-controlled way to prevent items in default folders from being archived 2 , short of removing the archive DPT from a retention policy. However, removing the archive DPT would result in messages not moving to archive automatically unless the user applies a personal tag to messages or folders. The workaround is to have users apply the Personal never move to archive personal tag (displayed as Never under Archive Policy in Outlook/ OWA ) to a default folder. The tag is included in the Default Archive and Retention Policy created by Exchange Setup. You can also add this tag to any Retention Policies you create. Figure 2: Users can apply the Never archive policy to a default folder to prevent items in that folder from being archived 1 Support for Calendar and Notes retention tags was added in Exchange 2010 SP2 RU4. 2 You can apply a disabled move tag to a folder in user's mailbox using EWS code/script. For details, see Using Exchange Web Services to Apply a Personal Tag to a Custom Folder. Applying a disabled archive policy to the Notes default folder You can't use Outlook 2010 or Outlook 2013 to apply an archive policy to the Notes default folder or individual notes items. If your users want to preven Notes items from being moved, they must apply a disabled move tag to the Notes folder using OWA . Figure 3: Apply Personal never move to archive policy to the Notes folder in Outlook Web App in Exchange 2013. The Exchange 2010 Outlook Web App UI differs slightly - it lists archive and retention policies separately. See a screenshot here. Bharat Suneja Updates 1/23/2013: In Exchange 2010 SP2 RU4, we added Calendar and Tasks retention tag support. You can prevent these from being moved or deleted by creating registry values. See Calendar and Tasks Retention Tag Support in Exchange 2010 SP2 RU4. 6/18/2013: Added screenshot - Applying disabled move tag to Notes folder in OWA and link to Using Exchange Web Services to Apply a Personal Tag to a Custom Folder.Exchange 2013/2016 Monitoring Mailboxes
Note 10/2022: This article also applies to Exchange Server 2019. Introduction Exchange Server 2013 introduced a new feature called Managed Availability, which is a built-in monitoring system with self-recovery capabilities. Managed Availability performs continuous tests (probes) that simulate end-user actions, to detect possible problems with Exchange components or their dependencies. If probes are failing, it performs gradual simple recovery actions to bring the affected component in healthy state. It uses special type of mailboxes, called Monitoring Mailboxes or health mailboxes, to simulate end-user kinds of tests. The life cycle of monitoring mailboxes is taken care entirely by Managed Availability components. In this post, we’ll see how Managed Availability takes care of monitoring mailboxes, what best practices to keep monitoring mailboxes happy are and some related troubleshooting. Monitoring Mailboxes Functioning of Managed Availability is implemented by Microsoft Exchange Health Manager Service running on each Exchange Server 2013 role. The Microsoft Exchange Health Manager Service is responsible for creating and maintaining monitoring mailboxes Let’s take a look at how the Health Manager creates them! How do we create monitoring mailboxes? The MS Exchange Health Manager Service runs in two processes, MSExchangeHMHost.exe and MSExchangeHMWorker.exe (let’s call it the HM worker). HM worker process, at the time of startup, checks for availability of monitoring mailboxes and creates monitoring mailboxes as needed. Starting with Exchange Server 2013 Cumulative Update 1, accounts for monitoring mailboxes are created under following container in the domain Exchange server resides: <ADdomain>\Microsoft Exchange System Objects\Monitoring Mailboxes Example: The logic HM worker process uses to detect and create monitoring mailboxes depends on Exchange Cumulative Update (CU) installed, Exchange role installed on the box and mailbox databases present. The Following logic was used to create monitoring mailboxes for Exchange Server 2013 servers between RTM to Cumulative Update 5: One monitoring mailbox per mailbox database copy, plus one for all of CAS servers. Here’s an example of monitoring mailboxes created on Exchange Server 2013 SP1 server that hosts both CAS and mailbox role: [PS] C:\>Get-Mailbox -Monitoring | ft displayname,whencreated DisplayName WhenCreated ----------- ----------- HealthMailboxb285a119be6649b3a89574f078e947f5 11/10/2014 9:07:29 AM HealthMailbox60d8a8d1285e41bfa5ce1ef1fb93d14e 11/10/2014 9:07:36 AM The display name of the monitoring mailbox created for database copy contained the GUID of the mailbox database for which it was created. Example: The display name of the monitoring mailbox created for CAS server contained the GUID of the Exchange server for which it was created. Example: The following logic is used to create monitoring mailbox for Exchange Server 2013 Cumulative Update 6 and onwards: One monitoring mailbox for each mailbox database copy hosted on mailbox role, plus ten monitoring mailboxes for each CAS server. The following naming convention is used for the display name of the monitoring mailbox created for a database: “HealthMailbox-“ + host name of server + “-“ + database name. At the startup, the HM worker checks for name of databases present on the server and then checks for presence of monitoring mailboxes with display name as explained above. If it doesn’t find a monitoring mailbox for a specific database, it creates a new monitoring mailbox. That means, if you rename DB1 to DB2, HM worker will create new monitoring mailbox for DB2 on next startup. And the following naming convention is used for display name of the monitoring mailbox created for a CAS server: “HealthMailbox-“ + host name of the CAS server + “-001” through “010.” We attempt to distribute the monitoring mailboxes created for CAS servers across the mailbox databases available. Let’s make this real: Imagine that you have only one Exchange server running Exchange 2013 CU6 or newer in the org (server is named EXCH1) that hosts both CAS and Mailbox role and has only one mailbox database, 11 monitoring mailboxes will be created. Example: The Health Manager Service will create more mailboxes according to the logic explained above as you go on adding more databases or server roles. Password resets HM Worker is responsible for maintaining the password for Monitoring mailboxes. HM worker uses a complex algorithm to generate password to be used for monitoring mailbox. The password for monitoring mailbox is reset under the following conditions: A new health mailbox is being created Each time HM Worker process starts and is not able to retrieve the existing password for the monitoring mailbox Any other scenario where HM Worker is not able to get hold of existing password for the monitoring mailbox Best practices Here are some best practices regarding management of user accounts associated with monitoring mailboxes as well as mailboxes themselves: Do not apply third party customized password policies to user accounts of monitoring mailboxes Exclude monitoring mailboxes from user account lockout policies Do not move the user accounts out from the Monitoring Mailboxes container Do not change the user account properties, like restricting password change etc. Do not change AD permission inheritance Since HM worker handles password resets for monitoring mailboxes, in a large environment, it is normal to see increased password reset traffic for monitoring mailbox accounts; note that doing one of the things above might increase the frequency of those resets Do not move the monitoring mailboxes between mailbox databases Do not apply mailbox quotas to monitoring mailboxes If applying a retention policy, ensure the data within the monitoring mailbox is retained for a minimum of 30 days before being deleted If you see a mailbox size increasing significantly for specific monitoring mailbox; you can simply disable the specific mailbox. The HM worker will create a new mailbox at next startup. Common tasks with monitoring mailboxes How to list monitoring mailboxes The Get-Mailbox command is provided with special parameter “-Monitoring” to list only the monitoring mailboxes. Here are some examples: To list all monitoring mailboxes present in the organization: Get-Mailbox –Monitoring To list monitoring mailboxes present on a database: Get-Mailbox –Monitoring -Database <database name> However, the mailboxes listed above may not be associated with the server on which database is hosted. As explained in creation logic, the name of Client Access Server or the mailbox database is important in searching for monitoring mailbox associated and in creation of monitoring mailbox. Use the following command to list the monitoring mailboxes associated with a specific server: Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-<servername>-*"} Example: Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-exch2-*"} | ft name,displayname,database Use the following command to list the monitoring mailbox associated with a specific database: Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-<name of database>"} Example: Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-db2exch2"} | ft name,displayname,database,servername Troubleshooting tips Here are some troubleshooting methods for monitoring mailboxes. How to re-create monitoring mailboxes (NOT considered regular maintenance!) The mailbox database removal doesn’t cleanup the AD user account associated with monitoring mailboxes. This can then result in orphaned AD user accounts. This happens because of deny permission inherited on Monitoring Mailboxes container. KB article 3046530has details on this, as well as the workaround to resolve it. If there are too many orphaned monitoring mailbox accounts, you may want to re-create them. Steps: 1) Make sure “Monitoring Mailboxes” container is present Open Active Directory Users & Computers Click on View and select “Advanced Features” The Browse to Microsoft Exchange System Objects Verify the presence of the “Monitoring Mailboxes” container. Example: If the Monitoring Mailboxes container is missing: Make sure you have Exchange Server 2013 CU1 or above installed. Perform PrepareAD with the Exchange Server 2013 version installed. 2) Stop the “Microsoft Exchange Server Health Manager” service on all Exchange Server 2013 servers. 3) Open Exchange Management Shell and use following command to disable existing health mailboxes: Get-Mailbox -Monitoring | Disable-Mailbox 4) Go back to Active Directory users & computers, right click on domain and search for “HealthMailbox” 5) Delete the health mailbox user accounts. 6) Wait for AD replication or force AD replication. 7) Start the “Microsoft Exchange Server Health Manager” on all Exchange Server 2013 servers. Bhalchandra Atre Updates 6/2/15: Updated best practices to include information on aging out data via retention policies.Store Driver Fault Isolation Improvements in Exchange 2010 SP1
Background The Exchange Store Driver is a core transport component which lives both on the Mailbox server role (as the mail submission service) and the Hub server role. It is responsible for: Retrieving messages from the mailbox server that have been submitted by end-users and submitting those to the Hub transport role for categorization and routing. Delivering messages to the appropriate mailbox server based on the location of the recipients mailbox. Extensibility platform for both mail submission & delivery. Store Driver currently hosts a number of agents that extend the functionality of Exchange. Examples include such agents as Inbox Rules, Conversations, meeting forward notifications, etc. Exchange 2010 is currently being utilized in Live@EDU, as well as the upcoming Office 365. As you can probably imagine, the Exchange servers that run in those datacenters are loaded and pushed harder than almost any other Exchange server imaginable. Prior to SP 1, there were several problems that were encountered with mail delivery to the Exchange mailbox store. In particular, there was a need to make sure that a handful of recipients did not starve the rest of the mail delivery system. While many of you may not have noticed this problem, Microsoft has seen many of these types of cases over the years; often isolated to a single event like an inadvertent public folder replication storm. This was despite the message throttling that was already available. Transport roles have also had functionality to avoid resource starvation known as Back Pressure, but this was not designed to protect the system from messages that were already in the Local Delivery queue. Changes in SP1 In order to further protect both the Mailbox servers and Hub servers from resource starvation, new thread limits were introduced in SP1: Key Description Scenario Error in Connectivity Log: <add key=”RecipientThreadLimit” value=”1”/> Limit beyond which no more threads can be allocated to the recipient for delivery. Note: If this is increased, you should increase MaxMailboxDeliveryPerMdbConnections as well, so that slow or hung deliveries to a single recipient will not block delivery for the entire MDB. Flood of messages to a single Mailbox or a performance problem associated with a single mailbox, has minimal impact on delivery to the rest of the Mailboxes in the database. Throttled delivery for recipient <recipient> due to concurrency limit <limit> <add key=”MaxMailboxDeliveryPerMdbConnections” value=”2”/> The maximum number of concurrent connections to a single “healthy” Mailbox Database. Database health is determined by the Health Monitor API and recorded in the connectivity logs as a value between -1, 0-100. 100 being healthy. Connections hang to a single problematic database have minimal impact on delivery of other queued messages Throttled Delivery due to server limit for <server FQDN> with threshold Note: These keys are not present in the EdgeTransport.exe.config file by default. Is it possible to have too much protection? Unfortunately, there are two scenarios after applying SP1 where we are seeing customers with messages backing up in the queue. The temporary error message is: 432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded As you can probably guess, the two scenarios are: Journaling Public Folders In both cases, the deliveries are occurring to a single recipient (or very small number of recipients). This is likely to occur during heavy mail flow. The screen shot below was taken from a lab server while reproducing the issue: Figure 1: Messages backed up in mailbox delivery queue due to recipient thread limits being exceeded (click here for larger screenshot) You can see a historical history of 4.3.2 events in connectivity logs on your Hub Transport servers (in the \Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\Connectivity\CONNECTLOGxxxxxxxx-x.LOG), like: #Software: Microsoft Exchange Server #Version: 14.0.0.0 #Log-type: Transport Connectivity Log #Date: 2011-01-12T00:00:00.775Z #Fields: date-time,session,source,Destination,direction,description 2011-01-12T00:00:00.775Z,08CD7F1200CBDBD0,MapiSubmission,5f24e416-c380-41b5-bfe0-37b6f1091f49,>,"Failed; HResult: 1140850693; DiagnosticInfo: Stage:LoadItem, SmtpResponse:432-4.3.2 STOREDRV; mailbox server is too busy" 2011-01-12T00:00:00.775Z,08CD7F1200CBDBD0,MapiSubmission,5f24e416-c380-41b5-bfe0-37b6f1091f49,-,RegularSubmissions: 0 ShadowSubmissions: 0 Bytes: 0 Recipients: 0 Failures: 1 ReachedLimit: False Idle: False 2011-01-12T00:00:05.792Z,08CD7F1200CBDBD1,MapiSubmission,5f24e416-c380-41b5-bfe0-37b6f1091f49,+,Win2k8R2Ex14.dom2k8r2ex14.lab In some cases, simply leaving the servers alone should cause the queues to slowly drain. In other cases, it may be necessary to take further action because the problem has persisted for a while or because it isn’t part of a one-time event like a Public Folder replication storm. Alright, I understand. Now how do I fix my situation? Like every other throttling & performance related feature that has ever been in Exchange, the solution isn’t exactly straight forward. For starters, we’ve seen some level of success simply incrementing both values up one, as follows: <add key="RecipientThreadLimit" value="2" /> <add key="MaxMailboxDeliveryPerMdbConnections" value="3" /> In fact, there has been en ough success with this that we’re considering changing the defaults in a future rollup or service pack. Of course, when we do, the new defaults will only apply to those who haven’t already modified the values. Although it has not yet been released, KB 2491972 will discuss the change when the time comes. So if +1 is better, then why not +2 or more? Well, the problem is that all Exchange servers will ultimately be bound by some hardware resource. You don’t want to introduce thrashing or resource starvation due to a public folder or journaling server event that now impacts all users as well. In addition, there are other limits that control the maximum number of threads that can service these types of deliveries. In short, we are not recommending going above 4 for MaxMailboxDeliveryPerMdbConnections, and RecipientThreadLimit should always be at least one fewer. In an extreme case, if you are in a very busy environment with dedicated journaling mailbox servers, it may be worth considering having dedicated hub transport servers to go along with that. Of course, they can be virtualized or multi-role boxes, but in order to be isolated, the whole bunch will need to be in a dedicated site. Then, you would be able to increase these limits without risking delivery to other mailboxes. Of course, this is an extreme example. For the rest of us, it may be as simple as trying the +1 approach while carefully monitoring the hub & mailbox servers. Scott Landry, Steve Schiemann, Frank Brown and Jason Nelson
Restore Teams private chat history to new mailbox
Hi, I'm having an problem with restoring Teams chat history to new user accounts and and their mailbox. I disabled Azure AD sync for a few users and deleted their account in the cloud tenant from the recyclebin and enabled the sync again to solve an issue with calendar syncing with on-premise mailboxes. Unfortunately their Teams chat history also vanished from their Teams account and now I'm searching for a way to restore their history. I can see the deleted mailboxes and should be able to restore the whole mailbox to a temporary account but I have no idea if the missing data is in there or how I can move the Teams chat history to the new mailbox. Can anyone give me hint or solution? Thanks and Regards
