Outlook clients receive error 0x8004010f when downloading the Offline Address Book
EDIT 10/01/2008: To read the updated (and more comprehensive) guidance for troubleshooting errors 0x8004010f, please go here. There are a multiple reasons for why an Outlook client can receive the 0x8004010f sync error. Unfortunately, 0x8004010f is just a generic MAPI error and will show up for a variety of problems. Here is what the error looks like under err.exe (Microsoft Exchange Server Error Code Look-up Tool): C:\WXP\system32>err 0x8004010f # for hex 0x8004010f / decimal -2147221233 ecNotFound ec.h ecAttachNotFound ec.h ecUnknownRecip ec.h ecPropNotExistent ec.h MAPI_E_NOT_FOUND mapicode.h # 5 matches found for "0x8004010f" This is what the error looks like from the sync log from within Outlook: 12:45:53 Synchronizing Mailbox <dgoldman> 12:45:53 Done 12:45:54 Microsoft Exchange offline address book 12:45:54 0x8004010f Some of the most common reasons for Outlook clients to receive the 0x8004010f error with regards to downloading the OAB are listed below. Most of these are documented and I have linked articles to each of these to help everybody out. Please note that these solutions can change a small bit depending on unknown factors in a company's environment. An administrator decommissioned the last Exchange server in a site and never pushes replicas to another Exchange server. A new OAB is created in the active directory and the information store never reads the active directory during its maintenance schedule. This will result in the OAB files never being generated and the Outlook client will fail to download anything. The information store has an invalid EntryID that points to the legacy EX:/ folders. Again there is nothing for the client to download. An outlook client logging in from one domain to another domain and attempting to log in to another users mailbox. The OAB was never generated or some OAB folders are missing from the public folder store. Multiple OAB Version folders exist of the same type. Clients are attempting to download the OAB files from a public folder store that have not received the replicated updates. The offline address book list object has a missing address list. The offline address book list object has an incorrect address list. Send/As changes in the store affect users accounts with no mailbox full rights to another mailbox. If you are seeing this error on an Exchange 2007 server and your OAB is generated by an Exchange 2007 server, please make sure of the following: 1. Make sure that you have added the replicas of OAB to the Exchange 2007 server 2. Make sure public folder replication is working. 3. Make sure the OAB is public folder enabled and you have OAB Version 2, OAB Version 3 and OAB Version 4 checked off so your legacy clients can download the OAB files from the public folder store. 4. Make sure that if you are using an Outlook 2007 client, your OAB is Web Distribution enabled and the OAB files have been replicated over to the Client Access Server. For more information on this process please see this blog: http://blogs.msdn.com/dgoldman/archive/2006/10/23/outlook-client-fails-to-download-the-oab-with-error-0x8004011b.aspx and http://blogs.msdn.com/dgoldman/archive/2006/08/25/How-Exchange-2007-OAB-Files-are-replicated-to-a-Client-Access-Server-for-download.aspx 5. If you are removing your last Exchange 2003 server from the org, please make sure that you follow our documentation on this process. - Dave GoldmanWant more control over Sent Items when using shared mailboxes?
Edit 6/15/2015: We are starting to make this feature available again in Office 365. Note that the new shared mailbox Sent Items behavior is disabled by default. If you want to enable it for your users, you can do so by using cmdlets mentioned below. Additionally, we are still on track to ship this to our on-premises customers in Exchange 2013 CU9, as mentioned before. Whether a mailbox is used by multiple users as a collaborative tool or a communication gateway to customers, retaining a record of emails sent from a shared mailbox remains an important business requirement. In Exchange 2010, there was a way to configure this behavior, but we did not have this feature starting with Exchange 2013. Our customers have told us that a shared mailbox should keep a copy of emails sent from the mailbox by all members of the mailbox in its own Sent Items folder. We have taken that feedback and decided to make some changes to how sent mails are handled for shared mailboxes. We are excited to announce that once this feature is enabled for you (see below), by default all shared mailboxes will retain a copy of emails sent from the mailbox. You will no longer have to figure out which mailbox member sent an email as the shared mailbox or on behalf of it. How does it work? Emails can be sent as the shared mailbox itself or on behalf of it by member(s) of the mailbox, assuming proper permissions have been granted. This feature is designed to retain a copy of an email sent from the shared mailbox in the Sent Items folder of the shared mailbox. The same behavior can be expected for emails sent on behalf of the shared mailbox, when configured to do so. A copy of the sent mail will also reside in the Sent Items folder of the member’s personal mailbox. Note: If the user has used the Outlook 2013 feature to change the folder that Sent Items are saved to, the messages will be copied to that folder instead of the user’s Sent Items folder. Users can reconfigure this by clicking the “Save Sent Items To” button on the Email Options tab. Administrators have control over this feature for either mails Sent As or Sent on Behalf of a shared mailbox. The table below summarizes where sent mails reside when members of a shared mailbox send mail from the shared mailbox. User Mailbox Shared Mailbox Sent Items Exchange 2010 Exchange 2010 Controlled through settings in KB2632409 Exchange 2010 Exchange 2013 (any version) Controlled through settings in KB2632409 Exchange 2013 CU9 (or newer) Office 365 Exchange 2010 The sent mail will be delivered to both the Inbox of the shared mailbox as an email attachment* and to the user mailbox' sent items Exchange 2013 CU9 (or newer) Office 365 Exchange 2013 CU9 (or newer) Office 365 The sent mail will be delivered to the Sent Items folder of shared mailbox and to the user mailbox' sent items *In a scenario where the user’s mailbox is on an Exchange 2013 CU9 server (or an Exchange 2013 without CU9 installed) and the shared mailbox is on an Exchange 2010 server, the shared mailbox will get an email message that looks like the following: This behavior will continue until the shared mailbox is moved to an Exchange 2013 CU9 server. Because this can happen even between Exchange 2013 servers (pre-CU9 and CU9), you might want to wait in turning the feature on for your on-premises deployment until you have completely rolled out Exchange 2013 CU9 to all of your servers. Who can use this feature? The feature is going to be available to all customers with shared mailboxes in Office 365 (phased rollout starting on 6/15/2015), as well as our on-premises customers (starting with Exchange 2013 CU9). Note: if after installation of Exchange 2013 CU9 in an on-premises environment you do not see the new CMDlet parameters, you should run /preparead explicitly with CU9 setup. How do I enable/disable this feature? In Office 365 and Exchange 2013 CU9, this feature is disabled by default. Enable the feature In Office 365, using the Admin portal, you can select your shared mailbox by going to Groups > Shared mailboxes first: Then Edit the Sent items copy status of the mailbox: Using PowerShell, for emails Sent As the shared mailbox: set-mailbox <mailbox name> -MessageCopyForSentAsEnabled $True For emails Sent On Behalf of the shared mailbox: set-mailbox <mailbox name> -MessageCopyForSendOnBehalfEnabled $True Disable the feature For messages Sent As the shared mailbox: set-mailbox <mailbox name> -MessageCopyForSentAsEnabled $False For emails Sent On Behalf of the shared mailbox: set-mailbox <mailbox name> -MessageCopyForSendOnBehalfEnabled $False What else do I need to know? The administrator for your organization has to create the shared mailbox and add you to it as an user, before you can use it. If you are an Office 365 Small Business administrator, see Create and use shared mailboxes. If you are an administrator for a different version of Office 365 or an on-premises Exchange administrator, see the TechNet article Create a Shared Mailbox. If you are currently using a client registry key to duplicate messages sent from a shared mailbox to a sent items folder of the sender, you should remember to remove client registry keys if you enable this feature server side. If you do not remove client registry keys, you could see duplicate messages in Sent Items folders (one created by client registry keys and one by the server). If one of the transport rules prevents messages larger than certain size and the message being sent is over that size, then the message will not be delivered to the its original recipient and will also not be copied to the sent items of the shared mailbox. Philip LohPrevent archiving of items in a default folder in Exchange 2010
In Exchange 2010, you can use Retention Policies to manage message retention. Retention Policies consist of delete tags, i.e. retention tags with either Delete and Allow Recovery or Permanently Delete actions, or archive tags, i.e. retention tags with the Move To Archive action, which move items to the user's archive mailbox. Depending on how they're applied to mailbox items, retention tags are categorized as the following three types: Default Policy Tags (DPTs), which apply to untagged items in the mailbox – untagged items being items that don't have a retention tag applied directly or by inheritance from parent folder. You can create three types of DPT s: an archive DPT, a delete DPT and a DPT for voicemail messages. Retention Policy Tags (RPTs), which are retention tags with a delete action, created for default folders such as Inbox and Deleted Items. Not all default folders are supported. You can find a table showing the default folders supported for RPT s in Understanding Retention Tags and Retention Policies. Notably, Calendar, Tasks and Contacts folders aren't supported 1 . Personal Tags, which are retention tags that users can apply to items and folders in Outlook 2010 and Outlook Web App. Personal tags can either be delete tags or archive tags. They're surfaced in Outlook 2010 and OWA as Retention policies and Archive policies. To deploy retention tags, you add them to a retention policy and apply the policy to mailbox users. In Exchange 2010 SP1, we added support for the Notes folder. In Exchange 2010 RTM, items in the Notes folder aren't processed. After you upgrade to SP1, if the user's retention policy doesn't have a RPT for the Notes folder, the DPT from the user's policy will apply to items in that folder. In existing deployments, your users may not be used to their notes being moved or deleted. To prevent the DPT from being applied to a default folder, you can create a disabled RPT for that folder (or disable any existing RPT for that folder). The Managed Folder Assistant, a mailbox assistant that processes mailbox items and applies retention policies, does not apply the retention action of a disabled tag. Since the item/folder still has a tag, it's not considered untagged and the DPT isn't applied to it. Figure 1: Create a disabled Retention Policy Tag for the Notes default folder to prevent the Default Policy Tag from being applied to items in that folder Note: You can create a disabled RPT for any supported default folder. Why are items in the Notes folder still archived? If you create a disabled RPT for the Notes folder, you'll see items in that folder are not deleted, but they do continue to be moved to the archive! Why does this happen? How do you prevent it? It's important to understand that: A retention policy can have a DPT to archive items (using the Move to Archive retention action) and a DPT to delete items (using the Delete and Allow Recovery or Permanently Delete retention actions). Both apply to untagged items. The move and delete actions are exclusive of each other. Mailbox folders and messages can have both types of tags applied - an archive tag and a delete tag. It's not an either/or proposition. If you create a disabled RPT for the Notes folder to not delete items, the archive DPT for the mailbox would still apply and move items. When it comes to archiving, there's only one archive policy that administrators can enforce – the DPT with 'Move to archive' action. You can't create a RPT with the 'Move to archive' action. This rules out using the disabled RPT approach to prevent items from being moved. How do you prevent items in a default folder from being archived? There's no admin-controlled way to prevent items in default folders from being archived 2 , short of removing the archive DPT from a retention policy. However, removing the archive DPT would result in messages not moving to archive automatically unless the user applies a personal tag to messages or folders. The workaround is to have users apply the Personal never move to archive personal tag (displayed as Never under Archive Policy in Outlook/ OWA ) to a default folder. The tag is included in the Default Archive and Retention Policy created by Exchange Setup. You can also add this tag to any Retention Policies you create. Figure 2: Users can apply the Never archive policy to a default folder to prevent items in that folder from being archived 1 Support for Calendar and Notes retention tags was added in Exchange 2010 SP2 RU4. 2 You can apply a disabled move tag to a folder in user's mailbox using EWS code/script. For details, see Using Exchange Web Services to Apply a Personal Tag to a Custom Folder. Applying a disabled archive policy to the Notes default folder You can't use Outlook 2010 or Outlook 2013 to apply an archive policy to the Notes default folder or individual notes items. If your users want to preven Notes items from being moved, they must apply a disabled move tag to the Notes folder using OWA . Figure 3: Apply Personal never move to archive policy to the Notes folder in Outlook Web App in Exchange 2013. The Exchange 2010 Outlook Web App UI differs slightly - it lists archive and retention policies separately. See a screenshot here. Bharat Suneja Updates 1/23/2013: In Exchange 2010 SP2 RU4, we added Calendar and Tasks retention tag support. You can prevent these from being moved or deleted by creating registry values. See Calendar and Tasks Retention Tag Support in Exchange 2010 SP2 RU4. 6/18/2013: Added screenshot - Applying disabled move tag to Notes folder in OWA and link to Using Exchange Web Services to Apply a Personal Tag to a Custom Folder.Exchange 2013/2016 Monitoring Mailboxes
Note 10/2022: This article also applies to Exchange Server 2019. Introduction Exchange Server 2013 introduced a new feature called Managed Availability, which is a built-in monitoring system with self-recovery capabilities. Managed Availability performs continuous tests (probes) that simulate end-user actions, to detect possible problems with Exchange components or their dependencies. If probes are failing, it performs gradual simple recovery actions to bring the affected component in healthy state. It uses special type of mailboxes, called Monitoring Mailboxes or health mailboxes, to simulate end-user kinds of tests. The life cycle of monitoring mailboxes is taken care entirely by Managed Availability components. In this post, we’ll see how Managed Availability takes care of monitoring mailboxes, what best practices to keep monitoring mailboxes happy are and some related troubleshooting. Monitoring Mailboxes Functioning of Managed Availability is implemented by Microsoft Exchange Health Manager Service running on each Exchange Server 2013 role. The Microsoft Exchange Health Manager Service is responsible for creating and maintaining monitoring mailboxes Let’s take a look at how the Health Manager creates them! How do we create monitoring mailboxes? The MS Exchange Health Manager Service runs in two processes, MSExchangeHMHost.exe and MSExchangeHMWorker.exe (let’s call it the HM worker). HM worker process, at the time of startup, checks for availability of monitoring mailboxes and creates monitoring mailboxes as needed. Starting with Exchange Server 2013 Cumulative Update 1, accounts for monitoring mailboxes are created under following container in the domain Exchange server resides: <ADdomain>\Microsoft Exchange System Objects\Monitoring Mailboxes Example: The logic HM worker process uses to detect and create monitoring mailboxes depends on Exchange Cumulative Update (CU) installed, Exchange role installed on the box and mailbox databases present. The Following logic was used to create monitoring mailboxes for Exchange Server 2013 servers between RTM to Cumulative Update 5: One monitoring mailbox per mailbox database copy, plus one for all of CAS servers. Here’s an example of monitoring mailboxes created on Exchange Server 2013 SP1 server that hosts both CAS and mailbox role: [PS] C:\>Get-Mailbox -Monitoring | ft displayname,whencreated DisplayName WhenCreated ----------- ----------- HealthMailboxb285a119be6649b3a89574f078e947f5 11/10/2014 9:07:29 AM HealthMailbox60d8a8d1285e41bfa5ce1ef1fb93d14e 11/10/2014 9:07:36 AM The display name of the monitoring mailbox created for database copy contained the GUID of the mailbox database for which it was created. Example: The display name of the monitoring mailbox created for CAS server contained the GUID of the Exchange server for which it was created. Example: The following logic is used to create monitoring mailbox for Exchange Server 2013 Cumulative Update 6 and onwards: One monitoring mailbox for each mailbox database copy hosted on mailbox role, plus ten monitoring mailboxes for each CAS server. The following naming convention is used for the display name of the monitoring mailbox created for a database: “HealthMailbox-“ + host name of server + “-“ + database name. At the startup, the HM worker checks for name of databases present on the server and then checks for presence of monitoring mailboxes with display name as explained above. If it doesn’t find a monitoring mailbox for a specific database, it creates a new monitoring mailbox. That means, if you rename DB1 to DB2, HM worker will create new monitoring mailbox for DB2 on next startup. And the following naming convention is used for display name of the monitoring mailbox created for a CAS server: “HealthMailbox-“ + host name of the CAS server + “-001” through “010.” We attempt to distribute the monitoring mailboxes created for CAS servers across the mailbox databases available. Let’s make this real: Imagine that you have only one Exchange server running Exchange 2013 CU6 or newer in the org (server is named EXCH1) that hosts both CAS and Mailbox role and has only one mailbox database, 11 monitoring mailboxes will be created. Example: The Health Manager Service will create more mailboxes according to the logic explained above as you go on adding more databases or server roles. Password resets HM Worker is responsible for maintaining the password for Monitoring mailboxes. HM worker uses a complex algorithm to generate password to be used for monitoring mailbox. The password for monitoring mailbox is reset under the following conditions: A new health mailbox is being created Each time HM Worker process starts and is not able to retrieve the existing password for the monitoring mailbox Any other scenario where HM Worker is not able to get hold of existing password for the monitoring mailbox Best practices Here are some best practices regarding management of user accounts associated with monitoring mailboxes as well as mailboxes themselves: Do not apply third party customized password policies to user accounts of monitoring mailboxes Exclude monitoring mailboxes from user account lockout policies Do not move the user accounts out from the Monitoring Mailboxes container Do not change the user account properties, like restricting password change etc. Do not change AD permission inheritance Since HM worker handles password resets for monitoring mailboxes, in a large environment, it is normal to see increased password reset traffic for monitoring mailbox accounts; note that doing one of the things above might increase the frequency of those resets Do not move the monitoring mailboxes between mailbox databases Do not apply mailbox quotas to monitoring mailboxes If applying a retention policy, ensure the data within the monitoring mailbox is retained for a minimum of 30 days before being deleted If you see a mailbox size increasing significantly for specific monitoring mailbox; you can simply disable the specific mailbox. The HM worker will create a new mailbox at next startup. Common tasks with monitoring mailboxes How to list monitoring mailboxes The Get-Mailbox command is provided with special parameter “-Monitoring” to list only the monitoring mailboxes. Here are some examples: To list all monitoring mailboxes present in the organization: Get-Mailbox –Monitoring To list monitoring mailboxes present on a database: Get-Mailbox –Monitoring -Database <database name> However, the mailboxes listed above may not be associated with the server on which database is hosted. As explained in creation logic, the name of Client Access Server or the mailbox database is important in searching for monitoring mailbox associated and in creation of monitoring mailbox. Use the following command to list the monitoring mailboxes associated with a specific server: Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-<servername>-*"} Example: Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-exch2-*"} | ft name,displayname,database Use the following command to list the monitoring mailbox associated with a specific database: Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-<name of database>"} Example: Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-db2exch2"} | ft name,displayname,database,servername Troubleshooting tips Here are some troubleshooting methods for monitoring mailboxes. How to re-create monitoring mailboxes (NOT considered regular maintenance!) The mailbox database removal doesn’t cleanup the AD user account associated with monitoring mailboxes. This can then result in orphaned AD user accounts. This happens because of deny permission inherited on Monitoring Mailboxes container. KB article 3046530has details on this, as well as the workaround to resolve it. If there are too many orphaned monitoring mailbox accounts, you may want to re-create them. Steps: 1) Make sure “Monitoring Mailboxes” container is present Open Active Directory Users & Computers Click on View and select “Advanced Features” The Browse to Microsoft Exchange System Objects Verify the presence of the “Monitoring Mailboxes” container. Example: If the Monitoring Mailboxes container is missing: Make sure you have Exchange Server 2013 CU1 or above installed. Perform PrepareAD with the Exchange Server 2013 version installed. 2) Stop the “Microsoft Exchange Server Health Manager” service on all Exchange Server 2013 servers. 3) Open Exchange Management Shell and use following command to disable existing health mailboxes: Get-Mailbox -Monitoring | Disable-Mailbox 4) Go back to Active Directory users & computers, right click on domain and search for “HealthMailbox” 5) Delete the health mailbox user accounts. 6) Wait for AD replication or force AD replication. 7) Start the “Microsoft Exchange Server Health Manager” on all Exchange Server 2013 servers. Bhalchandra Atre Updates 6/2/15: Updated best practices to include information on aging out data via retention policies.Store Driver Fault Isolation Improvements in Exchange 2010 SP1
Background The Exchange Store Driver is a core transport component which lives both on the Mailbox server role (as the mail submission service) and the Hub server role. It is responsible for: Retrieving messages from the mailbox server that have been submitted by end-users and submitting those to the Hub transport role for categorization and routing. Delivering messages to the appropriate mailbox server based on the location of the recipients mailbox. Extensibility platform for both mail submission & delivery. Store Driver currently hosts a number of agents that extend the functionality of Exchange. Examples include such agents as Inbox Rules, Conversations, meeting forward notifications, etc. Exchange 2010 is currently being utilized in Live@EDU, as well as the upcoming Office 365. As you can probably imagine, the Exchange servers that run in those datacenters are loaded and pushed harder than almost any other Exchange server imaginable. Prior to SP 1, there were several problems that were encountered with mail delivery to the Exchange mailbox store. In particular, there was a need to make sure that a handful of recipients did not starve the rest of the mail delivery system. While many of you may not have noticed this problem, Microsoft has seen many of these types of cases over the years; often isolated to a single event like an inadvertent public folder replication storm. This was despite the message throttling that was already available. Transport roles have also had functionality to avoid resource starvation known as Back Pressure, but this was not designed to protect the system from messages that were already in the Local Delivery queue. Changes in SP1 In order to further protect both the Mailbox servers and Hub servers from resource starvation, new thread limits were introduced in SP1: Key Description Scenario Error in Connectivity Log: <add key=”RecipientThreadLimit” value=”1”/> Limit beyond which no more threads can be allocated to the recipient for delivery. Note: If this is increased, you should increase MaxMailboxDeliveryPerMdbConnections as well, so that slow or hung deliveries to a single recipient will not block delivery for the entire MDB. Flood of messages to a single Mailbox or a performance problem associated with a single mailbox, has minimal impact on delivery to the rest of the Mailboxes in the database. Throttled delivery for recipient <recipient> due to concurrency limit <limit> <add key=”MaxMailboxDeliveryPerMdbConnections” value=”2”/> The maximum number of concurrent connections to a single “healthy” Mailbox Database. Database health is determined by the Health Monitor API and recorded in the connectivity logs as a value between -1, 0-100. 100 being healthy. Connections hang to a single problematic database have minimal impact on delivery of other queued messages Throttled Delivery due to server limit for <server FQDN> with threshold Note: These keys are not present in the EdgeTransport.exe.config file by default. Is it possible to have too much protection? Unfortunately, there are two scenarios after applying SP1 where we are seeing customers with messages backing up in the queue. The temporary error message is: 432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded As you can probably guess, the two scenarios are: Journaling Public Folders In both cases, the deliveries are occurring to a single recipient (or very small number of recipients). This is likely to occur during heavy mail flow. The screen shot below was taken from a lab server while reproducing the issue: Figure 1: Messages backed up in mailbox delivery queue due to recipient thread limits being exceeded (click here for larger screenshot) You can see a historical history of 4.3.2 events in connectivity logs on your Hub Transport servers (in the \Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\Connectivity\CONNECTLOGxxxxxxxx-x.LOG), like: #Software: Microsoft Exchange Server #Version: 14.0.0.0 #Log-type: Transport Connectivity Log #Date: 2011-01-12T00:00:00.775Z #Fields: date-time,session,source,Destination,direction,description 2011-01-12T00:00:00.775Z,08CD7F1200CBDBD0,MapiSubmission,5f24e416-c380-41b5-bfe0-37b6f1091f49,>,"Failed; HResult: 1140850693; DiagnosticInfo: Stage:LoadItem, SmtpResponse:432-4.3.2 STOREDRV; mailbox server is too busy" 2011-01-12T00:00:00.775Z,08CD7F1200CBDBD0,MapiSubmission,5f24e416-c380-41b5-bfe0-37b6f1091f49,-,RegularSubmissions: 0 ShadowSubmissions: 0 Bytes: 0 Recipients: 0 Failures: 1 ReachedLimit: False Idle: False 2011-01-12T00:00:05.792Z,08CD7F1200CBDBD1,MapiSubmission,5f24e416-c380-41b5-bfe0-37b6f1091f49,+,Win2k8R2Ex14.dom2k8r2ex14.lab In some cases, simply leaving the servers alone should cause the queues to slowly drain. In other cases, it may be necessary to take further action because the problem has persisted for a while or because it isn’t part of a one-time event like a Public Folder replication storm. Alright, I understand. Now how do I fix my situation? Like every other throttling & performance related feature that has ever been in Exchange, the solution isn’t exactly straight forward. For starters, we’ve seen some level of success simply incrementing both values up one, as follows: <add key="RecipientThreadLimit" value="2" /> <add key="MaxMailboxDeliveryPerMdbConnections" value="3" /> In fact, there has been en ough success with this that we’re considering changing the defaults in a future rollup or service pack. Of course, when we do, the new defaults will only apply to those who haven’t already modified the values. Although it has not yet been released, KB 2491972 will discuss the change when the time comes. So if +1 is better, then why not +2 or more? Well, the problem is that all Exchange servers will ultimately be bound by some hardware resource. You don’t want to introduce thrashing or resource starvation due to a public folder or journaling server event that now impacts all users as well. In addition, there are other limits that control the maximum number of threads that can service these types of deliveries. In short, we are not recommending going above 4 for MaxMailboxDeliveryPerMdbConnections, and RecipientThreadLimit should always be at least one fewer. In an extreme case, if you are in a very busy environment with dedicated journaling mailbox servers, it may be worth considering having dedicated hub transport servers to go along with that. Of course, they can be virtualized or multi-role boxes, but in order to be isolated, the whole bunch will need to be in a dedicated site. Then, you would be able to increase these limits without risking delivery to other mailboxes. Of course, this is an extreme example. For the rest of us, it may be as simple as trying the +1 approach while carefully monitoring the hub & mailbox servers. Scott Landry, Steve Schiemann, Frank Brown and Jason NelsonCommon mailbox recovery scenarios for hybrid environments
Within the support organization at Microsoft we definitely see cases where customers are trying to recover deleted mailboxes. Typically, by the time a customer has contacted us they have tried everything they know as well as suggestions found online to recover the mailbox. It is often a completely avoidable and honest mistake that led to the deletion of the user’s Active Directory account in the first place. If you ever find yourself having a similarly bad day, then this article is meant to be your guide to not only getting through it, but to also come out of it as a superhero who is able to recover user accounts and mailboxes to a fully functional state without data loss. What you will learn from this article The main objective of this article is to help you recover a cloud mailbox after the corresponding on-premises user account has been deleted. If you do not have Directory Synchronization in place, then this article is not for you. With no Directory Synchronization in place you should view this article instead. Scenario that are covered There are many different mailbox recovery scenarios you may find yourself in. We will cover the most common scenarios throughout the course of this article to assist you in identifying the best recovery option for your own situation. Recover a mailbox that was deleted due to Directory Synchronization filtering changes resulting in filtering out the on-premises Active Directory user account associated with the cloud mailbox Recover a mailbox when the on-premises Active Directory user account associated with the cloud mailbox was accidentally or purposely deleted Recover a mailbox when the on-premises Active Directory user account associated with the cloud mailbox was accidentally or purposely deleted and the mailbox is on litigation hold All of the above scenarios have the same result. The associated user account in Office 365 becomes deleted due to one of the scenarios above and causes the mailbox to go into a soft-deleted state. Mailboxes in a soft-deleted state are recoverable for a period of 30 days before they are permanently removed from Office 365 and become unrecoverable. It is extremely important to attempt a proper user account recovery before blindly creating a new user account and merging the mailbox data. If you are able to restore the user account properly you will likely not lose any of the user data from the other services such as OneDrive and SharePoint. In addition, the user impact is pretty much non-existent when the original account is restored. There would be no need to create new profiles, no need to reset passwords, the user could simply log in and resume working from where they left off before. Recovery Process One of the challenges in restoring a mailbox is knowing which recovery option to use. If you know the recovery option you need, then can jump to it using the hyperlinks below. Otherwise we invite you to follow along with the article and we will guide you to the proper place. Restore a user account that was removed due to Directory Synchronization scope changes Restore a user account that was removed from on-premises AD with the recycle bin enabled Restore a user account that was removed from on-premises AD with no recycle bin enabled Restore a deleted user’s mailbox data to a new or alternate mailbox Restore an inactive user mailbox Restore a user that was removed due to Directory Synchronization Scope changes In some of the more complex customer environments it is sometimes beneficial to synchronize only certain Active Directory groups or Organizational Units (OUs) into Office 365. While this is not a common practice for most of our customers the process to configure filtering (if you are not familiar) is documented here. When configured, if a user is moved from an OU that is being synchronized to an OU that is not being synchronized, then Office 365 will see this action as a user account deletion. This causes the user account to be deleted within Office 365 and as a result the user’s mailbox also ends up in the previously mentioned soft-deleted state. The good news is the recovery for this scenario is quite simple. All you need to do is move the user back into the OU they were originally in. Assuming that the OU the user was previously in is still being synchronized, the next time Directory Synchronization complete the user and all associated data will be restored. By default, directory synchronizations occur every three hours and after you move the user back to the proper OU you will have to wait for the next sync cycle to take place. However, if you are like me and cannot wait, then you can force the synchronization. This article contains the necessary information to force a synchronization to take place immediately. Restore a user account that was removed from on-premises AD with the recycle bin enabled This scenario is a bit more common than the previous one. Many of our customers have realized the benefits that come with having the Active Directory recycle bin enabled. If you are not familiar with this feature and you are interested in learning more, then you can check it out here. The Active Directory recycle bin works as it sounds, when an object is deleted you can essentially undo the deletion without the kind of complex AD authoritative restoration process we all used and loved in the past. The good news is if you have the Active Directory recycle bin feature, it is a valid option to recover the deleted user. However, if you deleted the user prior to enabling this feature in your environment, then it will be of no help. Recovery steps: Follow this guidance to restore the user account using LDP or PowerShell Note: While less common, if you are using Directory Synchronization filtering (explained here), you need to be sure you restore the user to an OU that is within the Directory Synchronization scope. In an environment where more than one domain controller exists, ensure that the restored object has replicated before you proceed. Wait until your next Directory Synchronization cycle has complete or follow this article to force an immediate synchronization if you are using AAD Connect It is that easy and you will end up with the mailbox and all of the rest of the user data intact. Restore a user account that was removed from on-premises AD with no recycle bin enabled If you made it this far in the document, you likely are thinking “darn it I should have enabled the recycle bin for Active Directory”. While I agree with that sentiment, all is not lost. You can still recover your user account and mailbox data. In addition, you can still recover the data associated with other services, you just have a more difficult process to follow. The reason we try so har d to restore the original user account is so all of the data associated with the user is also restored. If you were to recreate a new user account on-premises (even with the same name as the deleted user), when the user syncs to Office 365 it will have a new object GUID. This means that any SharePoint, OneDrive, Exchange, and any other data or permissions associated with the user will be lost. The last good way to restore a user and all of their associated data may seem a bit backward, but it works and the user will back up and running with their data in no time. Before continuing make sure Directory Synchronization is up to date. You can force a sync with AAD Connect rather than waiting for the normal sync window to complete its next cycle. In the Office 365 portal (http://portal.office.com), expand the Users pane on the left. Then select the Deleted Users container and identify the user you would like to have restored. Select the object and hit restore on the right hand side of the screen (see image 1). This will restore the cloud user and the associated mailbox, SharePoint, and other service data. Image 1: Restoring a deleted office 365 user Ensure that the restored user has a license assigned. For details on how to license a user go here. Return to Users pane on the left once again. Then from the Active Users container, find the restored cloud user. Double-click the user to view the properties and change the UserPrincipalName namespace to the contoso.onmicrosoft.com address of your tenant and then save the changes (see image 2). Image 2: Modifying the UPN namespace Using the Azure AD PowerShell module, clear the immutableID of the restored Msoluser object by running the cmdlet below. We need to clear the immutableID to allow for a softmatch as the restored Msoluser has the immutableID of the deleted AD user. Set-Msoluser -UserPrincipalName user@contoso.onmicrosoft.com -ImmutableID “ “ For details on how to install and connect to the Azure AD PowerShell, go here. Next, create a new remote mailbox from either the Exchange Admin Center or Exchange Management Console on-premises (see images 3 and 4). It is important to ensure the SMTP address of the new remote mailbox is the same as the SMTP address of the user account that was restored. Meaning if the SMTP address of the user you are restoring is Ted@contoso.com, then the new remote mailbox you should create on-premises should have the same SMTP address. Following this step accurately will ensure a process called soft matching is performed. Soft matching links the new on-premises user account that was created behind the scenes as part of creating a new remote mailbox to the restored cloud mailbox based on the SMTP address. Image 3: Creating a new remote mailbox in Exchange 2013 Admin Center Image 4: Creating a new remote mailbox in Exchange 2010 Management Console Again, force a Directory Synchronization. To do that force a sync with AAD Connect. Then back in the portal expand the Users pane on the left again. Then from the Active Users container, find the restored cloud user and double-click on the user to view the properties. Check the UPN to see if it is the same as the newly created on-premises user. Restore a deleted user’s mailbox data to a new or alternate mailbox If none of the above recovery options are able to work for your situation, then you can still recover the mailbox data. While this process works and is a great way to recover mailbox data that would otherwise be lost, you still lose data associated with other services such and OneDrive and SharePoint. I would treat option as a last resort after all other options have failed. The steps outlined in this article will take you through a recovery process that involves creating a new user on-premises, synchronizing that user to Office 365, and merging the data from the soft deleted mailbox. Restore an inactive user mailbox The last scenario we are covering is the inactive mailbox scenario. For those that may not know, an Inactive Mailbox is a mailbox associate with a user that was placed on Litigation Hold then deleted. In order to preserve the data and keep it searchable we retain the mailbox contents and allow you to reuse the license that was previously assigned to the deleted user. More information on Inactive Mailboxes can be found here. If you accidentally deleted a user that was on Litigation Hold and you needed to restore the user, you can follow the steps below. Connect to the exchange online PowerShell using your tenant admin credentials, for details go here. Run the cmdlet: Get-Mailbox "<UPN of inactive mailbox>" -InactiveMailboxOnly | Select Name, DisplayName, MicrosoftOnlineServicesID, ExchangeGuid Run the cmdlet: New-Mailbox -Name "<Name from Step 2>" -InactiveMailbox " <ExchangeGuid from Step 2>" -MicrosoftOnlineServicesID "<MicrosoftOnlineServicesID from Step 2>" -Password (ConvertTo-SecureString -String 'Pa##w0rd goes here' -AsPlainText -Force) After the cmdlet in Step 3 completes successfully, wait at least 5 minutes for replication between the exchange online forest and the Azure AD forest. Once the Azure AD object for the new mailbox is visible, apply an exchange online license. Then create a new remote mailbox from either Exchange Admin Center or Exchange Management Console on-premises (see image 5 and 6). It is important to make sure that the SMTP address of the new remote mailbox is the same as the SMTP address of the user that was restored. Meaning if the SMTP address of the user you are restoring is Ted@contoso.com, then the new remote mailbox you should create on-premises should have the same SMTP address. This will ensure that we do a process called soft matching that links the on-premises user that was just created to the restored cloud mailbox based on the SMTP address. Image 5: Creating a new remote mailbox in Exchange 2013 Admin Center Image 6: Creating a new remote mailbox in Exchange 2010 Management Console Again, force a Directory Synchronization, to do that just force a sync with AAD Connect. Then back in the portal expand the Users pane on the left again. Then from the Active Users container, find the restored cloud user and double-click on the user to view the properties. Check the UPN to see if it is the same as the newly created on-premises user. In Summary It is best to set yourself and your organization up for the easiest possible mailbox and user recovery scenarios. When possible, try to do things like enabling the Active Directory Recycle Bin and educate all of your IT staff on the ramification of deleting users. Also know that in the end there are a lot of ways to recover a user and the associated data, make sure you use the option that fits your needs. I wanted to thank Timothy Heeney for a lot of help and discussion during the creation of this article. Bio AwojobiAsk the Perf Guy: Sizing Exchange 2016 Deployments
Uh oh. You are probably thinking to yourself, here comes another one of those incredibly long blog posts about sizing. Thankfully, it’s not. If you want to fully understand the sizing process for Exchange 2016, you are certainly welcome to read the previous post that I did for Exchange 2013, as the overall process is effectively the same with one major caveat. Since we have eliminated the CAS role in Exchange 2016, you must follow the process for multi-role deployment sizing. Overall, the inputs to our sizing formulas stay the same from Exchange 2013 to Exchange 2016. This means that our IOPS requirements, memory requirements, and all of the other values provided in the Exchange 2013 sizing guidance should continue to be used for Exchange 2016. We are changing one set of inputs, however. Processor requirements We are slightly increasing the processor requirements for Exchange 2016 (compared to Exchange 2013) as this is a very new release, and we are still learning how it performs in production. This slight increase in CPU provides some additional headroom for unanticipated issues, and may be changed in the future as we learn more from our internal deployments as well as customer feedback. The same SPECint_rate2006 baseline value described in the Exchange 2013 guidance should continue to be used (33.75 per-core). Messages sent or received per mailbox per day Mcycles per User, Active DB Copy or Standalone Mcycles per User, Passive DB Copy 50 2.99 0.70 100 5.97 1.40 150 8.96 2.10 200 11.94 2.80 250 14.93 3.50 300 17.91 4.20 350 20.90 4.90 400 23.88 5.60 450 26.87 6.30 500 29.85 7.00 These changes are reflected in v7.8 and later in the calculator. System scalability The previously released guidance on maximum recommended cores and maximum memory size for Exchange 2013 is generally applicable to Exchange 2016 in terms of background and general scalability issues, however we have increased the recommended maximum memory size for currently supported versions of Exchange 2016 to 192GB. We recommend not exceeding the following sizing characteristics for Exchange 2016 servers. Recommended Maximum Processor Core Count 24 Recommended Maximum Memory 192 GB Note: Version 9.1 and later of the Exchange Server Role Requirements Calculator aligns with this guidance. Summary If you are at all familiar with the process for sizing the last couple of Exchange releases, you will be very comfortable with Exchange 2016. As with any new release, you should plan to roll-out in stages and monitor the health and performance of the solution carefully. We do expect that our performance and scalability guidance for Exchange 2016 will evolve over the lifespan of the product so watch the Exchange team blog for future updates. Jeff Mealiffe Principal PM Manager Office 365 Customer ExperienceExchange 2013 Calculator Updates
Today, we released an updated version of the Exchange 2013 Server Role Requirements Calculator. In addition to numerous bug fixes, this version includes new functionality: CPU utilization table, ReplayLagManager support, MaximumPreferredActiveDatabases support, Restore-DatabaseAvailabilityGroup scenario support, and guidance on sizing recommendations. You can view what changes have been made or downloadthe update directly. For details on the new features, read on. CPU Utilization Table The Role Requirements tab includes a table that outlines the expected theoretical CPU utilization for various modes: Normal Run Time (where the active copies are distributed according to ActivationPreference=1) Single Server Failure (redistribution of active copies based on a single server failure event) Double Server Failure (redistribution of active copies based on a double server failure event) Site Failure (datacenter activation) Worst Failure Mode (in some cases, this value will equal one of the previous scenarios, it could also be a scenario like Site Failure + 1 server failure; the worst failure mode is what is used to calculate memory and CPU requirements) Here’s an example: In the above scenario, the worst failure mode is a site failure + 1 additional server failure (since this is a 4 database copy architecture). ReplayLagManager Support ReplayLagManager is a new feature in Exchange Server 2013 that automatically plays down the lagged database copy when availability is compromised. While it is disabled by default, we recommend it be enabled as part of the Preferred Architecture. Prior to version 7.5, the calculator only supported ReplayLagManagerin the scripts created via the Distribution tab (the Role Requirements and Activation Scenarios tabs did not support it). As a result, the calculator did not factor the lagged database copy as a viable activation target for the worst failure mode. Naturally, this is an issue because sizing is based on the number of active copies and the more copies activated on a server, the greater the impact to CPU and memory requirements. In a 4-copy 2+2 site resilient design, with the fourth copy being lagged, what this meant in terms of failure modes, is that the calculator sized the environment based on what it considered the worst case failure mode – Site Failure (2 HA copies lost, only a single HA copy remaining). Using the CPU table above as an example, calculator versions prior to 7.5 would base the design requirements on 18 active database copies (site failure) instead of 22 active database copies (3 copies lost, lagged copy played down and being utilized as the remaining active). ReplayLagManageris only supported (from the calculator perspective) when the design leverages: Multiple Databases / Volume 3+ HA copies MaximumPreferredActiveDatabases Support Exchange 2010 introduced the MaximumActiveDatabasesparameter which defines the maximum number of databases that are allowed to be activated on a server by BCS. It is this value that is used in sizing a Mailbox server (and is defined the worst failure mode in the calculator). Exchange 2013 introduced an additional parameter, MaximumPreferredActiveDatabases. This parameter specifies a preferred maximum number of databases that the Mailbox server should have. The value of MaximumPreferredActiveDatabasesis only honored during best copy and server selection (phases 1 through 4), database and server switchovers, and when rebalancing the DAG. With version 7.5 or later, the calculator recommends setting MaximumPreferredActiveDatabases when there are four or more total database copies. Also, the Export DAG List form exposes the MaximumPreferredActiveDatabasessetting and createdag.ps1 sets the value for the parameter. Restore-DatabaseAvailabilityGroup Scenario Support In prior releases, the Distribution tab only supported the concept of Fail WAN, which allowed you to simulate the effects of a WAN failure and model the surviving datacenter’s reaction depending on the location of the Witness server. However, Fail WAN did not attempt to shrink the quorum, so if you attempted to fail an additional server you would end up in this condition: With this version 7.5 and later, the calculator adds a new mode: Fail Site. When Fail Site is used, the datacenter switchover steps are performed (and thus the quorum is shrunk, alternate witness is utilized, if required, etc.) thereby allowing you to fail additional servers. This allows you to simulate the worst failure mode that is identified in the Role Requirements and Activation Scenarios tabs. Note: In order to recover from the Fail Site mode, you must click the Refresh Database Layout button. Sizing Guidance Recommendations As Jeff recently discussed in Ask The Perf Guy: How Big Is Too Big?, we are now providing explicit recommendations on the maximum number of processor cores and memory that should be deployed in each Exchange 2013 server. The calculator will now warn you if you attempt a design that exceeds these recommendations. As always, we welcome your feedback. Ross Smith IV Principal Program Manager Office 365 Customer Experience
