Common mailbox recovery scenarios for hybrid environments
Within the support organization at Microsoft we definitely see cases where customers are trying to recover deleted mailboxes. Typically, by the time a customer has contacted us they have tried everything they know as well as suggestions found online to recover the mailbox. It is often a completely avoidable and honest mistake that led to the deletion of the user’s Active Directory account in the first place. If you ever find yourself having a similarly bad day, then this article is meant to be your guide to not only getting through it, but to also come out of it as a superhero who is able to recover user accounts and mailboxes to a fully functional state without data loss. What you will learn from this article The main objective of this article is to help you recover a cloud mailbox after the corresponding on-premises user account has been deleted. If you do not have Directory Synchronization in place, then this article is not for you. With no Directory Synchronization in place you should view this article instead. Scenario that are covered There are many different mailbox recovery scenarios you may find yourself in. We will cover the most common scenarios throughout the course of this article to assist you in identifying the best recovery option for your own situation. Recover a mailbox that was deleted due to Directory Synchronization filtering changes resulting in filtering out the on-premises Active Directory user account associated with the cloud mailbox Recover a mailbox when the on-premises Active Directory user account associated with the cloud mailbox was accidentally or purposely deleted Recover a mailbox when the on-premises Active Directory user account associated with the cloud mailbox was accidentally or purposely deleted and the mailbox is on litigation hold All of the above scenarios have the same result. The associated user account in Office 365 becomes deleted due to one of the scenarios above and causes the mailbox to go into a soft-deleted state. Mailboxes in a soft-deleted state are recoverable for a period of 30 days before they are permanently removed from Office 365 and become unrecoverable. It is extremely important to attempt a proper user account recovery before blindly creating a new user account and merging the mailbox data. If you are able to restore the user account properly you will likely not lose any of the user data from the other services such as OneDrive and SharePoint. In addition, the user impact is pretty much non-existent when the original account is restored. There would be no need to create new profiles, no need to reset passwords, the user could simply log in and resume working from where they left off before. Recovery Process One of the challenges in restoring a mailbox is knowing which recovery option to use. If you know the recovery option you need, then can jump to it using the hyperlinks below. Otherwise we invite you to follow along with the article and we will guide you to the proper place. Restore a user account that was removed due to Directory Synchronization scope changes Restore a user account that was removed from on-premises AD with the recycle bin enabled Restore a user account that was removed from on-premises AD with no recycle bin enabled Restore a deleted user’s mailbox data to a new or alternate mailbox Restore an inactive user mailbox Restore a user that was removed due to Directory Synchronization Scope changes In some of the more complex customer environments it is sometimes beneficial to synchronize only certain Active Directory groups or Organizational Units (OUs) into Office 365. While this is not a common practice for most of our customers the process to configure filtering (if you are not familiar) is documented here. When configured, if a user is moved from an OU that is being synchronized to an OU that is not being synchronized, then Office 365 will see this action as a user account deletion. This causes the user account to be deleted within Office 365 and as a result the user’s mailbox also ends up in the previously mentioned soft-deleted state. The good news is the recovery for this scenario is quite simple. All you need to do is move the user back into the OU they were originally in. Assuming that the OU the user was previously in is still being synchronized, the next time Directory Synchronization complete the user and all associated data will be restored. By default, directory synchronizations occur every three hours and after you move the user back to the proper OU you will have to wait for the next sync cycle to take place. However, if you are like me and cannot wait, then you can force the synchronization. This article contains the necessary information to force a synchronization to take place immediately. Restore a user account that was removed from on-premises AD with the recycle bin enabled This scenario is a bit more common than the previous one. Many of our customers have realized the benefits that come with having the Active Directory recycle bin enabled. If you are not familiar with this feature and you are interested in learning more, then you can check it out here. The Active Directory recycle bin works as it sounds, when an object is deleted you can essentially undo the deletion without the kind of complex AD authoritative restoration process we all used and loved in the past. The good news is if you have the Active Directory recycle bin feature, it is a valid option to recover the deleted user. However, if you deleted the user prior to enabling this feature in your environment, then it will be of no help. Recovery steps: Follow this guidance to restore the user account using LDP or PowerShell Note: While less common, if you are using Directory Synchronization filtering (explained here), you need to be sure you restore the user to an OU that is within the Directory Synchronization scope. In an environment where more than one domain controller exists, ensure that the restored object has replicated before you proceed. Wait until your next Directory Synchronization cycle has complete or follow this article to force an immediate synchronization if you are using AAD Connect It is that easy and you will end up with the mailbox and all of the rest of the user data intact. Restore a user account that was removed from on-premises AD with no recycle bin enabled If you made it this far in the document, you likely are thinking “darn it I should have enabled the recycle bin for Active Directory”. While I agree with that sentiment, all is not lost. You can still recover your user account and mailbox data. In addition, you can still recover the data associated with other services, you just have a more difficult process to follow. The reason we try so har d to restore the original user account is so all of the data associated with the user is also restored. If you were to recreate a new user account on-premises (even with the same name as the deleted user), when the user syncs to Office 365 it will have a new object GUID. This means that any SharePoint, OneDrive, Exchange, and any other data or permissions associated with the user will be lost. The last good way to restore a user and all of their associated data may seem a bit backward, but it works and the user will back up and running with their data in no time. Before continuing make sure Directory Synchronization is up to date. You can force a sync with AAD Connect rather than waiting for the normal sync window to complete its next cycle. In the Office 365 portal (http://portal.office.com), expand the Users pane on the left. Then select the Deleted Users container and identify the user you would like to have restored. Select the object and hit restore on the right hand side of the screen (see image 1). This will restore the cloud user and the associated mailbox, SharePoint, and other service data. Image 1: Restoring a deleted office 365 user Ensure that the restored user has a license assigned. For details on how to license a user go here. Return to Users pane on the left once again. Then from the Active Users container, find the restored cloud user. Double-click the user to view the properties and change the UserPrincipalName namespace to the contoso.onmicrosoft.com address of your tenant and then save the changes (see image 2). Image 2: Modifying the UPN namespace Using the Azure AD PowerShell module, clear the immutableID of the restored Msoluser object by running the cmdlet below. We need to clear the immutableID to allow for a softmatch as the restored Msoluser has the immutableID of the deleted AD user. Set-Msoluser -UserPrincipalName user@contoso.onmicrosoft.com -ImmutableID “ “ For details on how to install and connect to the Azure AD PowerShell, go here. Next, create a new remote mailbox from either the Exchange Admin Center or Exchange Management Console on-premises (see images 3 and 4). It is important to ensure the SMTP address of the new remote mailbox is the same as the SMTP address of the user account that was restored. Meaning if the SMTP address of the user you are restoring is Ted@contoso.com, then the new remote mailbox you should create on-premises should have the same SMTP address. Following this step accurately will ensure a process called soft matching is performed. Soft matching links the new on-premises user account that was created behind the scenes as part of creating a new remote mailbox to the restored cloud mailbox based on the SMTP address. Image 3: Creating a new remote mailbox in Exchange 2013 Admin Center Image 4: Creating a new remote mailbox in Exchange 2010 Management Console Again, force a Directory Synchronization. To do that force a sync with AAD Connect. Then back in the portal expand the Users pane on the left again. Then from the Active Users container, find the restored cloud user and double-click on the user to view the properties. Check the UPN to see if it is the same as the newly created on-premises user. Restore a deleted user’s mailbox data to a new or alternate mailbox If none of the above recovery options are able to work for your situation, then you can still recover the mailbox data. While this process works and is a great way to recover mailbox data that would otherwise be lost, you still lose data associated with other services such and OneDrive and SharePoint. I would treat option as a last resort after all other options have failed. The steps outlined in this article will take you through a recovery process that involves creating a new user on-premises, synchronizing that user to Office 365, and merging the data from the soft deleted mailbox. Restore an inactive user mailbox The last scenario we are covering is the inactive mailbox scenario. For those that may not know, an Inactive Mailbox is a mailbox associate with a user that was placed on Litigation Hold then deleted. In order to preserve the data and keep it searchable we retain the mailbox contents and allow you to reuse the license that was previously assigned to the deleted user. More information on Inactive Mailboxes can be found here. If you accidentally deleted a user that was on Litigation Hold and you needed to restore the user, you can follow the steps below. Connect to the exchange online PowerShell using your tenant admin credentials, for details go here. Run the cmdlet: Get-Mailbox "<UPN of inactive mailbox>" -InactiveMailboxOnly | Select Name, DisplayName, MicrosoftOnlineServicesID, ExchangeGuid Run the cmdlet: New-Mailbox -Name "<Name from Step 2>" -InactiveMailbox " <ExchangeGuid from Step 2>" -MicrosoftOnlineServicesID "<MicrosoftOnlineServicesID from Step 2>" -Password (ConvertTo-SecureString -String 'Pa##w0rd goes here' -AsPlainText -Force) After the cmdlet in Step 3 completes successfully, wait at least 5 minutes for replication between the exchange online forest and the Azure AD forest. Once the Azure AD object for the new mailbox is visible, apply an exchange online license. Then create a new remote mailbox from either Exchange Admin Center or Exchange Management Console on-premises (see image 5 and 6). It is important to make sure that the SMTP address of the new remote mailbox is the same as the SMTP address of the user that was restored. Meaning if the SMTP address of the user you are restoring is Ted@contoso.com, then the new remote mailbox you should create on-premises should have the same SMTP address. This will ensure that we do a process called soft matching that links the on-premises user that was just created to the restored cloud mailbox based on the SMTP address. Image 5: Creating a new remote mailbox in Exchange 2013 Admin Center Image 6: Creating a new remote mailbox in Exchange 2010 Management Console Again, force a Directory Synchronization, to do that just force a sync with AAD Connect. Then back in the portal expand the Users pane on the left again. Then from the Active Users container, find the restored cloud user and double-click on the user to view the properties. Check the UPN to see if it is the same as the newly created on-premises user. In Summary It is best to set yourself and your organization up for the easiest possible mailbox and user recovery scenarios. When possible, try to do things like enabling the Active Directory Recycle Bin and educate all of your IT staff on the ramification of deleting users. Also know that in the end there are a lot of ways to recover a user and the associated data, make sure you use the option that fits your needs. I wanted to thank Timothy Heeney for a lot of help and discussion during the creation of this article. Bio AwojobiAsk the Perf Guy: Sizing Exchange 2016 Deployments
Uh oh. You are probably thinking to yourself, here comes another one of those incredibly long blog posts about sizing. Thankfully, it’s not. If you want to fully understand the sizing process for Exchange 2016, you are certainly welcome to read the previous post that I did for Exchange 2013, as the overall process is effectively the same with one major caveat. Since we have eliminated the CAS role in Exchange 2016, you must follow the process for multi-role deployment sizing. Overall, the inputs to our sizing formulas stay the same from Exchange 2013 to Exchange 2016. This means that our IOPS requirements, memory requirements, and all of the other values provided in the Exchange 2013 sizing guidance should continue to be used for Exchange 2016. We are changing one set of inputs, however. Processor requirements We are slightly increasing the processor requirements for Exchange 2016 (compared to Exchange 2013) as this is a very new release, and we are still learning how it performs in production. This slight increase in CPU provides some additional headroom for unanticipated issues, and may be changed in the future as we learn more from our internal deployments as well as customer feedback. The same SPECint_rate2006 baseline value described in the Exchange 2013 guidance should continue to be used (33.75 per-core). Messages sent or received per mailbox per day Mcycles per User, Active DB Copy or Standalone Mcycles per User, Passive DB Copy 50 2.99 0.70 100 5.97 1.40 150 8.96 2.10 200 11.94 2.80 250 14.93 3.50 300 17.91 4.20 350 20.90 4.90 400 23.88 5.60 450 26.87 6.30 500 29.85 7.00 These changes are reflected in v7.8 and later in the calculator. System scalability The previously released guidance on maximum recommended cores and maximum memory size for Exchange 2013 is generally applicable to Exchange 2016 in terms of background and general scalability issues, however we have increased the recommended maximum memory size for currently supported versions of Exchange 2016 to 192GB. We recommend not exceeding the following sizing characteristics for Exchange 2016 servers. Recommended Maximum Processor Core Count 24 Recommended Maximum Memory 192 GB Note: Version 9.1 and later of the Exchange Server Role Requirements Calculator aligns with this guidance. Summary If you are at all familiar with the process for sizing the last couple of Exchange releases, you will be very comfortable with Exchange 2016. As with any new release, you should plan to roll-out in stages and monitor the health and performance of the solution carefully. We do expect that our performance and scalability guidance for Exchange 2016 will evolve over the lifespan of the product so watch the Exchange team blog for future updates. Jeff Mealiffe Principal PM Manager Office 365 Customer ExperienceExchange Server Role Requirements Calculator Update
v7.8 of the calculator introduces support for Exchange 2016! Yes, that’s right, you don’t need a separate calculator, v7.8 and later supports Exchange 2013 or Exchange 2016 deployments. Moving forward, the calculator is branded as the Exchange Server Role Requirements Calculator. When you open the calculator you will find a new drop-down option in the Input tab that allows you to select the deployment version. Simply choose 2013 or 2016: When you choose 2016, you will notice the Server Multi-Role Configuration option is disabled due to the fact that Exchange 2016 no longer provides the Client Access Server role. As discussed in the Exchange 2016 Architecture and Preferred Architecture articles, the volume format best practice recommendation for Exchange data volumes has changed in Exchange 2016 as we now recommend ReFS (with the integrity feature disabled). By default, for Exchange 2016 deployments, the calculator scripts will default to ReFS (Exchange 2013 deployments will default to NTFS). This is exposed in the Export Mount Points File dialog: The DiskPart.ps1 and CreateDag.ps1 scripts have been updated to support formatting the volume as ReFS (and disabling the integrity feature at the volume level) and enabling AutoReseed support for ReFS. This release also improves the inputs of all dialogs for the distribution scripts by persisting values across the various dialogs (e.g., global catalog values). For all the other improvements and bug fixes, please review the readme or download the update. As always we welcome feedback and please report any issues you may encounter while using the calculator by emailing strgcalc AT microsoft DOT com. Ross Smith IV Principal Program Manager Office 365 Customer ExperienceAsk The Perf Guy: What’s The Story With Hyperthreading and Virtualization?
There’s been a fair amount of confusion amongst customers and partners lately about the right way to think about hyperthreading when virtualizing Exchange. Hopefully I can clear up that confusion very quickly. We’ve had relatively strong guidance in recent versions of Exchange that hyperthreading should be disabled. This guidance is specific to physical server deployments, not virtualized deployments. The reasoning for strongly recommending that hyperthreading be disabled on physical deployments can really be summarized in 2 different points: The increase in logical processor count at the OS level due to enabling hyperthreading results in increased memory consumption (due to various algorithms that allocate memory heaps based on core count), and in some cases also results in increased CPU consumption or other scalability issues due to high thread counts and lock contention. The increased CPU throughput associated with hyperthreading is non-deterministic and difficult to measure, leading to capacity planning challenges. The first point is really the largest concern, and in a virtual deployment, it is a non-issue with regard to configuration of hyperthreading. The guest VMs do not see the logical processors presented to the host, so they see no difference in processor count when hyperthreading is turned on or off. Where this concern can become an issue for guest VMs is in the number of virtual CPUs presented to the VM. Don’t allocate more virtual CPUs to your Exchange server VMs that are necessary based on sizing calculations. If you allocate extra virtual CPUs, you can run into the same class of issues associated with hyperthreading on physical deployments. In summary: If you have a physical deployment, turn off hyperthreading. If you have a virtual deployment, you can enable hyperthreading (best to follow the recommendation of your hypervisor vendor), and: Don’t allocate extra virtual CPUs to Exchange server guest VMs. Don’t use the extra logical CPUs exposed to the host for sizing/capacity calculations (see the hyperthreading guidance at https://aka.ms/e2013sizing for further details on this). Jeff Mealiffe Principal PM Manager Office 365 Customer ExperienceAsk the Perf Guy: How big is too BIG?
We’ve seen an increasing amount of interest lately in deployment of Exchange 2013 on “large” servers. By large, I mean servers that contain significantly more CPU or memory resources than what the product was designed to utilize. I thought it might be time for a reminder of our scalability recommendations and some of the details behind those recommendations. Note that this guidance is specific to Exchange 2013 – there are many architectural differences in prior releases of the product that will impact scalability guidance. In a nutshell, we recommend not exceeding the following sizing characteristics for Exchange 2013 servers, whether single-role or multi-role (and you are running multi-role, right?). Recommended Maximum Processor Core Count 24 Recommended Maximum Memory 96 GB Note: Version 7.5 and later of the Exchange Server 2013 Role Requirements Calculator aligns with this guidance and will flag server configurations that exceed these guidelines. As we have mentioned in various places like TechNet and our Preferred Architecture, commodity-class 2U servers with 2 processor sockets are our recommended server type for deployment of Exchange 2013. The reason for this is quite simple: we utilize massive quantities of these servers for deployment in Exchange Online, and as a result this is the platform that we architect for and have the best visibility into when evaluating performance and scalability. You might now be asking the fairly obvious follow up question: what happens if I ignore this recommendation and scale up? It’s hard, if not impossible, to provide a great answer to this question, because there are so many things that could go wrong. We have certainly seen a number of issues raised through support related to scale-up deployments of Exchange in recent months. An example of this class of issue appears in the “Oversizing” section of Marc Nivens’ recent blog article on troubleshooting high CPU issues in Exchange 2013. Many of the issues we see are in some way related to concurrency and reduced throughput due to excessive contention amongst threads. This essentially means that the server is trying to do so much work (believing that it has the capability to do so given the massive amount of hardware available to it) that it is running into architectural bottlenecks and actually spending a great deal of time dealing with locks and thread scheduling instead of handling transactions associated with Exchange workloads. Because we architect and tune the product for mid-range server hardware as described above, no tuning has been done to get the most out of this larger hardware and avoid this class of issues. We have also seen some cases in which the patterns of requests being serviced by Exchange, the number of CPU cores, and the amount of physical memory deployed on the server resulted in far more time being spent in the .NET Garbage Collection process than we would expect, given our production observations and tuning of memory allocation patterns within Exchange code. In some of these cases, Microsoft support engineers may determine that the best short-term workaround is to switch one or more Exchange services from the Workstation Garbage Collection mode to Server Garbage Collection mode. This allows the .NET Garbage Collector to manage memory more efficiently but with some significant tradeoffs, like a dramatic increase in physical memory consumption. In general, each individual service that makes up the Exchange server product has been tuned as carefully as possible to be a good consumer of memory resources, and wherever possible, we utilize the Workstation Garbage Collector to avoid a dramatic and typically unnecessary increase in memory consumption. While it’s possible that adjusting a service to use Server GC rather than Workstation GC might temporarily mitigate an issue, it’s not a long-term fix that the product group recommends. When it comes to .NET Garbage Collector settings, our advice is to ensure that you are running with default settings and the only time these settings should be adjusted is with the advice and consent of Microsoft Support. As we make changes to Exchange through our normal servicing rhythm, we may change these defaults to ensure that Exchange continues to perform as efficiently as possible, and as a result, manual overrides could result in a less optimal configuration. As server and processor technology changes, you can expect that we will make adjustments to our production deployments in Exchange Online to ensure that we are getting the highest performance possible at the lowest cost for the users of our service. As a result, we anticipate updating our scalability guidance based on our experience running Exchange on these updated hardware configurations. We don’t expect these updates to be very frequent, but change to hardware configurations is absolutely a given when running a rapidly growing service. It’s a fact that many of you have various constraints on the hardware that you can deploy in your datacenters, and often those constraints are driven by a desire to reduce server count, increase server density, etc. Within those constraints, it can be very challenging to design an Exchange implementation that follows our scalability guidance and the Preferred Architecture. Keep in mind that in this case, virtualization may be a feasible option rather than a risky attempt to circumvent scalability guidance and operate extremely large Exchange servers. Virtualization of Exchange is a well understood, fairly common solution to this problem, and while it does add complexity (and therefore some additional cost and risk) to your deployment, it can also allow you to take advantage of large hardware while ensuring that Exchange gets the resources it needs to operate as effectively as possible. If you do decide to virtualize Exchange, remember to follow our sizing guidance within the Exchange virtual machines. Scale out rather than scale up (the virtual core count and memory size should not exceed the guidelines mentioned above) and try to align as closely as possible to the Preferred Architecture. When evaluating these scalability limits, it’s really most important to remember that Exchange high availability comes from staying as close to the product group’s guidance and Preferred Architecture as possible. We want you to have the very best possible experience with Exchange, and we know that the best way to achieve that is to deploy like we do. Jeff Mealiffe Principal PM Manager Office 365 Customer ExperienceExchange 2013 Calculator Updates
Today, we released an updated version of the Exchange 2013 Server Role Requirements Calculator. In addition to numerous bug fixes, this version includes new functionality: CPU utilization table, ReplayLagManager support, MaximumPreferredActiveDatabases support, Restore-DatabaseAvailabilityGroup scenario support, and guidance on sizing recommendations. You can view what changes have been made or downloadthe update directly. For details on the new features, read on. CPU Utilization Table The Role Requirements tab includes a table that outlines the expected theoretical CPU utilization for various modes: Normal Run Time (where the active copies are distributed according to ActivationPreference=1) Single Server Failure (redistribution of active copies based on a single server failure event) Double Server Failure (redistribution of active copies based on a double server failure event) Site Failure (datacenter activation) Worst Failure Mode (in some cases, this value will equal one of the previous scenarios, it could also be a scenario like Site Failure + 1 server failure; the worst failure mode is what is used to calculate memory and CPU requirements) Here’s an example: In the above scenario, the worst failure mode is a site failure + 1 additional server failure (since this is a 4 database copy architecture). ReplayLagManager Support ReplayLagManager is a new feature in Exchange Server 2013 that automatically plays down the lagged database copy when availability is compromised. While it is disabled by default, we recommend it be enabled as part of the Preferred Architecture. Prior to version 7.5, the calculator only supported ReplayLagManagerin the scripts created via the Distribution tab (the Role Requirements and Activation Scenarios tabs did not support it). As a result, the calculator did not factor the lagged database copy as a viable activation target for the worst failure mode. Naturally, this is an issue because sizing is based on the number of active copies and the more copies activated on a server, the greater the impact to CPU and memory requirements. In a 4-copy 2+2 site resilient design, with the fourth copy being lagged, what this meant in terms of failure modes, is that the calculator sized the environment based on what it considered the worst case failure mode – Site Failure (2 HA copies lost, only a single HA copy remaining). Using the CPU table above as an example, calculator versions prior to 7.5 would base the design requirements on 18 active database copies (site failure) instead of 22 active database copies (3 copies lost, lagged copy played down and being utilized as the remaining active). ReplayLagManageris only supported (from the calculator perspective) when the design leverages: Multiple Databases / Volume 3+ HA copies MaximumPreferredActiveDatabases Support Exchange 2010 introduced the MaximumActiveDatabasesparameter which defines the maximum number of databases that are allowed to be activated on a server by BCS. It is this value that is used in sizing a Mailbox server (and is defined the worst failure mode in the calculator). Exchange 2013 introduced an additional parameter, MaximumPreferredActiveDatabases. This parameter specifies a preferred maximum number of databases that the Mailbox server should have. The value of MaximumPreferredActiveDatabasesis only honored during best copy and server selection (phases 1 through 4), database and server switchovers, and when rebalancing the DAG. With version 7.5 or later, the calculator recommends setting MaximumPreferredActiveDatabases when there are four or more total database copies. Also, the Export DAG List form exposes the MaximumPreferredActiveDatabasessetting and createdag.ps1 sets the value for the parameter. Restore-DatabaseAvailabilityGroup Scenario Support In prior releases, the Distribution tab only supported the concept of Fail WAN, which allowed you to simulate the effects of a WAN failure and model the surviving datacenter’s reaction depending on the location of the Witness server. However, Fail WAN did not attempt to shrink the quorum, so if you attempted to fail an additional server you would end up in this condition: With this version 7.5 and later, the calculator adds a new mode: Fail Site. When Fail Site is used, the datacenter switchover steps are performed (and thus the quorum is shrunk, alternate witness is utilized, if required, etc.) thereby allowing you to fail additional servers. This allows you to simulate the worst failure mode that is identified in the Role Requirements and Activation Scenarios tabs. Note: In order to recover from the Fail Site mode, you must click the Refresh Database Layout button. Sizing Guidance Recommendations As Jeff recently discussed in Ask The Perf Guy: How Big Is Too Big?, we are now providing explicit recommendations on the maximum number of processor cores and memory that should be deployed in each Exchange 2013 server. The calculator will now warn you if you attempt a design that exceeds these recommendations. As always, we welcome your feedback. Ross Smith IV Principal Program Manager Office 365 Customer Experience
