The case of Reply Log Manager not letting lagged copy lag
In a previous blog post Ross Smith IV had explained what the Replay Lag Manager is and what it does. It's a great feature that's somewhat underappreciated. We've seen a few support cases that seemed to have been opened out of the misunderstanding of what the Replay Lag Manager is doing. I wanted to cover a real world scenario I had dealt with recently with a customer that I believe will clarify some things. What is a Replay Lag Manager? In a nutshell, Replay Lag Manager provides higher availability for Exchange through the automatic invocation of a lagged database copy. To further explain, a lagged database copy is a database that Exchange delays committing changes to for a specified period of time. The Replay Lag Manager was first introduced in Exchange 2013 and is actually enabled by default beginning with Exchange 2016 CU1. To understand what it is let's look at the Preferred Architecture (PA) in regards to a database layout. The PA uses 4 database copies like the following: As you can see the 4th copy is a lagged copy. Even though we're showing it in a secondary site, it can exist in any site where a node in the same DAG resides. The Replay Lag Manager will constantly watch for any of the three things to happen to the copies of DB1. Ross Smith's post does a wonderful job of explaining them and how Exchange will take other factors (i.e. disk IO) into consideration before invoking the lagged copy. In general, a log play down will occur: When a low disk space threshold (10,000MB) is reached When the lagged DB copy has physical corruption and needs to be page patched When there are fewer than three available healthy HA copies for more than 24 hours A log "play down" essentially means that Replay Lag Manager is going to force that lagged database copy to catch up on all of the changes to make that copy current. By doing this it ensures that Exchange maintains at least 3 copies of each database. When things are less than perfect… In the real world we don't always see Exchange setup according to our Preferred Architecture because of environment constraints or business requirements. There was a recent case that was the best example of Lag Replay Manager working in the real world. The customer had over 100 DB's, all with 6 copies each. There were 3 copies in the main site and 3 copies in the Disaster Recovery site with one of those copies at each site being lagged. The DB copies were configured like this for all databases. As you can see in this particular instance the lagged copy at Site A was being forced to play down while the other copy showed a Replay Queue Length (RQL) of 4919. This case was opened due to the fact that the lagged DB copy at Site A was not lagging. The customer stated that the DB was lagging fine until recently. However, after a quick check of the Replay Queue Length counter in the Daily Performance Logs it didn't appear to have ever lagged successfully for this copy. So, what we're seeing is the database has 6 copies, 2 lagged but 1 of those lagged copies isn't lagging. Naturally, you may try removing the lag by setting the -ReplayLagTime to 0 then changing back to 7 (or what it was before). You may even try recreating the database copy thinking something was wrong with it. These still don't cause Exchange to lag this copy. The next step is to check if it's actually the Replay Lag Manager causing the log play down. You can quickly see this by running the following command specifying the lagged DB\Server Name. On this example will use SERVER3 as the server hosting the lagged copy of DB1. Get-MailboxDatabaseCopyStatus DB1\SERVER3 | Select Id,ReplayLagStatus Id : DB1\SERVER3 ReplayLagStatus : Enabled:False; PlayDownReason:LagDisabled; ReplaySuspendReason:None; Percentage:0; Configured:7.00:00:00; MaxDelay:1.00:00:00; Actual:00:01:22 What we see is that the ReplayLagStatus is actually disabled and the PlayDownReason is LagDisabled. That tells us it's disabled but it doesn't really give us more detail as to why.. We can dig further by looking at the Microsoft-Exchange/HighAvailability log and we see a pattern of 3 events. The first event we encounter is the 708 but it doesn't give us any more information than the previous command does. Time: 11/31/2017 3:32:55 PM ID: 708 Level: Information Source: Microsoft-Exchange-HighAvailability Machine: server3.domain.com Message: Log Replay for database 'DB1' is replaying logs in the replay lag range. Reason: Replay lag has been disabled. (LogFileAge=00:06:00.8929066, ReasonCode=LagDisabled) The second event we see has a little more information. At this point we know for sure it's the Replay Lag Manger because of its FastLagPlaydownDesired status. Time: 11/31/2017 3:32:55 PM ID: 2001 Level: Warning Source: Microsoft-Exchange-HighAvailability Machine: server3.domain.com Message: Database scanning during passive replay is disabled on 'DB1'. Explanation: FastLagPlaydownDesired. On the third event we see the 738 which actually explains what's going on here. Time: 11/30/2017 1:50:15 PM ID: 738 Level: Information Source: Microsoft-Exchange-HighAvailability Machine: server3.domain.com Message: Replay Lag Manager suppressed a request to disable replay lag for database copy 'DB1\SERVER3' after a suppression interval of 1.00:00:00. Disable Reason: There were database availability check failures for database 'DB1' that may be lowering its availability. Availability Count: 3. Expected Availability Count: 3. Detailed error(s): SERVER4: Server 'server4.domain.com' has database copy auto activation policy configuration of 'Blocked'. SERVER5: Server 'server5.domain.com' has database copy auto activation policy configuration of 'Blocked'. SERVER6: Server 'server6.domain.com' has database copy auto activation policy configuration of 'Blocked'. The "Availability Count: 3. Expected Availability Count: 3." is a tad confusing but the heart the issue is in the detailed errors below that… It's Replay Lag Manager doing it… but why? The entire reason for this blog post comes out of the fact that we've seen the Replay Lag Manager blamed for not letting a lagged copy lag. So, the next step someone will do is to disable it. Please don't do that! It only wants to help! Let's look at how we can resolve the our above example. The logs are showing that it's expecting 3 copies but there aren't 3 available. How can that be? They have at least 4 copies of this database available?!? If we run the following command we see a hint at culprit. Get-mailboxdatabasecopystatus DB1 | Select Identity,AutoActivationPolicy Identity AutoActivationPolicy -------- -------------------- DB1\SERVER1 Unrestricted DB1\SERVER2 Unrestricted DB1\SERVER3 Unrestricted - Lagged Copy (Not lagging) DB1\SERVER4 Blocked DB1\SERVER5 Blocked DB1\SERVER6 Blocked - Lagged Copy (Working) There it is! There are 6 database copies, however, the copies in Site B are all blocked due to the AutoActivationPolicy. Now things are starting to make sense. In the eyes of the Replay Lag Manager those copies in Site B are not available because Exchange cannot activate them automatically. So, what's happening is the Replay Lag Manager only sees the 2 copies (in the green square below) as available. Therefore, it forces a play down of the logs on the lagged copy to maintain it's 3 available copies. That explains why the lagged copy at Site A isn't lagging but why is the lagged copy at Site B working fine? This is because from the perspective of that database there are 3 available copies in Site A once that lagged copy was played down. That's cool… how do I fix it? There are essentially two ways to resolve this example and allow that lagged copy at Site A to properly lag. The first way is to revisit the decision to block Auto Activation at Site B. The mindset in this particular instance was that their other site was actually for Disaster Recovery. They wanted some manual intervention if databases needed to fail over to the DR site. That's all well and good but it doesn't allow for a lagged copy at Site A to work properly due to the Replay Lag Manager. The customer did actually end up allowing 1 copy at the DR site (site B in our example) for Auto Activation. To do this you can run the following command: Set-Mailboxdatabasecopy SERVER4\DB1 -DatabaseCopyAutoActivationPolicy Unrestricted The other option here would be to create another database copy at Site A. Obviously, that's going to require a lot more effort and storage. However, doing this would allow for the Replay Lag Manager to resume lagging on the lagged database copy. I hope this post clarifies some things in regards to the Replay Lag Manager. It's a great feature that will provide some automation in keeping your Exchange databases highly available. Michael SchatteReleased: Exchange Server Role Requirements Calculator 8.3
Today, we released an updated version of the Exchange Server Role Requirements Calculator. This release focuses around two specific enhancements. Exchange 2016 designs now take into account the CU3 improvement that reduces the bandwidth required between active and passive HA copies as the local search instance can read data from its local database copy. The calculator now supports the ability to automatically calculate the number of DAGs and the corresponding number of Mailbox servers that should be deployed to support the defined requirements. This process takes into account memory, CPU cores, and disk configuration when determining the optimal configuration, ensuring that recommended thresholds are not exceeded. As a result of this change, you will find that the Input tab has been rearranged. Specifically, the DAG variables have been moved to the end of the worksheet to ensure that you have completely entered all information before attempting an automatic calculation. As with everything else in the calculator, you can turn the automatic calculation off and manually select the number of Mailbox servers and DAGs you would like to deploy. For all the other improvements and bug fixes, please review the readme or download the update. As always we welcome feedback and please report any issues you may encounter while using the calculator by emailing strgcalc AT microsoft DOT com. Ross Smith IV Principal Program Manager Office 365 Customer ExperienceDAG Activation Preference Behavior Change in Exchange Server 2016 CU2
Every copy of a mailbox database in a DAG is assigned an activation preference number. This number is used by the system as part of the passive database activation process, and by administrators when performing database balancing operations for a DAG. This number is expressed as the ActivationPreference property of a mailbox database copy. The value for the ActivationPreference property is a number equal to or greater than 1, where 1 is at the top of the preference order. When a DAG is first implemented, by default all active database copies have an ActivationPreference of 1. However, due to the inherent nature of DAGs (e.g., databases experience switchovers and failovers), active mailbox database copies will change hosts several times throughout a DAG's lifetime. As a result of this inherent behavior, a mailbox database may remain active on a database copy which is the not the most preferred copy. Prior to Exchange 2016 Cumulative Update 2 (CU2), Exchange Server administrators had to either manually activate their preferred database copy, or use the RedistributeActiveDatabases.ps1 script to balance the databases copies across a DAG. Starting with CU2 (which will be releasing soon), the Primary Active Manager in the DAG performs periodic discretionary moves to activate the copy that the administrator has defined as most preferred is now built into the product. A new DAG property called PreferenceMoveFrequency has been added that defines the frequency (measured in time) when the Microsoft Exchange Replication service will rebalance the database copies by performing a lossless switchover that activates the copy with an ActivationPreference of 1 (assuming the target server and database copy are healthy). Note: In order to take advantage of this feature, ensure all Mailbox servers within the DAG are upgraded to Exchange 2016 CU2. By default, the Replication service will inspect the database copies and perform a rebalance every one hour. You can modify this behavior using the following command: Set-DatabaseAvailabilityGroup <Name> -PreferenceMoveFrequency <value in the format of 00:00:00> To disable this behavior, configure the PreferenceMoveFrequency value to ([System.Threading.Timeout]::InfiniteTimeSpan). If you are leaving the behavior enabled, and you have created a scheduled task to execute RedistributeActiveDatabases.ps1, you can remove the scheduled task after upgrading the DAG to CU2. We recommend taking advantage of this behavior to ensure that your DAG remains optimally balanced. This feature continues our work to improve the Preferred Architecture by ensuring that users have the best possible experience on Exchange Server. As always, we welcome your feedback. Ross Smith IV Principal Program Manager Office 365 Customer Experience Updates 6/21/16: Updated information on how to disable PreferenceMoveFrequency without requiring a Replication service restart. If you set it to [Timespan]::Zero, you will need to cycle the Replication service.Lagged Database Copy Enhancements in Exchange Server 2016 CU1
The high availability capabilities of the lagged database copy are enhanced in the upcoming release of Exchange 2016 Cumulative Update 1. ReplayLagManager As you may recall, lagged copies can care for themselves by invoking automatic log replay to play down the log files in certain scenarios: When a low disk space threshold (10,000MB) is reached When the lagged copy has physical corruption and needs to be page patched When there are fewer than three available healthy HA copies for more than 24 hours Play down based on health copy status requires ReplayLagManager to be enabled. Beginning with Exchange 2016 CU1, ReplayLagManager is enabled by default. You can change this via the following command: Set-DatabaseAvailabilityGroup <DAGName> -ReplayLagManagerEnabled $false Deferred Lagged Copy Play Down When one of the above conditions is triggered, the Replication Service will initiate a play down event for the lagged database copy. However, there are times where this may not be ideal. For example, consider the scenario where there are four database copies on a disk, one passive, one lagged, and two active. Initiating a play down event on the lagged copy has the potential to impact any active copies on that disk – replaying log files generates IO and introduces disk latency as the disk head moves, which impacts users accessing their data on the active copies. To address this concern, beginning with Cumulative Update 1 for Exchange 2016, the lagged copy’s play down activity is tied to the health of the disk by evaluating the disk’s IO latency: If the disk’s read IO latency is above 35ms, the play down event is deferred. In the event that there is a disk capacity concern, the disk latency deferral will be ignored and the lagged copy will play down. Once the disk’s read IO latency drops below 25ms, the play down event is resumed. As a result, deferred lagged copy play down reduces the IO burstiness of lagged copy play down events and ensures that local active copies on the lagged copies disk are not affected. IO sizing of a lagged database copy does not change with this feature (nor does it affect the IO sizing of an active copy); you still must ensure there is available IO headroom in the event that the lagged copy becomes active. Consider the following example: The y axis is disk latency, measured in milliseconds. The x axis is a 24-hour period. As you can see from the graph, between the hours of 1am to 9am, the disk IO latency is below 25ms, meaning that lagged copy replay is allowed. At 10am, the latency exceeds 35ms and this continues until about 2pm; during this time period, lagged copy replay is delayed or deferred. At 2pm, the latency drops below 25ms and lagged copy replay resumes. Latency increases again at 4pm and the process repeats itself. By default, the maximum amount of time that a play down event can be deferred is 24 hours. You can adjust this via the following command: Set-MailboxDatabaseCopy <database name\server> -ReplayLagMaxDelay:<value in the format of 00:00:00> If you want to disable deferred play down, you can set the ReplayLagMaxDelay value to ([TimeSpan]::Zero). The following events are recorded in the Microsoft-Exchange-HighAvailability/Monitoring crimson channel when log replay is deferred or resumed: Event 750 – Replay Lag Manager requested activating replay lag delay (suspending log replay) for database copy '%1\%2' after a suppression interval of %4. Delay Reason: %6" Event 751 – Replay Lag Manager successfully activated replay lag delay (suspended log replay) for database copy '%1\%2'. Delay Reason: %4" Event 752 – Replay Lag Manager failed to activate replay lag delay (suspend log replay) for database copy '%1\%2'. Error: %4" Event 753 – Replay Lag Manager requested deactivating replay lag (resuming log replay) for database copy '%1\%2' after a suppression interval of %4. Reason: %5" Event 754 – Replay Lag Manager successfully deactivated replay lag (resumed log replay) for database copy '%1\%2'. Reason: %4 Event 755 - Replay Lag Manager failed to deactivate replay lag (resume log replay) for database copy '%1\%2'. Error: %4 Event 756 - Replay Lag Manager will attempt to deactivate replay lag (resume log replay) for database copy '%1\%2' because it has reached the maximum allowed lag duration. Detailed Reason: %5 The following events are recorded in the Microsoft-Exchange-HighAvailability/Operational crimson channel when log replay is deferred or resumed: Event 748 – Log Replay suspend/resume state for database '%1' has changed. (LastSuspendReason=%3, CurrentSuspendReason=%4, CurrentSuspendReasonMessage=%5) Event 2050 – Suspend log replay requested for database guid=%1, reason='%2'. Event 2051 – Suspend log replay for database guid=%1 succeeded. Event 2052 – Suspend log replay for database guid=%1 failed: %2. Event 2053 – Resume log replay requested for database guid=%1. Event 2054 – Resume log replay for database guid=%1 succeeded. Event 2055 – Resume log replay for database guid=%1 failed: %2. Summary The changes discussed above continue our work in improving the Preferred Architecture by ensuring that users have the best possible experience on the Exchange platform. As always, we welcome your feedback. Ross Smith IV Principal Program Manager Office 365 Customer ExperienceExchange Server Role Requirements Calculator Update
v7.8 of the calculator introduces support for Exchange 2016! Yes, that’s right, you don’t need a separate calculator, v7.8 and later supports Exchange 2013 or Exchange 2016 deployments. Moving forward, the calculator is branded as the Exchange Server Role Requirements Calculator. When you open the calculator you will find a new drop-down option in the Input tab that allows you to select the deployment version. Simply choose 2013 or 2016: When you choose 2016, you will notice the Server Multi-Role Configuration option is disabled due to the fact that Exchange 2016 no longer provides the Client Access Server role. As discussed in the Exchange 2016 Architecture and Preferred Architecture articles, the volume format best practice recommendation for Exchange data volumes has changed in Exchange 2016 as we now recommend ReFS (with the integrity feature disabled). By default, for Exchange 2016 deployments, the calculator scripts will default to ReFS (Exchange 2013 deployments will default to NTFS). This is exposed in the Export Mount Points File dialog: The DiskPart.ps1 and CreateDag.ps1 scripts have been updated to support formatting the volume as ReFS (and disabling the integrity feature at the volume level) and enabling AutoReseed support for ReFS. This release also improves the inputs of all dialogs for the distribution scripts by persisting values across the various dialogs (e.g., global catalog values). For all the other improvements and bug fixes, please review the readme or download the update. As always we welcome feedback and please report any issues you may encounter while using the calculator by emailing strgcalc AT microsoft DOT com. Ross Smith IV Principal Program Manager Office 365 Customer ExperienceThe Exchange 2016 Preferred Architecture
The Preferred Architecture (PA) is the Exchange Engineering Team’s best practice recommendation for what we believe is the optimum deployment architecture for Exchange 2016, and one that is very similar to what we deploy in Office 365. While Exchange 2016 offers a wide variety of architectural choices for on-premises deployments, the architecture discussed below is our most scrutinized one ever. While there are other supported deployment architectures, they are not recommended. The PA is designed with several business requirements in mind, such as the requirement that the architecture be able to: Include both high availability within the datacenter, and site resilience between datacenters Support multiple copies of each database, thereby allowing for quick activation Reduce the cost of the messaging infrastructure Increase availability by optimizing around failure domains and reducing complexity The specific prescriptive nature of the PA means of course that not every customer will be able to deploy it (for example, customers without multiple datacenters). And some of our customers have different business requirements or other needs which necessitate a different architecture. If you fall into those categories, and you want to deploy Exchange on-premises, there are still advantages to adhering as closely as possible to the PA, and deviate only where your requirements widely differ. Alternatively, you can consider Office 365 where you can take advantage of the PA without having to deploy or manage servers. The PA removes complexity and redundancy where necessary to drive the architecture to a predictable recovery model: when a failure occurs, another copy of the affected database is activated. The PA is divided into four areas of focus: Namespace design Datacenter design Server design DAG design Namespace Design In the Namespace Planning and Load Balancing Principles articles, I outlined the various configuration choices that are available with Exchange 2016. For the namespace, the choices are to either deploy a bound namespace (having a preference for the users to operate out of a specific datacenter) or an unbound namespace (having the users connect to any datacenter without preference). The recommended approach is to utilize the unbounded model, deploying a single Exchange namespace per client protocol for the site resilient datacenter pair (where each datacenter is assumed to represent its own Active Directory site - see more details on that below). For example: autodiscover.contoso.com For HTTP clients: mail.contoso.com For IMAP clients: imap.contoso.com For SMTP clients: smtp.contoso.com Each Exchange namespace is load balanced across both datacenters in a layer 7 configuration that does not leverage session affinity, resulting in fifty percent of traffic being proxied between datacenters. Traffic is equally distributed across the datacenters in the site resilient pair, via round robin DNS, geo-DNS, or other similar solutions. From our perspective, the simpler solution is the least complex and easier to manage, so our recommendation is to leverage round robin DNS. For the Office Online Server farm, a namespace is deployed per datacenter, with the load balancer utilizing layer 7, maintaining session affinity using cookie based persistence. Figure 1: Namespace Design in the Preferred Architecture In the event that you have multiple site resilient datacenter pairs in your environment, you will need to decide if you want to have a single worldwide namespace, or if you want to control the traffic to each specific datacenter by using regional namespaces. Ultimately your decision depends on your network topology and the associated cost with using an unbound model; for example, if you have datacenters located in North America and Europe, the network link between these regions might not only be costly, but it might also have high latency, which can introduce user pain and operational issues. In that case, it makes sense to deploy a bound model with a separate namespace for each region. However, options like geographical DNS offer you the ability to deploy a single unified namespace, even when you have costly network links; geo-DNS allows you to have your users directed to the closest datacenter based on their client’s IP address. Figure 2: Geo-distributed Unbound Namespace Site Resilient Datacenter Pair Design To achieve a highly available and site resilient architecture, you must have two or more datacenters that are well-connected (ideally, you want a low round-trip network latency, otherwise replication and the client experience are adversely affected). In addition, the datacenters should be connected via redundant network paths supplied by different operating carriers. While we support stretching an Active Directory site across multiple datacenters, for the PA we recommend that each datacenter be its own Active Directory site. There are two reasons: Transport site resilience via Shadow Redundancy and Safety Net can only be achieved when the DAG has members located in more than one Active Directory site. Active Directory has published guidance that states that subnets should be placed in different Active Directory sites when the round trip latency is greater than 10ms between the subnets. Server Design In the PA, all servers are physical servers. Physical hardware is deployed rather than virtualized hardware for two reasons: The servers are scaled to use 80% of resources during the worst-failure mode. Virtualization adds an additional layer of management and complexity, which introduces additional recovery modes that do not add value, particularly since Exchange provides that functionality. Commodity server platforms are used in the PA. Commodity platforms are and include: 2U, dual socket servers (20-24 cores) up to 192GB of memory a battery-backed write cache controller 12 or more large form factor drive bays within the server chassis Additional drive bays can be deployed per-server depending on the number of mailboxes, mailbox size, and the server’s scalability. Each server houses a single RAID1 disk pair for the operating system, Exchange binaries, protocol/client logs, and transport database. The rest of the storage is configured as JBOD, using large capacity 7.2K RPM serially attached SCSI (SAS) disks (while SATA disks are also available, the SAS equivalent provides better IO and a lower annualized failure rate). Each disk that houses an Exchange database is formatted with ReFS (with the integrity feature disabled) and the DAG is configured such that AutoReseed formats the disks with ReFS: Set-DatabaseAvailabilityGroup <DAG> -FileSystem ReFS BitLocker is used to encrypt each disk, thereby providing data encryption at rest and mitigating concerns around data theft or disk replacement. For more information, see Enabling BitLocker on Exchange Servers. To ensure that the capacity and IO of each disk is used as efficiently as possible, four database copies are deployed per-disk. The normal run-time copy layout ensures that there is no more than a single active copy per disk. At least one disk in the disk pool is reserved as a hot spare. AutoReseed is enabled and quickly restores database redundancy after a disk failure by activating the hot spare and initiating database copy reseeds. Database Availability Group Design Within each site resilient datacenter pair you will have one or more DAGs. DAG Configuration As with the namespace model, each DAG within the site resilient datacenter pair operates in an unbound model with active copies distributed equally across all servers in the DAG. This model: Ensures that each DAG member’s full stack of services (client connectivity, replication pipeline, transport, etc.) is being validated during normal operations. Distributes the load across as many servers as possible during a failure scenario, thereby only incrementally increasing resource use across the remaining members within the DAG. Each datacenter is symmetrical, with an equal number of DAG members in each datacenter. This means that each DAG has an even number of servers and uses a witness server for quorum maintenance. The DAG is the fundamental building block in Exchange 2016. With respect to DAG size, a larger DAG provides more redundancy and resources. Within the PA, the goal is to deploy larger DAGs (typically starting out with an eight member DAG and increasing the number of servers as required to meet your requirements). You should only create new DAGs when scalability introduces concerns over the existing database copy layout. DAG Network Design The PA leverages a single, non-teamed network interface for both client connectivity and data replication. A single network interface is all that is needed because ultimately our goal is to achieve a standard recovery model regardless of the failure - whether a server failure occurs or a network failure occurs, the result is the same: a database copy is activated on another server within the DAG. This architectural change simplifies the network stack, and obviates the need to manually eliminate heartbeat cross-talk. Note: While your environment may not use IPv6, IPv6 remains enabled per IPv6 support in Exchange. Witness Server Placement Ultimately, the placement of the witness server determines whether the architecture can provide automatic datacenter failover capabilities or whether it will require a manual activation to enable service in the event of a site failure. If your organization has a third location with a network infrastructure that is isolated from network failures that affect the site resilient datacenter pair in which the DAG is deployed, then the recommendation is to deploy the DAG’s witness server in that third location. This configuration gives the DAG the ability to automatically failover databases to the other datacenter in response to a datacenter-level failure event, regardless of which datacenter has the outage. If your organization does not have a third location, consider placing the witness in Azure; alternatively, place the witness server in one of the datacenters within the site resilient datacenter pair. If you have multiple DAGs within the site resilient datacenter pair, then place the witness server for all DAGs in the same datacenter (typically the datacenter where the majority of the users are physically located). Also, make sure the Primary Active Manager (PAM) for each DAG is also located in the same datacenter. Data Resiliency Data resiliency is achieved by deploying multiple database copies. In the PA, database copies are distributed across the site resilient datacenter pair, thereby ensuring that mailbox data is protected from software, hardware and even datacenter failures. Each database has four copies, with two copies in each datacenter, which means at a minimum, the PA requires four servers. Out of these four copies, three of them are configured as highly available. The fourth copy (the copy with the highest Activation Preference number) is configured as a lagged database copy. Due to the server design, each copy of a database is isolated from its other copies, thereby reducing failure domains and increasing the overall availability of the solution as discussed in DAG: Beyond the “A”. The purpose of the lagged database copy is to provide a recovery mechanism for the rare event of system-wide, catastrophic logical corruption. It is not intended for individual mailbox recovery or mailbox item recovery. The lagged database copy is configured with a seven day ReplayLagTime. In addition, the Replay Lag Manager is also enabled to provide dynamic log file play down for lagged copies when availability is compromised. By using the lagged database copy in this manner, it is important to understand that the lagged database copy is not a guaranteed point-in-time backup. The lagged database copy will have an availability threshold, typically around 90%, due to periods where the disk containing a lagged copy is lost due to disk failure, the lagged copy becoming an HA copy (due to automatic play down), as well as, the periods where the lagged database copy is re-building the replay queue. To protect against accidental (or malicious) item deletion, Single Item Recovery or In-Place Hold technologies are used, and the Deleted Item Retention window is set to a value that meets or exceeds any defined item-level recovery SLA. With all of these technologies in play, traditional backups are unnecessary; as a result, the PA leverages Exchange Native Data Protection. Office Online Server Design At a minimum, you will want to deploy two Office Online Servers in each datacenter that hosts Exchange 2016 servers. Each Office Online Server should have 8 processor cores, 32GB of memory and at least 40GB of space dedicated for log files. Note: The Office Online Server infrastructure does not need to be exclusive to Exchange. As such, the hardware guidance takes into account usage by SharePoint and Skype for Business. Be sure to work with any other teams using the Office Online Server infrastructure to ensure the servers are adequately sized for your specific deployment. The Exchange servers within a particular datacenter are configured to use the local Office Online Server farm via the following cmdlet: Set-MailboxServer <East MBX Server> –WACDiscoveryEndPoint https://oos-east.contoso.com/hosting/discovery Summary Exchange Server 2016 continues in the investments introduced in previous versions of Exchange by reducing the server role architecture complexity, aligning with the Preferred Architecture and Office 365 design principles, and improving coexistence with Exchange Server 2013. These changes simplify your Exchange deployment, without decreasing the availability or the resiliency of the deployment. And in some scenarios, when compared to previous generations, the PA increases availability and resiliency of your deployment. Ross Smith IV Principal Program Manager Office 365 Customer ExperienceNew Support Policy for Repaired Exchange Databases
The database repair process is often used as a last ditch effort to recover an Exchange database when no other means of recovery is available. The process should only be followed at the advice of Microsoft Support and after determining that all other recovery options have been exhausted. For many years in many versions of Exchange, the repair process has largely been the same. However, that process is changing, based on information Microsoft has gathered from an extensive analysis of support cases. In short, Microsoft is changing the support policy for databases that have had a repair operation performed on them. Originally a database was supported if the repair was performed using ESEUTIL and ISINTEG/repair cmdlets. Under the new support policy, any database where the repair count is greater than 0 will need to be evacuated – all mailboxes on such a database will need to be moved to a new database. Existing Repair Process The process consists of three steps: Repair the database at the page level Defragmentation of the database to restructure and recreate the database Repair of the logical structures within the database Step 1 of the repair process is accomplished by using ESEUTIL /p. This is typically performed when there is page level corruption in the database - for example, a -1018 JET error, or when a database is left in dirty shutdown state as the result of not having the necessary log files to bring the database to a clean shutdown state. After executing ESEUTIL /p you are prompted to confirm that data loss may result. Selecting OK is required to continue. [PS] C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group>eseutil /p '.\Mailbox Database.edb' Extensible Storage Engine Utilities for Microsoft(R) Exchange Server Version 08.03 Copyright (C) Microsoft Corporation. All Rights Reserved. Initiating REPAIR mode... Database: .\Mailbox Database.edb Temp. Database: TEMPREPAIR4520.EDB Checking database integrity. The database is not up-to-date. This operation may find that this database is corrupt because data from the log files has yet to be placed in the database. To ensure the database is up-to-date please use the 'Recovery' operation. Scanning Status (% complete) 0 10 20 30 40 50 60 70 80 90 100 |----|----|----|----|----|----|----|----|----|----| . Rebuilding MSysObjectsShadow from MSysObjects. Scanning Status (% complete) 0 10 20 30 40 50 60 70 80 90 100 |----|----|----|----|----|----|----|----|----|----| ................................................... Checking the database. Scanning Status (% complete) 0 10 20 30 40 50 60 70 80 90 100 |----|----|----|----|----|----|----|----|----|----| ................................................... Scanning the database. Scanning Status (% complete) 0 10 20 30 40 50 60 70 80 90 100 |----|----|----|----|----|----|----|----|----|----| ................................................... Repairing damaged tables. Scanning Status (% complete) 0 10 20 30 40 50 60 70 80 90 100 |----|----|----|----|----|----|----|----|----|----| ................................................... Repair completed. Database corruption has been repaired! Note: It is recommended that you immediately perform a full backup of this database. If you restore a backup made before the repair, the database will be rolled back to the state it was in at the time of that backup. Operation completed successfully with 595 (JET_wrnDatabaseRepaired, Database corruption has been repaired) after 30.187 seconds. At this point, the database should be in a clean shutdown state and the repair process may proceed. This can be verified with ESEUTIL /mh. [PS] C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group>eseutil /mh '.\Mailbox Database.edb' State: Clean Shutdown Step 2 is to defragment the database using ESEUTIL /d. Defragmentation requires significant free space on the volume that will host the temporary database (typically 110% of the size of the database must be available as free disk space). [PS] C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group>eseutil /d '.\Mailbox Database.edb' Extensible Storage Engine Utilities for Microsoft(R) Exchange Server Version 08.03 Copyright (C) Microsoft Corporation. All Rights Reserved. Initiating DEFRAGMENTATION mode... Database: .\Mailbox Database.edb Defragmentation Status (% complete) 0 10 20 30 40 50 60 70 80 90 100 |----|----|----|----|----|----|----|----|----|----| ................................................... Moving 'TEMPDFRG3620.EDB' to '.\Mailbox Database.edb'... DONE! Note: It is recommended that you immediately perform a full backup of this database. If you restore a backup made before the defragmentation, the database will be rolled back to the state it was in at the time of that backup. Operation completed successfully in 7.547 seconds. Step 3 is the logical repair of the objects within the database. The method used to accomplish this varies by Exchange version. In Exchange 2007, ISINTEG is used to perform the logical repair, as illustrated in the following example: C:\>isinteg -s wingtip-e2k7 -fix -test alltests -verbose -l c:\isinteg.log Databases for server wingtip-e2k7: Only databases marked as Offline can be checked Index Status Database-Name Storage Group Name: First Storage Group 1 Offline Mailbox Database Enter a number to select a database or press Return to exit. 1 You have selected First Storage Group / Mailbox Database. Continue?(Y/N)y Test Categorization Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s Test Restriction Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s Test Search Folder Links result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);time: 0h:0m:0s Test Global result: 0 error(s); 0 warning(s); 0 fix(es); 1 row(s); time: 0h:0m:0s Test Delivered To result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s Test Repl Schedule result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time:0h:0m:0s Test Timed Events result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s Test reference table construction result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s Test Folder result: 0 error(s); 0 warning(s); 0 fix(es); 4996 row(s); time: 0h:0m:2s Test Deleted Messages result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s Test Message result: 0 error(s); 0 warning(s); 0 fix(es); 1789 row(s); time: 0h:0m:0s Test Attachment result: 0 error(s); 0 war ning(s); 0 fix(es); 406 row(s); time: 0h:0m:0s Test Mailbox result: 0 error(s); 0 warning(s); 0 fix(es); 249 row(s); time: 0h:0m:0s Test Sites result: 0 error(s); 0 warning(s); 0 fix(es); 996 row(s); time: 0h:0m:0s Test Categories result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s Test Per-User Read result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time:0h:0m:0s Test special folders result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s Test Message Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s Test Folder Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time: 0h:0m:0s Now in test 20(reference count verification) of total 20 tests; 100% complete. Typically when ISINTEG completes, it advises reviewing the isinteg.log file. At the end of the file is a summary section, listing the number of errors encountered. If the number of errors is greater than zero, you need to re-run the command. Continued repairs need to be performed until the error count reaches 0 or the same number of errors is encountered after two executions. . . . . . SUMMARY . . . . . Total number of tests : 20 Total number of warnings : 0 Total number of errors : 0 Total number of fixes : 0 Total time : 0h:0m:3s In Exchange 2010 and later, ISINTEG was deprecated and certain functions were replaced by the New-MailboxRepairRequest and New-PublicFolderDatabaseRepairRequest cmdlets, both of which allow for repair operations to occur while the database is online. Exchange 2010: [PS] C:\Windows\system32>New-MailboxRepairRequest -Mailbox user252 -CorruptionType SearchFolder,FolderView,AggregateCounts,ProvisionedFolder,MessagePtagCN,MessageID RequestID Mailbox ArchiveMailbox Database Server --------- ------- -------------- -------- ------ 7f499ce3-e Wingtip False Mailbox. WINGTIP-E2K10.Wingti... Exchange 2013: [PS] C:\>New-MailboxRepairRequest -Mailbox User532 -CorruptionType SearchFolder,FolderView,AggregateCounts, ProvisionedFolder,ReplState,MessagePTAGCn,MessageID,RuleMessageClass,RestrictionFolder,FolderACL, UniqueMidIndex,CorruptJunkRule,MissingSpecialFolders,DropAllLazyIndexes,ImapID,ScheduledCheck,Extension1, Extension2,Extension3,Extension4,Extension5 Identity Task Detect Only Job State Progress -------- ---- ----------- --------- -------- a44acf2b {Sea False Queued 0 Upon completion of these repair options, typically the database could be mounted and normal user operations continued. Support Change for Repaired Databases Over the course of the last two years, we have reviewed Watson dumps for Information Store crashes that have been automatically uploaded by customers’ servers. The crashes were cause by inexplicable, seemingly impossible store level corruption. The types of store level corruption varied and they come from many different databases, servers, Exchange versions, and customers. In almost all of these cases one significant fact was noted – the repair count recorded on the database was > 0. When ESEUTIL /p is executed, and a repair to the database is necessary, the repair count is incremented and the repair time is recorded in the header of the database. The repair information stored in the database header will be retained after offline defragmentation . Repair information in the header may be viewed with ESEUTIL /mh. [PS] C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group>eseutil /mh '.\Mailbox Database.edb' Extensible Storage Engine Utilities for Microsoft(R) Exchange Server Version 08.03 Copyright (C) Microsoft Corporation. All Rights Reserved. Initiating FILE DUMP mode... Database: .\Mailbox Database.edb File Type: Database Format ulMagic: 0x89abcdef Engine ulMagic: 0x89abcdef Format ulVersion: 0x620,12 Engine ulVersion: 0x620,12 Created ulVersion: 0x620,12 DB Signature: Create time:04/05/2015 08:39:24 Rand:2178804664 Computer: cbDbPage: 8192 dbtime: 1059112 (0x102928) State: Clean Shutdown Log Required: 0-0 (0x0-0x0) Log Committed: 0-0 (0x0-0x0) Streaming File: No Shadowed: Yes Last Objid: 4020 Scrub Dbtime: 0 (0x0) Scrub Date: 00/00/1900 00:00:00 Repair Count: 2 Repair Date: 04/05/2015 08:39:24 Old Repair Count: 0 Last Consistent: (0x0,0,0) 04/05/2015 08:39:25 Last Attach: (0x0,0,0) 04/05/2015 08:39:24 Last Detach: (0x0,0,0) 04/05/2015 08:39:25 Dbid: 1 Log Signature: Create time:00/00/1900 00:00:00 Rand:0 Computer: OS Version: (6.1.7601 SP 1 NLS 60101.60101) Previous Full Backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00 Previous Incremental Backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00 Previous Copy Backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00 Previous Differential Backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00 Current Full Backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00 Current Shadow copy backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00 cpgUpgrade55Format: 0 cpgUpgradeFreePages: 0 cpgUpgradeSpaceMapPages: 0 ECC Fix Success Count: none Old ECC Fix Success Count: none ECC Fix Error Count: none Old ECC Fix Error Count: none Bad Checksum Error Count: none Old bad Checksum Error Count: none Operation completed successfully in 0.78 seconds. Uncorrectable corruption can linger in a repaired database and cause store crashes and server instability, we have changed our support policy to require an evacuation of any Exchange database that persistently has a repair count or old repair count equal to or greater than 1. Moving mailboxes (and public folders) to new databases will ensure that the underlying database structure is good, free from any corruption that might not be corrected by the database repair process, and it helps prevent store crashes and server instability. Tim McMichaelUsing an Azure VM as a DAG Witness Server
Update 8/13/15: Since publication of this blog post, there have been change in our Azure support statement related to Exchange Server. Please see KB 2721672 for more information. I’m happy to announce support for use of an Azure virtual machine as an Exchange 2013 Database Availability Group witness server. Automatic datacenter failover in Exchange 2013 requires three physical sites, but many of our customers with stretched DAGs only have two physical sites deployed today. By enabling the use of Azure as a third physical site, this provides many of our customers with a cost-effective method for improving the overall availability and resiliency of their Exchange deployment. You can learn more about the deployment and configuration process, as well as learn about our best practices in the TechNet Library article. It’s important to remember that deployment of production Exchange servers is still unsupported on Azure virtual machines, so it’s not yet possible to stretch a DAG into Azure. This announcement is limited to deployment of a file share witness in the Azure cloud. Also note that this is not related to the “Cloud Witness” feature in the Windows Server Technical Preview. Stay tuned for future announcements about additional support for Azure deployment scenarios. Jeff Mealiffe Principal PM Manager Office 365 Customer ExperienceThose pesky lazy indices
In Exchange 2013 there are indices within a given mailbox database. The indices are created, maintained, and deleted by the Information Store Worker Process associated with a given database. These indices are not to be confused with the Exchange content indexes that are built via the Search Foundation engine as they are completely different. Within an Exchange database there can exist any combination of primary, secondary, and lazy indices There is exactly one primary index and one secondary index on the messages table. For example, a message table exists where a primary index is created on DocumentID and a secondary index created on FolderID, IsHidden, and MessageID. Additionally, other lazy indices may be created that reflect client views, search folders, and even views of search folders. Exchange maintains these indices through two methods, an eager method and a lazy method. Primary and secondary indices are always maintained eagerly. Lazy indices are maintained through the lazy indexing process (although some may be maintained eagerly). There are many lazy indices per mailbox and usually multiple per folder. Confused yet? Let us see if we can explain further… The eager method says that when an object is inserted into the table the indices must be immediately updated. In the previous example an insert into the mailbox would require updating the primary index and then all secondary indices created against the same mailbox. This results in a random write being issued on each insertion. If a mailbox had 10 indices this would result in 10 random writes. The performance impacts could be significant depending on the structure of the indices. In some cases, the lazy indices exist but are actually not utilized. This results in update cycles being incurred for data that may actually not be utilized. There do exist certain indices where immediate updating is required – this is why the eager method exists. The lazy method is often utilized to mitigate the performance impact that indexing could cause. When an insertion occurs to the folder an entry is created in a lazy indexing maintenance table with information on the lazy indices that require updating. In this example, two random writes are incurred regardless of the number of lazy indices that require updating. Subsequently when an index is accessed, before returning the results of that index, we apply the maintenance records found within the table ensuring the index is up to date. Three major benefits are derived from this method: Less random writes are incurred on indices. If an index exists but is never used we expend no random writes updating it. If multiple records are inserted at the same time when bringing an index current we can derive some write coalescing. Issue Customers have noted that on versions of Exchange 2013 prior to Cumulative Update 6 the following errors are recorded in the Application log and are seen resulting in Information Store Worker Process termination and subsequently database failover. The failovers and terminations may effect single or multiple databases and often result in databases failing over multiple times a day. Source: MSExchangeIS Event ID: 1001 Level: Error Description: Microsoft Exchange Server Information Store has encountered an internal logic error. Internal error text is (Unable to apply maintenance GetNonKeyColumnValuesForPrimaryKey-norow, index corruption?) with a call stack of ( at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message) at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.HandleIndexCorruptionInternal(Context context, Boolean allowFriendlyCrash, String maintenanceOperation, Nullable`1 messageDocumentId, Exception exception) ...(remaining stack redacted) at EcPoolSessionDoRpc_Managed(_RPC_ASYNC_STATE* pAsyncState, Void* cpxh, UInt32 ulSessionHandle, UInt32* pulFlags, UInt32 cbIn, Byte* rgbIn, UInt32* pcbOut, Byte** ppbOut, UInt32 cbAuxIn, Byte* rgbAuxIn, UInt32* pcbAuxOut, Byte** ppbAuxOut)). Source MSExchangeIS Event ID: 1002 Level: Error Description: Unhandled exception (Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: Unable to apply maintenance GetNonKeyColumnValuesForPrimaryKey-norow, index corruption? at Microsoft.Exchange.Diagnostics.ExAssert.AssertInternal(String formatString, Object[] parameters) at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message) at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.HandleIndexCorruptionInternal(Context context, Boolean allowFriendlyCrash, String maintenanceOperation, Nullable`1 messageDocumentId, Exception exception) at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.HandleIndexCorruption(Context context, Boolean allowFriendlyCrash, String maintenanceOperation, Nullable`1 messageDocumentId, Exception exception) at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.GetNonKeyColumnValuesForPrimaryKey(Context context, Object[] primaryKeyValues) at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.DoMaintenanceDelete(Context context, Byte[] propertyBlob) at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.ApplyMaintenance(Context context, LogicalOperation operation, Byte[] propertyBlob) at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.SaveOrApplyMaintenanceRecord(Context context, MaintenanceRecordData maintenanceRecord, Boolean allowDeferredMaintenanceMode) at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.BuildDeleteRecords(Context context, IColumnValueBag updatedPropBag, Int64& firstUpdateRecord) at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.BuildUpdateRecords(Context context, IColumnValueBag updatedPropBag, Int64& firstUpdateRecord) at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndex.LogUpdate(Context context, IColumnValueBag updatedPropBag, LogicalOperation operation) at Microsoft.Exchange.Server.Storage.LazyIndexing.LogicalIndexCache.TrackIndexUpdate(Context context, Mailbox mailbox, ExchangeId folderId, LogicalIndexType indexType, LogicalOperation operation, IColumnValueBag updatedPropBag) ...(remaining stack redacted) at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch (TryDelegate tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state)). Source: MSExchange Common Event ID: 4999 Level: Error Description: Watson report about to be sent for process id: 9112, with parameters: E12, c-RTL-AMD64, 15.00.0913.022, M.E.Store.Worker, M.E.S.Storage.LazyIndexing, M.E.S.S.L.LogicalIndex.HandleIndexCorruptionInternal, M.E.Diagnostics.ExAssertException, a762, 15.00.0913.000. ErrorReportingEnabled: True Why does this issue occur? Prior to CU6 it was possible that certain index maintenance operations would overlap each other. This would result in an inconsistency between an eager index update and a lazy index update that would happen later. More specifically, we missed outputting an update that was needed later which gets the index into a corrupted state. This would essentially mark the index as corrupted when the lazy index operation could not be applied later, causing the crash, and requiring the index to be rebuilt. How did Microsoft find this bug? Although the result of the index corruption is a failover there was no availability impact in the service. The failovers successfully fixed the indices and no client impact is occurred. (It should be noted that the same is reported in on-premises installations). Thanks to customers that have enabled automatic error reporting an uptick in reports related to lazy indexing was noticed. This allowed our development teams to evaluate the code in question and issue a fix. Why does the Information Store Worker Process terminate due to lazy indexing? The lazy indexing maintenance process encounters issues arising to the maintenance of the indices. When inserting a record into an index we assume the record should not already be in the index. When removing or updating a record within the index we expect that the record already exists. Due to an issue in how the indices were previously built these constraints are violated. Our method of handling this is to terminate the Information Store Worker Process, which results in a database copy failover. The index itself is also deleted and then rebuilt the next time the index is accessed. Although a failover occurred the high availability framework should quickly restore access to the database and the end user should not be impacted. The corrupted index is self healed. Indices and index information is logged via transaction logging and subsequently replicated to other database copies if a Database Availability Group is utilized. Does rebuilding the content index or reseeding the content indices correct this issue? The status of the content index catalogs has no impact on this issue. They are two separate indexing concepts unrelated to each other in the context of this issue. Does reseeding the database copies or removing /adding the database copies correct this issue? No. The indices are stored within the database and any corrupted indices would be reseeded with the database. How do I correct the lazy indexing failures and prevent database failovers? The majority of incorrect indices occur with Exchange 2013 Cumulative Update 5. The issue was first identified through automatic error reporting in Exchange 2013 and subsequently identified in Office 365. It was then fixed in a post Exchange 2013 CU5 build deployed to Office 365 where the incidents of index corruption decreased. The fix has been incorporated into Exchange 2013 Cumulative Update 6. Customers should upgrade to Exchange 2013 CU6 to correct the index creation issue and allow future index operations to proceed successfully. Does Exchange 2013 Cumulative Update 6 prevent a lazy indexing failure? No. There are certain reasons an index may be considered corrupted. Isolated index corruption with subsequent self healing may occur. It is also important to note that indices could have been created in Exchange 2013 CU5 that are corrupted. When these indices are accessed on Exchange 2013 CU6 and newer a database failover may result as the indices are still corrupted. Exchange 2013 CU6 corrects the building of the initial indices which should decrease the frequency of lazy indexing resulting in database failovers. Can I expect failovers for the foreseeable future? In the short term after application of Exchange 2013 CU6 customers may continue to experience failovers if a corrupted index is accessed. The Information Store process automatically cleans up indices that are not accessed after 30 days. The issue should self correct either though failover and immediate cleanup or after the indices are aged out. Can I do something today to correct the corrupted indices? There are two interventions administrators can utilize to discard corrupted indices. The first is to move the mailboxes to a different database. The move process discards indices during the move. The second is to execute a New-MailboxRepairRequest with the CorruptionType “DropAllLazyIndices” parameter against the mailbox. The New-MailboxRepairRequest effectively sets the index age timeout for a given mailbox to 0. The repair process will render the mailbox inaccessible while the repair is in progress and could have significant performance impacts on the server. WE DO NOT recommend either of these options since they would have to be run in bulk against all mailboxes whether or not they have corrupted indices. There is no proactive method to scan for index corruption and identify mailboxes to target the move or request against. Customers that have opened cases with support have reported a significant decrease in the number of failovers associated with lazy indexing after the application of Exchange 2013 CU6. Failovers will continue until all corrupted indices have been accessed, deleted, and subsequently rebuilt. Customers who experience this issue are advised to test and deploy Exchange 2013 CU6 as soon as possible. If upgrading is not possible the database failovers may continue with no other negative side effects noted. Tim McMichael Senior Support Escalation Engineer
