Holy COW! Changes to Recoverable Items versioning in Exchange 2010 SP2 RU3

[Ross Smith IV]

Over the course of the last several months, we've received reports of excessive growth of the Recoverable Items folder (what many term Dumpster, although we no longer refer to it as such). There are two main scenarios that can contribute to this excessive growth.

1. Single Item Recovery and Litigation Hold versioning
2. Calendar Version Logging

Single Item Recovery and Litigation Hold

Single Item Recovery and Litigation Hold enable preservation of data within the mailbox. If you are unfamiliar with these features, see my previous post Single Item Recovery in Exchange 2010 for more information.

In addition to retaining data within the mailbox, both Single Item Recovery and Litigation Hold also enable versioning. Essentially, when an item is changed, a copy-on-write (COW) is performed to preserve the original version of the item. The original item is placed in the Recoverable Items\Versions folder. This folder is not exposed to users.

Given that Exchange 2010 includes a change in how clients connect and access mailbox related data, the copy-on-write activity occurs in the Exchange System Objects (XSO) layer on the Client Access server.

What triggers a copy-on-write?

• For messages and posts (IPM.Note* and IPM.Post*), copy-on-write captures changes in the subject, body, attachments, senders/recipients, and sent/received dates.
• For other types of items, copy-on-write captures any change to the item except for moves between folders and read/unread status changes.
• Drafts are exempt from copy-on-write to prevent excessive copies when they're auto-saved.

What can cause excessive growth in the Recoverable Items folder?

Copy-on-write behavior can lead to excessive growth. We discovered that the following scenario using Microsoft Outlook as a leading cause of excessive copy-on-write:

1. You create a calendar appointment.
2. You attach an Office document to the appointment and save it
3. Later, you decide to open the appointment to reference something in the Office document. You open the document.
4. Behind the scenes, Outlook begins auto-saving this open appointment and its corresponding open document (by default Outlook auto-saves every 3 minutes).
5. Each auto-save event triggers a copy-on-write. But because the auto-save is saving both the Office document and the appointment, there are two copy-on-write events. For each additional attachment, a subsequent copy-on-write version would also be created.

Ouch, is right.

Since attachments are part of the message, it doesn't make sense to create a copy for every attachment in the item. Essentially, when you save an item that has an attachment, Outlook performs the following:

1. CreateAttachment
2. SaveAttachment
3. SaveMessage

The copy-on-write was taking place at both SaveAttachment and SaveMessage. Digging into the code, we found that the call to SaveAttachment ultimately calls the Flush method (a means by which the client syncs the state with the server) on the message it's associated with after the attachment is saved. It was this Flush call that was signaling the copy-on-write code to act.

After further analysis, we realized that the copy-on-write logic is triggered on any Flush call. This was a breakthrough finding because Flush can be initiated in many scenarios, potentially leading to the cause of the thousands of copy-on-write events that several customers saw in their environments.

With Exchange 2010 SP2 RU3 and later, copy-on-write now understands the difference between Flush and Save operations and will only be triggered when a Save operation occurs.

Calendar Version Logging

Calendar Version Logging is the process by which calendar changes that occur within a mailbox are saved via copy-on-write. Calendar Version Logging was introduced in Exchange 2010 to help troubleshoot and repair calendar reliability issues.

The design of Calendar Version Logging is to create a log every time a calendar item has been changed. These logs provide a history of the meeting. You can use the Get-CalendarDiagnosticLog cmdlet to review the history and determine which clients performed a destructive action. The second use for Calendar Version Logging is by the Calendar Repair Assistant, which uses the logs when trying to determine the history of a given calendar item for inconsistency detection.

Calendar Version Logging is enabled by default on mailboxes in Exchange 2010. You can disable or enable this on a mailbox via the CalendarVersionStoreDisabled property. Note, the property name is CalendarVersionStoreDisabled, so the default value of $false means that Calendar Version Logging is enabled by default.  Depending on the mailbox configuration, a different process is followed for storing copies of the calendar items:

1. If the mailbox is not enabled for Single Item Recovery or Litigation Hold, then stripped versions of calendar items are maintained in the root of the Recoverable Items folder for 120 days. The stripped versions (body and non-first level or non-embedded message type attachments are removed) are created using copy-on-write.
2. If the mailbox is enabled for Single Item Recovery or Litigation Hold, full copies of the calendar items are maintained in the Recoverable Items\Deletions or Recoverable Items\Versions folders. A stripped version is created via the copy-on-write infrastructure whenever a hard delete operation is performed against the calendar item in the Recoverable Items\Deletions or Recoverable Items\Versions folders. This stripped version is placed in the root of the Recoverable Items folder and kept for 120 days. The only time a stripped version is not created is when the item's age in the Recoverable Items\Deletions or Recoverable Items\Versions folder is more than 134 days (120 + 14). This can happen if you change the retention period, if the Mailbox Folder Assistant hasn’t been executing, if Litigation Hold was disabled, etc.).

Due to the aforementioned issue with how the copy-on-write logic did not distinguish between Flush and Save operations, Calendar Version Logging, in some cases, could consume a large percentage of the Recoverable Items folder quota (and if you recall, the warning threshold is 20GB and the hard quota is 30GB).

Although SP2 RU3 addresses the copy-on-write issues, to address concerns that Calendar Version Logging could consume all of the Recoverable Items folder’s quota, in SP2 RU2 we introduced a change to the architecture whereby Calendar Version Logging now takes into account the Recoverable Items folder size prior to initiating a copy-on-write.

If the folder size is greater than the RecoverableItemsWarningQuota, Calendar Version Logging is disabled for the mailbox. The RecoverableItemsWarningQuota value that is used depends on the mailbox’s settings:

1. If the mailbox has the UseDatabaseQuotaDefaults set to $true, then the mailbox database’s RecoverableItemsWarningQuota will be used.
2. If the mailbox has the UseDatabaseQuotaDefaults set to $false, then the mailbox’s RecoverableItemsWarningQuota will be used.

When Calendar Version Logging is disabled, the following event is generated in the Application event log on the Client Access server:

Event ID: 5003
Source: MSExchange Mid-Tier Storage
Task Category: CopyOnWrite
Level: Information
Description: User mailbox <legacyExchangeDN> has exceeded the dumpster warning quota. Calendar logging has been disabled for the mailbox.

And that's not all we're doing. I won’t go into specifics at this early stage of development, but we're working to further improve Calendar Version Logging to minimize the impact this feature has in your deployment. When I can share more, I will do so via this blog.

Ross Smith IV
Principal Program Manager
Exchange Customer Experience