Over the course of the last several months, we've received reports of excessive growth of the Recoverable Items folder (what many term Dumpster, although we no longer refer to it as such). There are two main scenarios that can contribute to this excessive growth.
Single Item Recovery and Litigation Hold enable preservation of data within the mailbox. If you are unfamiliar with these features, see my previous post Single Item Recovery in Exchange 2010 for more information.
In addition to retaining data within the mailbox, both Single Item Recovery and Litigation Hold also enable versioning. Essentially, when an item is changed, a copy-on-write (COW) is performed to preserve the original version of the item. The original item is placed in the Recoverable Items\Versions folder. This folder is not exposed to users.
Given that Exchange 2010 includes a change in how clients connect and access mailbox related data, the copy-on-write activity occurs in the Exchange System Objects (XSO) layer on the Client Access server.
Copy-on-write behavior can lead to excessive growth. We discovered that the following scenario using Microsoft Outlook as a leading cause of excessive copy-on-write:
Ouch, is right.
Since attachments are part of the message, it doesn't make sense to create a copy for every attachment in the item. Essentially, when you save an item that has an attachment, Outlook performs the following:
The copy-on-write was taking place at both SaveAttachment and SaveMessage. Digging into the code, we found that the call to SaveAttachment ultimately calls the Flush method (a means by which the client syncs the state with the server) on the message it's associated with after the attachment is saved. It was this Flush call that was signaling the copy-on-write code to act.
After further analysis, we realized that the copy-on-write logic is triggered on any Flush call. This was a breakthrough finding because Flush can be initiated in many scenarios, potentially leading to the cause of the thousands of copy-on-write events that several customers saw in their environments.
Calendar Version Logging is the process by which calendar changes that occur within a mailbox are saved via copy-on-write. Calendar Version Logging was introduced in Exchange 2010 to help troubleshoot and repair calendar reliability issues.
The design of Calendar Version Logging is to create a log every time a calendar item has been changed. These logs provide a history of the meeting. You can use the Get-CalendarDiagnosticLog cmdlet to review the history and determine which clients performed a destructive action. The second use for Calendar Version Logging is by the Calendar Repair Assistant, which uses the logs when trying to determine the history of a given calendar item for inconsistency detection.
Calendar Version Logging is enabled by default on mailboxes in Exchange 2010. You can disable or enable this on a mailbox via the CalendarVersionStoreDisabled property. Note, the property name is CalendarVersionStoreDisabled, so the default value of $false means that Calendar Version Logging is enabled by default. Depending on the mailbox configuration, a different process is followed for storing copies of the calendar items:
Due to the aforementioned issue with how the copy-on-write logic did not distinguish between Flush and Save operations, Calendar Version Logging, in some cases, could consume a large percentage of the Recoverable Items folder quota (and if you recall, the warning threshold is 20GB and the hard quota is 30GB).
If the folder size is greater than the RecoverableItemsWarningQuota, Calendar Version Logging is disabled for the mailbox. The RecoverableItemsWarningQuota value that is used depends on the mailbox’s settings:
When Calendar Version Logging is disabled, the following event is generated in the Application event log on the Client Access server:
Event ID: 5003
Source: MSExchange Mid-Tier Storage
Task Category: CopyOnWrite
Description: User mailbox <legacyExchangeDN> has exceeded the dumpster warning quota. Calendar logging has been disabled for the mailbox.
And that's not all we're doing. I won’t go into specifics at this early stage of development, but we're working to further improve Calendar Version Logging to minimize the impact this feature has in your deployment. When I can share more, I will do so via this blog.
Ross Smith IV
Principal Program Manager
Exchange Customer Experience
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.