Office 365: Auto-Expanding Archives FAQ

In recent months we’ve been receiving quite a few questions from customers regarding auto-expanding archives. I’d like to clear up some of the misconceptions and answer some of the most frequently asked questions we get on this subject. Think of this as a FAQ on the subject. Before we get going, I strongly advise folks to understand Message Records Management concepts like retention policies and tags. For example - there have been a few cases that we saw where customers enabled auto-expanding archives, but no archiving policies have been configured for the user.

What are auto-expanding archives?

It’s essentially a component that provides our customers the ability to grow their mailbox archives over the normal 100GB quota limit. Much of the core concepts are covered in Overview of unlimited archiving in Office 365 and Enable unlimited archiving in Office 365 - Admin Help. Let’s look at a simple picture to demonstrate this at a high level: The user’s primary mailbox has an archive mailbox (the main archive) associated with it. The main archive is expanded with additional storage as content is moved to the archive mailbox either by the user or as part of an archival policy on the mailbox.

What are the licensing requirements for auto-expanding archives?

Auto-expanding archives are supported under the following service plans (EDU organizations included):

• Exchange Online Plan 2
• Exchange Online Archiving addon
• Office 365 Enterprise E3/E5

The above service plans contain the following capabilities we check on the mailbox:

• BPOS_S_ArchiveAddOn,
• BPOS_S_Archive,
• BPOS_S_Enterprise

PS C:\> Get-Mailbox testing1|select PersistedCapabilities
PersistedCapabilities
---------------------
{BPOS_S_EquivioAnalytics, BPOS_S_CustomerLockbox, BPOS_S_Analytics, BPOS_S_Enterprise}

The supported recipient types are:

• SharedMailbox, or
• UserMailbox, or
• MailUser

When can I expect the archive to expand?

We have a few values that we track on the Main Archive within this feature. The important value for our customers is essentially called the transition threshold. This is the size threshold of the main archive at which we will start provisioning new auxiliary archives (“Additional storage” in the illustration above) and start moving content from the main archive to the auxiliary archive. This threshold size is currently 90GB.

Why do I need to wait 30 days for expansion to take place?

This question comes up a lot and can be confusing, but technically, archives are expanded as soon as the managed folder assistant is able to successfully run on the main archive that has reached the transition size (90GB). The 30-day period is the retention period for data that has been moved from the main archive to the auxiliary archive. We call this data ghosted content within the archive mailbox location and we keep a copy for 30 days to provide us (and you) with a safety net in the service in case we run into any issues during the data move from main archive to the auxiliary archive. After that period expires we flush the data in the main archive to release the space.

Will the archive immediately expand when it reaches the threshold?

Expansion does not occur in real-time, in other words we only check if a mailbox should be expanded when managed folder assistant runs on the main archive. Managed folder assistant is expected to process the mailbox successfully within 7 days (it might be sooner depending on service load) or – you can initiate managed folder assistant on the main archive manually. The below is an example:

PS C:\> Get-Mailbox testing1|select -ExpandProperty MailboxLocations
1;b2b6421a-8bb0-4a91-9504-73cf42af570f;MainArchive;namprd00.prod.outlook.com;e4811b9b-4a10-4575-bb21-db3d32a9f4c2
1;df788e2e-cdaf-44fa-bc52-b7ef9f9bb40c;Primary;namprd00.prod.outlook.com;3128bfb0-76a7-4492-9de2-4924c8b8e443
1; ab7b6054-eab4-4ea4-bb71-58dc6c8bacc1;AuxArchive;namprd00.prod.outlook.com; 76d35997-4d27-402c-aa3e-ffcd3f898faf
PS C:\> Start-ManagedFolderAssistant b2b6421a-8bb0-4a91-9504-73cf42af570f

What happens if the archive has reached the quota limit and I enable auto-expansion?

In the past, if a mailbox under litigation hold was enabled for auto-expanding archive we would not increase the archive and recoverable items quotas above 100GB. We have seen many cases where customers would enable auto-expanding archives on a mailbox that has already reached the archive quota limit and expect expansion to occur immediately. This puts a lot of pressure on the mailbox since there isn't any buffer to allow the expansion to complete successfully. We have recently made some changes where we now increase the archive quota and recoverable items quota to 110GB if the mailbox is under litigation hold and auto-expanding archive is being enabled on the mailbox level. If you enabled auto-expanding at the org level and/or mailbox level prior to this change, but the archive and recoverable items quotas are still 100GB, you can re-enable auto-expanding archives on the mailbox to get the additional 10GB increase. This will have no other effect on the archive when you run Enable-Mailbox and any quotas above 110GB will not be affected. Let's look at a quick example of a mailbox under litigation hold that was enabled for auto-expanding archive prior to the change mentioned above:

PS C:\> Get-mailbox testing1|select archivequota,recoverableitemsquota
ArchiveQuota RecoverableItemsQuota
------------ ---------------------
100 GB (107,374,182,400 bytes) 100 GB (107,374,182,400 bytes)
PS C:\> enable-mailbox testing1 -AutoExpandingArchive
WARNING: Auto-expanding Archive is already enabled for 'testing1'.
Name Alias ServerName ProhibitSendQuota
---- ----- ---------- -----------------
Testing1 Testing1 co2pr00mb0198 99 GB (106,300,440,576 bytes)
PS C:\> get-mailbox testing1|select archivequota,recoverableitemsquota
ArchiveQuota RecoverableItemsQuota
------------ ---------------------
110 GB (118,111,600,640 bytes) 110 GB (118,111,600,640 bytes)

As you can see, after re-enabling auto-expanding archive, the quotas received an additional 10GB increase.

How do quotas work when a mailbox is enabled for auto-expanding archives?

Quotas seem to be a subject of confusion when auto-expanding archives are in the mix, so I want to focus a little bit on this topic. The Archive and Recoverable Items quotas are essentially ignored for the aggregate size of all locations for the archive. We do however, still utilize and enforce the archive and recoverable items quotas to ensure each mailbox location does not grow out of control; we essentially keep the size of each location within the quota threshold. If we look at the current mailbox example, each mailbox location for the archive (Main Archive and Aux Archive) will not grow past the Archive and Recoverable items quota limit which is set at 100GB and 30GB. But the aggregate size over all the auxiliary archives can go past the 100GB and 30GB values.

PS C:\> get-mailbox testing1 |select RecoverableItemsQuota,ArchiveQuota,ArchiveWarningQuota
RecoverableItemsQuota ArchiveQuota ArchiveWarningQuota
--------------------- ------------ -------------------
30 GB (32,212,254,720 bytes) 100 GB (107,374,182,400 bytes) 90 GB (96,636,764,160 bytes)

We don’t increase the recoverable items quota in this case since the mailbox is not on any hold and recoverable items are removed after the RetainDeletedItemsFor period (default 14 days). The behavior for customers that have mailboxes on litigation hold will be a little different. In this scenario we increase the recoverable items and archive quotas to 110GB as seen below at the mailbox level (not the organization level). The mailbox was on litigation hold prior to enabling auto-expanding archives on the mailbox, so we increased the quotas to 110GB for recoverable items and archive quotas.

PS C:\> Get-mailbox testing2 |select RecoverableItemsQuota,ArchiveQuota,ArchiveWarningQuota
RecoverableItemsQuota ArchiveQuota ArchiveWarningQuota
--------------------- ------------ -------------------
110 GB (118,111,600,640 bytes) 110 GB (118,111,600,640 bytes) 100 GB (107,374,182,400 bytes)

Now let’s look at the aggregate item size of a mailbox with quite a lot of data. Here we can see that the TotalDeletedItemSize is way above the 110GB recoverable items quota for the mailbox, so expansion is working great.

PS C:\> Get-mailbox TheWhale|select RecoverableItemsQuota,ArchiveQuota,ArchiveWarningQuota
RecoverableItemsQuota ArchiveQuota ArchiveWarningQuota
--------------------- ------------ -------------------
110 GB (118,111,600,640 bytes) 110 GB (118,111,600,640 bytes) 100 GB (107,374,182,400 bytes)
PS C:\> Get-MailboxStatistics TheWhale -Archive |select TotalItemSize,TotalDeletedItemSize
TotalItemSize TotalDeletedItemSize
------------- --------------------
393 GB (421,964,016,454 bytes) 724 GB (777,408,505,123 bytes)

Let’s look at TotalDeletedItemSize for the individual mailbox locations for the main archive and the auxiliary archive.

PS C:\> Get-MailboxStatistics e7d87e1c-b39f-493f-8e56-9d271d6d23ea |select TotalDeletedItemSize
TotalDeletedItemSize
--------------------
102.6 GB (110,125,981,616 bytes)

We can see that the main archive mailbox location is currently sitting at 102.6GB and below we see one of the auxiliary archives is at 53.75GB.

PS C:\> Get-MailboxStatistics 62928e39-bc01-4763-84f5-84ac314ea5e1 |select TotalDeletedItemSize
TotalDeletedItemSize
--------------------
53.75 GB (57,714,208,258 bytes)

This mailbox will get another auxiliary expansion location assigned to it soon and content will then start to move into that new location. The main archive in this mailbox also now contains 51.63GB of ghosted content that will be flushed within 30 days to release the space within the main archive. If you recall, we keep the content that we move to a new auxiliary location in the main archive for 30 days as a safety net. The most recent auxiliary archive provisioned for this mailbox was mailbox guid 62928e39-bc01-4763-84f5-84ac314ea5e1. Let’s see what content was moved and is now marked as ghosted content in the main archive (e7d87e1c-b39f-493f-8e56-9d271d6d23ea).

PS C:\> Get-MailboxFolderStatistics e7d87e1c-b39f-493f-8e56-9d271d6d23ea|?{$_.LastMovedTimeStamp -and $_.Foldersize -gt 0}|select FolderSize,LastMovedTimeStamp,Contentmailboxguid
FolderSize LastMovedTimeStamp ContentMailboxGuid
---------- ------------------ ------------------
15.74 GB (16,900,662,851 bytes) 3/29/2018 2:24:04 PM 62928e39-bc01-4763-84f5-84ac314ea5e1
898.9 KB (920,440 bytes) 3/29/2018 2:24:04 PM 62928e39-bc01-4763-84f5-84ac314ea5e1
28.4 GB (30,495,702,247 bytes) 3/29/2018 2:24:05 PM 62928e39-bc01-4763-84f5-84ac314ea5e1
7.485 GB (8,037,032,197 bytes) 3/29/2018 2:24:05 PM 62928e39-bc01-4763-84f5-84ac314ea5e1

We can see the 51.63GB that are marked as ghosted content since we have a LastMovedTimeStamp of 3/29/2018 and a ContentMailboxGuid associated with the folders – which is the newly provisioned auxiliary location 62928e39-bc01-4763-84f5-84ac314ea5e1. This content will be flushed within approximately 30 days.

How much content is being moved to auxiliary archives during the transition?

These numbers can change as the service evolves, but right now we will move 50% of the occupied size in the main archive. If you have 100GB in the Main Archive at the point where Managed Folder Assistant processes the main archive, we will expand and provision a new auxiliary archive and move about 50GB into that auxiliary archive. We only move 50% to cater for additional folder growth in those folders that have been moved.

How is the transition size calculated?

Well, we simply look at how much occupied space the main archive is utilizing. Occupied space is essentially any folder in the main archive that is movable or type DeletedItems or RecoverableItems (folders marked green below). You may notice that the DeletedItems and RecoverableItems folder types are not movable. For these folders we create movable folders under the root of the folders and move the content to those newly created folders to get picked up by managed folder assistant. All of this happens in the background, so you don’t have to worry about it. Folders that do not fall into those categories are not taken into consideration during expansion. So please do not move 10’s of GB into the /Top of Information Store or the /Archive folders – we don’t cater for those… yet.

PS C:\> Get-MailboxFolderStatistics b2b6421a-8bb0-4a91-9504-73cf42af570f|select folderpath,movable,*type*
FolderPath Movable FolderType
---------- ------- ----------
/Top of Information Store False Root
/Archive False Archive
/Calendar True User Created
/Calendar/United States holidays True User Created
/Clutter False Clutter
/Deleted Items False DeletedItems
/Drafts True User Created
/ExternalContacts False ExternalContacts
/Files False Files
/Inbox True User Created
/PersonMetadata True User Created
/Sent Items True User Created
/Recoverable Items False RecoverableItemsRoot
/Calendar Logging False CalendarLogging
/Deletions False RecoverableItemsDeletions
/DiscoveryHolds False RecoverableItemsDiscoveryHolds
/Purges False RecoverableItemsPurges
/Versions False RecoverableItemsVersions

Can I ingest more than 100GB into the archive via a third party tool?

This is quite a tricky one for us. We don’t officially support bulk ingestion of more than 100GB into the archive at any one given time. There is currently work on going to provide solutions for these scenarios in the future, but we don’t have any timelines to share right now. Since there are multiple ingestion scenarios we suggest that you contact Microsoft so that we can understand your situation and advise you on the best way forward for your ingestion project. Special thanks to the following folks for reviewing and providing valuable input:

• Gagandeep Kohli – Snr Software Engineer
• David Santamaria – Snr Escalations Engineer
• Angela Taylor – Snr Program Manager

That’s all we have for now, I hope this helps you understand this feature a little better. The component team is working hard to continually improve the feature as we grow in the service and find unique scenarios that our customers use. Michael Hall Snr. Service Engineer