SharePoint Online Limits: Should we distribute large document libraries across multiple sites?

[layer9de]

Dear all,

we're currently planning on moving more and more files and folders from classic file servers to document libraries in SharePoint online.

The website about SharePoint online limits states that i.e. a single site can hold up to 25TB of data. In case we have to create multiple large libraries, - for example 10 libraries with 2TB of data and tens of thousands of files each - : Can we easily create all libraries below a single SharePoint site or should we distribute the libraries across multiple sites?

Background: We would like to create a single "Public" document library for each division (about 10) of our company. I'm currently drawn between two ways to go:

a) Creating the document library as additional library within each divisions Team (MS Teams)

b) Creating a dedicated SharePoint site and adding a document library for each division to it

Any suggestions or recommendations are highly appreciated!

Thanks
Michael

[#2]

@layer9de 

Dear Michael,

Your question is normal in that kind of Situation "Strategy to Move to Cloud", and Office 365 SPO is one of the option you can use for File Server Move.

But I would suggest to add one criteria into your plan is to add the freshness of data you want to load.

Because SPO has a certain number of limitations and load a huge volume without any real usage will create a lot of frustrations from user point of view.

Take also care about native usage from File Server difficult to retrieve with SPO: Windows Explorer mode 

The only supported solution from MS to benefit that feature will be via OneDrive SyncApp, but this module is based on sync module not immediate.

Fab

[#3]

@Fabrice Romelard 

Dear Fabrice,

thanks for your quick reply. We are already aware of the limitations on OneDrive syncing (issues when syncing more than 100.000/300.000 items even with Files on Demand) and are also thinking about a guideline for our users regarding which files should be placed in O365 and which not.

Because of the current limits and the preference of most users to work with files in Explorer we will not perform a "Lift and Shift" migration (just move everything to SharePoint document libraries). Instead I am currently thinking of recommending to place only files in O365 for which collaboration and/or external access is needed. And also specify limits - i.e. not placing extremely large amounts of data there - at least speaking to IT in case this is requested.

Regarding distribution of files across multiple document libraries and sites: We are now just creating the foundation for placing documents in the cloud and I would like to avoid misconceptions here so we aren't creating a structure that runs into well known limits very quickly.

I think distributing files and folders across multiple document libraries definitely makes sense (since creating a structure within just one library will hit limits quickly). However, I am still unsure regarding my initial question: Could we create multiple document libraries within one site or should we rather distribute the libraries across multiple SharePoint sites?

From what I can see the 25TB data amount limit applies to a single site which is sufficient. However, the OneDrive sync limit officially is 300.000 items and in reality problems already start at a much lower amount of files. So I'm not sure if the 25TB limit of a single site is actually realistic.

Best Regards
Michael

[#4 · Accepted solution]

The limit is a site collection limit, so you can put many document libraries in the same site just fine. The biggest issue you will want to concern yourself with besides number of files etc. will be security. After 100k files into a parent folder you can no longer break permission inheritance and thus you cannot share files in those folders unless you can path down enough to get under that limit. So make sure if you have a directory with over 100k Items, you set the permissions on the parent folder prior to moving the files in or you won't be able to. If everything should never have custom permission you should be fine, there really isn't anything preventing using one site with multiple libraries.

The reasons you would use multiple libraries or sites would be settings. For example, retention policies are site level so you would want different sites if different "Libraries" needed different retention periods. Do libraries can have search flags turned off, or have various other settings, and same for sites you would want to consider, but mostly, outside of policies, single site is fine there.

Breaking libraries up helps prevent someone from syncing the "Entire File library" by trying to sync at the root of a library. By having multiples this prevents that goof. Site's have no bearing here in that regard since sync isn't site based.

I know I'm just rambling here just kind of typing as it comes in, you seem to have the idea around splitting into libraries, but for sites, it mainly boils down to another permission level and or policies that can be applied only at site level so if having all libraries in that same container would never be different, then one site would ultimately be fine.

[#5]

@Chris Webb 

Thanks Chris for your detailed answer,

regarding the 100.000 item limit when breaking permission inheritance: As mentioned above, our thought is to create an additional document library within each divisions team (MS Teams) which then could be considered as a "public document folder" for them - an official place for documents which need to be accessible for the whole company. We then would break permissions on that library so that the permissions are independent from the team (or SharePoint site / O365 group) so we could i.e. grant everyone in the company access to this folder.

For this to achieve, we would disable permission inheritance on the root of that document library before placing items in it. However regarding Sharing Links: As far as I know when sharing an item, permission is broken for that item. Once the library for which we have broken permissions on the root contains more than 100.000 items: Will it no longer be possible to share items within the whole library (all subfolders) afterwards? Or can inheritance still be broken (and items shared) within subfolders if they don't contain more then 100.000 items (even if inheritance is already broken at the root)?

Thanks
Michael

[#6]

@Chris Webb 

One question to the 100,000 item limit when breaking permission inheritance: The article Customize permissions for a SharePoint list or library mentions the following:

-----------------

When a list or library contains more than 100,000 items, you can’t break permissions inheritance on the list itself. Nor can you re-inherit permissions on the list itself.

When a folder contains more than 100,000 items, you can’t break permissions inheritance on that folder itself. Nor can you re-inherit permissions on that folder itself.

Items within the library or folder hitting the limit (say a single file or folder) won't be impacted--so you could still, for example, break inheritance on any single file inside a library with greater than 100,000 items.

-----------------

If I understand correctly, this means that sharing of files and folders (for which permission inheritance needs to be broken) should still be possible for individual items, even if the parent holds more than 100,000 items. If a library or folder contains more than 100,000 items it just cannot be shared itself. Is that correct?

Thanks
Michael