Forum Discussion
Large document libraries in SharePoint Online
I would like to start over a discussion over large document libraries in SharePoint Online.
I hope I get a lot response over your experience with it.
We have the following scenario:
We use MS Dynamics CRM Online.
For each opportunity in CRM, CRM creates a corresponding folder with a subfolder structure in SharePoint.
(I know: “Use Metadata instead of folders” – but the user loves folders and we need it – see below)
In these Folders we’re atomically create some documents and also a OneNote for each opportunity.
CRM creates also a link in the opportunity to the SharePoint folder.
Some of these Folders have separate Permissions for other peoples (automatically).
These peoples uses a direct link to access these Folders.
Also we made a special view for them.
We are working with a lot of documents there:
Actually we have over 1.400 root Folder and over 100.000 items in these library.
In the moment all is working very well and we made some good things to overcome the 5.000 List items view threshold.
And we do a lot for the peoples to help them finding there Folders (Views / Links / Search…)
But I think in the future we get problems with so much documents and we have to create a new strategy for it.
One idea is to create a Document Library based on a template for each opportunity instead of Folders.
Then we also can sync all needed Libraries to the local workstation for offline use.
But I didn’t found any information about a limit for libraries per site.
Another idea is to work further with folders and create a list as overview to better find the things I need.
Now I would like to get some information’s from you how you are working with so big data pools in SharePoint online or on premise.
Feel free to contact me for more information.
Thank you.
Heiko W.
20 Replies
This is a very interesting discussion - I'm curious what have you considered for an archival strategy? Do you just keep the files for unlimited time, or are there any rules based on Document Type? For example do you store contracts/SOW's in this library? Do those need to live forever? Are there other artifacts that could perhaps be retired?
Hi JaredMatfess,
Even content types aren't used when using this kind of CRM integration it's just a file store.No metadata, no content types. Just folders (or equivalent things like Libraries and document sets) that can be addressed with a single Url.
Within the entity instances wihtin CRM this url can be specified and that is it.
Of course it is possible with CRM add-ins to build something more advanced.
Thanks to all for their replies to this discussion.
It helps me to make a decision which way we go in this case.
First a good news from my sales manager: We don’t need to synchronize the Documents for Offline use.
Now we don’t need to create a new library for each opportunity. Very good.
(Additional Info: With the new sync client it is not possible to sync a list with 100.000 Elements. I didn’t see the sync button in my big list. And it is also not useful.)
Here is the way we go:
- We create a Folder with his subfolders, all templates and permissions we need for each opportunity. Like we do it in the moment.
- When we create the folder we also create a separate list entry for the peoples to found the folder they need faster. Also they can use search in this list.
- In CRM a link is generated to the folder in SharePoint
- Also we get an status update from CRM – based on this we can move old Data to an archive.
I think this is the best way and we don’t need to move the exists documents to a new structure.
We only have to create the list – but it is no problem to write a small tool with CSOM to do this.
The instructions at https://technet.microsoft.com/en-us/library/jj863709.aspx show one approach in which there are doc libraries for each Entity from Dynamics. I would recommend spending some time reading the Technet docs to learn about the various integration options, it looks like there are several.
The integration from CRM into SharePoint is very nice but it is also very basic.
For each entity there is a document library generated and per case a folder.
But It doesn’t create subfolders or other things we need.
Also is the name then not so nice.
It is good to use only from CRM – but not directly.
We have some user they have to do some work for the opportunity they doesn’t use CRM.
They have to go directly to the SharePoint Folder.
- Mmm...without reading this doc: imagine you have 10000 entities in the CRM...are you going to create 10000 document libraries? It sounds quite crazy
All that most people use as a SharePoint integration in CRM is a link to a location (library or folder) in SharePoint. So this isn't something complicated. Additonal thing is that all metadata related to the entity is stored wihtin CRM and therefore not stored with the documents.
So you should almost compare an Entity with a document set.
Also, most CRM users will never really use SharePoint to it's full potential as all they will do is click throguh to a location within sharepoint that holds their documents related to that object in CRM.
Document set is really useful in such situation where you have to group content based on customer. It would be easier to find sets and move them to another document library if and when needed based on the metadata/customer. I think you achieve this through code and also through workflow. Some information about the document set:
http://Document set is really useful in such situation where you have to group content based on customers. It would be easier to find sets and move them to another document library if and when needed. I think you achieve this through code and also through workflow. Some information about the document set https
https://en.share-gate.com/blog/document-sets-making-your-metadata-shineI also wonder since you will be moving content around the links will change and if it would be useful to activate https://support.office.com/en-gb/article/Activate-and-configure-Document-IDs-in-a-site-collection-66345c77-f079-4104-ac7a-e25826849306 feature.
- Well, I will add here that a Document Library can store millions of documents...so the question you have to do yourself is about when you expect to reach those limits. On the other hand, taking into account we are talking here about CRM Online and SPO integration, to me it does not sounds good to create sites for CRM entities or even a document library per CRM entity...if the default integration is designed in that way, then you should carefully think about this. What you could think is on designing a solution to move closed opportunities to a kind of Archive so you don't have so much information in the document library integrated with CRM
Hello jcgonzalezmartin,
Thank you for your Reply.
The limit for the amount of document in a library is actually 30 Millions – this is a lot of and when we create 100.000 document each year we can wait 30 years until we reach them. Until them I’m retired 😊
In the moment I’m checking whether it is necessary to synchronize the documents for offline use.
Then we have the need to create Libraries with a limit of 5.000 documents.
When we move old opportunity’s to another folder we have to make a refresh the link in CRM.
But I think we can do it when the state change comes from CRM.
Hi Heiko,
First off....I have no experience of Dynamics at the current time, so apologies if my points lead nowhere...
What I would do from SP side is use Content Organiser, with metadata and rules set by you - ideally you'll be able to set up and send the relevant metadata from Dynamics to SP (something I don't know if possible) - which you can use to send the documents to the correct list/library. Unique permissions as per your folder scenario above so the relevant permissions are granted to the correct users.
From a front end perspective (search/filtering) I would use AngularJS for such a large amount of data. In my previous point I mentioned using metadata, for your benefit for routing, not necessarily the end users benefit with their love of folders - you can style that how you please.
As Pieter suggested, I would also add on retention rules to either delete or archive 'old' data - if archived....again, I would AngularJS to make the archive searchable.
I would recommend categorizing the opportunities into some type of logical structure (industry, region, client type etc), then putting subsites for the top level of the category, libraries for the opportunities and document sets for each opportunity to contain the files. Doc Sets are an advanced type of folder that provides some additional metadata which can be inherited by the files in it which can facilitate future searching.
I don't think that there is an actual limit to the number of libraries in a site, but navigation techniques impose some practical limits that should be considered (i.e, if you have 100 libraries, it is very difficult to find the one you want:)
Hi Dean_Gross,
Thank you for these information’s.
I think we have to split the folders on anything else (Customer Name / Number…).
But for the user is it hard to find the right folder in these complex structure.
The idea is to split these and create a list with all information’s about the opportunity and the user can search this list and find the right folder or subfolder.
Also the user can go over a link directly to the folder.
Then we have no problem with a lot of libraries.
I knew the use of document set and it is very useful.
But in the moment these feature is only available in the old library experience.
We use the new experience and the user know how to work with it – we can’t go back to the old one.
I hope MS is switching here also to the new experience.
Did you know when it is on the roadmap – I only see a user voice for it.
Hi Heiko,
The main ways of resolving these issues would be:
1. use SharePoint search to present the documents to users
2. split the documents across more folders/document sets/libraries/sites etc.
3. clean up. Are there any documents that can be moved into record centers, or maybe even can documents be deleted?
Hi Pieter Veenstra,
thank you for your Tipps:
- We use search to find the CRM Folder I need.
- This is what we plan to do with our new concept. The question in the moment is to split it in several Folders or in Library. I think the sync / offline Feature is here decisive.
- We plan to archive old opportunity – but when an opportunity goes to an order then we can’t archive the folder (but we can move it to another location)
