Forum Discussion
Site collection size limit
I am totally new to sharepoint so please be patient with me.
A company that lost a contract for my company’s customer had created a system that contained around 2 or 3 TB of files. Unfortunately, the customer did NOT put in the contract that the system that contained the files was their property so they were only able to get the files themselves. Those files were moved to a different sub-contractor’s control. The security was compromised so the files had to be quickly moved somewhere else. The company I work for already had SharePoint 2013 so they moved all the files to SharePoint for the customer. Because SharePoint has a 200 GB limit per site (otherwise you have problems), the files before a certain date had to be put into an archive. The archive files do show up on SharePoint but can’t be successfully searched. The newer files were put into a SharePoint site also. The newer files that aren’t archived can be searched, etc. However, search does not work properly for the archived files so any search needs have to be painstakingly done manually, which is quite tedious and time consuming.
Because of the sheer size requirements to store all the files and the size limitations of SharePoint, it may not be the best tool to use. However, the customer does NOT want to spend a lot of money so we have to find a way to make SharePoint work. I have been given the task. I have no previous experience whatsoever with SharePoint. We do have a SharePoint Team of people who do. However, they are in a different group within the company and are not very willing to spend a lot of time on this task. Basically, we have to determine what needs to be done and then send a request for them to do it. So they don’t really work as closely with us as I would like. They know how to create and run PowerShell scripts to do things in SharePoint. I have Site Administrator privileges on the development environment so I can do quite a bit if I know what needs to be done and how to do it.
I need to know if there is a good way to put the files into SharePoint so that size limitation will not be a problem and all the files can be searched, rather than being forced to archive a majority of the files. There are many groupings and sub-groupings of files. The files also contain metadata but entering that metadata manually would be way too time consuming. I have been tasked with finding the quickest and easiest way to put all the files into SharePoint so they are all as searchable as possible.
The technical person who passed this task on to me told me that the only solution he can think of is to create a new SharePoint farm, then divide all the files into many separate site collections of less than 150 GB. However, if the size of a site grows over time it could become larger than 200 GB. I am surprised that SharePoint can’t handle sites that contain more than 200 GB without problems but that’s what I have been told.
Before I start down a certain path I need to find out what the best solution to the problem is from somebody who knows enough about SharePoint that can give me some good possible solutions. This task is very high profile so I am under quite a bit of pressure to figure things out and get it done. Any suggestions or recommendations would be greatly appreciated.
4 Replies
Some things to consider:
1. If these files are no longer active, they could be archived on a File Share which SharePoint can index and then they will be findable using SP Search.
2. If you have compliance requirements, you may want to set up a dedicated Records Management site collection, see https://technet.microsoft.com/en-us/library/ee424394.aspx?f=255&MSPPError=-2147217396
3. if you don't have compliance requirements, a Document Center would be a good solution https://support.office.com/en-us/article/use-a-document-center-site-06096070-d83f-45b8-b02d-ec7a4cf85cac
4. Since this is a high profile project, people with experience should be involved with planning the architecture and implementation techniques and they should then show you how to implement their plan.
5. Asking someone without SP experience to do this is unfair to you and not good management.
As Juan responded the 200 GB is not a hard limit. It depends on the IOPS of the underlying SQL server but also on the backup/restore capabilities of the SQL server.
There are organisations out there that have 100's of TB stored in SharePoint. So it can be done.
I would indeed split the current data volume into smaller parts, i.e. use multiple site collections to keep the content database sizes at an acceptable level (say 200-300 GB unless your SQL administrators tell you they have the IOPS to use larger sizes.
Metadata
You mentioned the files contain metadata? are we talking Office files like docx or other types (pdf, msg, eml, ..)?
Capturing these metadata values into SharePoint columns would improve the search experience for your users.
Migration
You can use PowerShell or migration tools. There are plenty of them. see e.g. https://directory.collab365.community/office365-sharepoint/office-365-migration-software/I would not consider moving such a large data volume using explorer view. Throughput will be insufficient and there is no logging.
SQL Server
You need to involve the SQL Administrators. They need to be aware of the coming increase in database files. and also cater for large transaction logs during migrations.
Paul | SLIM Applications (http://www.slimapplications.com/There are all different types of files.
- The 200 GB limit is just a recommendation for scenarios where there is heavy collaboration in your SP Site and also where the hardware that supports the farm is not a good hardware in terms of IOPS...SP can deal per database with much more than 200 GB...but again, it really depends on what you are going to do with content. Take a look at the SP 2013 SW boundaries and limits to understand what I say: https://technet.microsoft.com/en-us/library/mt493258(v=office.16).aspx
