Forum Discussion
Solved
Migrating from network drive to SharePoint
We currently work with a 5TB network drive with 2,400,000 files, adding 15,000 files per month. The drive is organized with one folder per year, then one subfolder per month, etc.
I was looking into migrating the network drive to one SharePoint library with on demand files, but I found this page that says "for optimum performance we recommend syncing no more than 300,000 files across all document libraries".
So... if I can't use one library and I can't split it in multiple libraries, how do I organize it?
Is SharePoint not the right tool?
stenci To get around the 300000 file limit it's an issue where you have to throw hardware at it. Balance the load across multiple machines using multiple clients.
SharePoint may not be the best place for this. As I said it sounds like this process just using SharePoint as a dumping ground.
I would say that if you're required to use SharePoint - this is incurring a lot of technical debt because of the need to use OneDrive client to sync contents as if they were in a local drive. Re-engineer the process to work for the place where you are storing the files there are plenty of different ways to automate the file upload process. I would also be mindful of how many files that are created that are like versioned by adding something to the end of the file because that could be de-duped by using versioning in SharePoint.
Whatever the outcome may be document what is going on because IMO - this isn't a process or development that I would want to come into having to figure out what is going on with it.
stenci The key word in the statements is SYNCING - don't use OneDrive sync client to pull down libraries where the combined total of items is over 300000 items as that is a known performance issue. Document libraries in SharePoint Online are built to handle large amounts of content - but there are best practices and set up to consider before migrating a large number of files to a single library.
TimLB Smart caching / on demand file sync is a requirement, so SharePoint is not the right tool for us?
Is it possible to configure the library with the sync on demand for the last year and use the hand picked sync for the older folders?
For example we could keep available with on demand caching the last one year, or 180,000 files, and the older stuff, without on demand caching, will only be available if the user manually syncs the folder?
What you'll need is to structure of libraries/folders so that you don't have to "Sync" or access folders/files that may go over the limit. You'll need to take into consideration how many folders/files do you produce in a year, month, etc. separating them as libraries if necessary.
Also, another storage may be more suited if you think files/projects are no longer going to be accessed.
Fred
