Forum Discussion
OneDrive Client, Files on Demand and Syncing large libraries
There are a lot of variables involved that can impact sync performance. One thing to note however:
The OneDrive Team has been working on a method to sync individual folders that will be more performant, its related to the notion of "Sync Root". Essentially, SharePoint currently only allows the "Sync Root" to be the root of a Document Library. How this manifests in practice is that the Sync Client will download ALL the metadata regarding the entire library, even if you're only syncing a single subfolder. When they do manage to finish the work to allow SharePoint to set the Sync Root to a different level, you'll see improved performance when synchronizing a single subfolder.
However, another thing to keep in mind (that my organization learned the hard way), is that large libraries place additional pressure on the SQL infrastructure behind the scenes. Even if everything is working, you'll notice that all operations against a larger library will be slower across the board.
We are in the process of breaking up our larger libraries into smaller ones to ensure that we can maintain better performance in web browsing, sync, etc.
dustintadam I am wondering if you could share any updates on your experience with using OneDrive for Business Sync for large libraries. We have a use case where the company is attempting to replace certain network drives used by many users. i.e. For example, a drive with 5,000+ Folders and 7,500+files
Would you be able to offer any thoughts or input on such plan looking forward into 2020? Any input would be greatly appreciated!
- We also work with Microsoft on the issue and here is the last insight I can share.
Microsoft Support engineer answer:
“I understand that the OneDrive sync has performance issues if we try to sync large libraries due to which we recommend syncing no more than 300,000 files across all document libraries. Also, Performance issues can occur if you have 300,000 items or more across all libraries that you are syncing, even if you are not syncing all items within those libraries”
The second sentence make OneDrive sync useless if we are working with large amount of data. You have to think more then twice how you wan to organise your library... “ex: active vs archive contents” where archived contents cannot be sync.
We personally have issue syncing 1 folder containing 1 document in a library containing 250k documents. Sync never end or take forever wich result updated document get re-uploaded/downloaded after 1hour or more.A couple things to look for and consider:
One of the things we've discovered that isn't really documented anywhere is that the more content you shove into a single Document Library, the worse that library performs. Adding additional Indexes to the Library manually can help, but in general, the fewer items you put into a library the better. This becomes apparent even when browsing the library via the UI: a library with fewer total objects browses faster than one with hundreds of thousands. The Sync client will ultimately be affected by that increased overhead as well: when it makes API calls to detect or replicate changes, it's going to take longer to complete. We learned this the hard way ourselves and are actively working on breaking up our libraries. If you haven't migrated data yet, find a way to break up your content into as many libraries as possible to reduce the total volume. Also bear in mind that as is the nature of all file storage, it never gets smaller, nobody ever deletes anything, if you start with an overly large library, your experience will never get better from that point.
I know that the eventual goal is to get the OD Client to gracefully handle syncing up to a million objects, but that hasn't been publicly communicated and there is no timeline for when that might be realized.
So we took the plunge of moving a file server to OneDrive for Business Plan 2, on request from a client. The migration spanned approx. 10 parent folders (Shared Drives) and roughly 600 000 - 800 000 files in total - 2.4TB. There was one or two folders in excess of 100 000 files which we split out as we learned about the 100 000 limit. All users have the Files on Demand feature enabled and we shared folders from the OBP2 account to respective users (approx. 20 users).
Unfortunately it has been a disaster. With the most common issue being that the end users cannot even sync 1 shared folder to their PC's, with files on demand enabled. It often just hangs in "processing changes" state without any files appearing for days on end.
We raised it with Microsoft support (Premier support) - but here's the strange thing. While their communication has been absolutely dismal - what I've gathered between the radio silence and infrequent responses is that they have run some "diagnostics tool" on some of the affected accounts. Within less than an hour suddenly those affected accounts start syncing the shared folders immediately, things start appearing in the app at light speed. It works brilliantly. But then after say 48 hours the user's OneDrive account/sync is "broken" again and just hangs forever.
I've often struggled to gather any precise responses from MS Support team on the issue and what they did when, but the client is now cancelling with MS and wants us to find another solution. Perhaps the scope was too large for OneDrive for Business, or we did it wrong or missed the fine print, but we've also learned a hard lesson that support for the product is also poor and not business ready. I have subsequently cancelled all OneDrive migrations lined up in future for fear of this happening to others.
