Forum Discussion
OneDrive Client, Files on Demand and Syncing large libraries
The latest Production ring client: 20.134.0705.0008
No broken inheritance.
Around 5-10 users syncing at the same time.
Yes, high rate of change.
I gave myself permissions and started syncing the library and I'm having similar issues. So I don't think its machine related. I can't even reset OneDrive now, it gives an error that is couldn't shut down OneDrive.
Its likely the rate of change that's causing you the most problems. Every time something changes (especially in a file copy scenario), OneDrive has to re-scan everything that's on disk to ensure that it hasn't lost a file somewhere (i.e. "Was this thing renamed? or was it moved? or both?"). When multiple machines are moving a lot around, it exponentially increases the local demand on each individual machine. The only way to alleviate this is to break up the content across multiple libraries. Do all 100,000+ files need to all be in the same library? do they have to all be brought down locally? if you are lucky enough to answer no to both of those questions, the solution is simple: break up the content into smaller libraries and make stale data online-only (disable sync through in the library settings).
Unfortunately, this isn't exactly OneDrive's fault, Windows still has no mechanism where an app could simply subscribe to a subsystem and say "tell me anytime something changes on disk, including all the context". That leaves it with no other alternative than to literally go searching the disk every time anything updates to make sure it hasn't lost anything... but if it gets overwhelmed with too many changes it simply cant keep up.
Ok thanks. That was the direction I was heading by either choosing folders/files not to sync or archiving/moving files into a different Document Library to cut down on the files that sync.
- Splitting content in multiple library could help if you think about active/inactive content or archive content. But if you split 100 000 in 10 library of 10k doc each and sync all 10 library you will face the same issues. Unfortunately this problem still présent years after years and Microsoft has not done anything to fix that. Your only real solution is to train user use Office Apps or Web interface to work with documents.
I think you asked two key questions: "Do all 100,000+ files need to all be in the same library? do they have to all be brought down locally?"
Even though cloud storage seems nearly free nowadays documents still need to be organized. They still have a lifecycle. I suspect that it is pretty rare that many groups in a company will need access to 100,000 documents this month or this year. If things are organized then each group only needs to sync a smaller number of current document libraries. Put the old stuff in archive libraries and use the web browser where you can use the SharePoint search.
I can think of almost no cases where people would need to bring down 100,000 files locally. You only need them locally if you aren't connected to the internet and, in today's world, most of us are seldom without internet. And, if you are somewhere where good internet isn't widely available, trying to sync 100,000 files locally is probably a lost cause.
The thing that gets me most about syncing locally is the security risk. If all these files "must" be synced locally then one would assume they are important. If you have dozens of people syncing all these files to their laptops then there is a good chance that, sooner or later, one will be lost or stolen. Once someone has physical possession of the laptop you have to assume that they will have access to all the files on it. If there is anything sensitive or juicy there you have to be prepared to see it in the news.
There are a few things I found that help:
- I tell our people to always have Files on Demand turned on. It reduces the security risk, network traffic, and CPU usage.
- Turn on Storage Sense. We tend to use the same files over and over again and then work on different ones for a while. With appropriate settings for Storage Sense and Files on Demand turned on you can reduce security risk and still almost always have the files you want local. Besides some people never check their C: Drives and then start having problems when it fills up.
- Don't use special characters other than - or _ in filenames. At some point you will encounter some application that doesn't like it. I had one person who's OneDrive always said "processing" It turned out he liked to use ~ in filenames and the OneDrive client didn't like that. That can also happen if Windows itself puts a ~temp file in a folder that is synced.
Just one point: OneDrive has issues about syncing large libraries (whatever "large" means to you, apparently 100,000+ is when it starts getting tricky), regardless if you are downloading or "hydrating" those files locally or not.
Also it has issues with long paths, and in general the filename and filepath support gap between Sharepoint and NTFS/Windows File Explorer/Individual Apps/Etc.
I lost faith months ago, as It appears that Microsoft is not even actively developing a more robust sync engine (i.e, releasing an x64 version of OneDrive).
We have files-on-demand enforced by Group Policy on all machines. Still having random sync issues.
I tested a Document Library sync that has 600,000 objects. Took about 90 minutes to sync, then my OneDrive client got stuck at "processing changes". When I try to reset OneDrive, it gives an error that it couldn't shut down OneDrive. I can't end task the OneDrive.exe in task manager, and my sync client is broken at this point. I'm having the same experience with libraries with 100,000 objects.
I have a ticket open with support, but I'm not expecting much.
Still having issues syncing a 100,000 item Document Library after reinstalling OneDrive and cleaning up the files on the C:\ drive. The library will sync and complete then get stuck at processing changes and its dead. Starting to lose all confidence in this being a solution as a replacement for network shares.
