Mar 18 2019 07:55 AM
Mar 18 2019 07:55 AM
I thought I'd post some observations regarding the OneDrive sync client we've observed that aren't documented anywhere but we needed to figure out when planning a massive move to SharePoint from on-premise file servers:
Microsoft documents that you shouldn't sync more than 300,000 files across all libraries that the client is connected to, but there was no documentation about Files on Demand limits, and we have observed the following:
The OneDrive client will fail when the dat file that stores object metadata reaches exactly 2GB in size (%localappdata%\Microsoft\OneDrive\settings\Business1). Now, while Microsoft says you shouldn't sync more than 300,000 files, you can connect using files on demand to libraries that contain more than this. The trick here is that in this case, the total number of files and folders matter, lets call them collectively "objects". (Interestingly, when you first connect to a library and the client says "Process changes" and gives you a count, "changes" is the total number of objects in the library that it's bringing down using files on demand and storing in the dat file.)
My suspicion is that since the OneDrive client is still 32bit, it's still subject to certain 32bit process restrictions, but I don't really know. What matters in this case is that up until build 19.033.0218.0009 (19.033.0218.0006 insiders build), the client would fill up the dat file and reach the 2GB limit after about 700-800,000 objects. After build 19.033.0218.0009, it appears that the client has been optimized and no longer needs to store quite as much metadata about each object, "increasing" the upper limit of files on demand. (It seems that in general, each object takes up just over 1KB of data in the dat file, putting the limit somewhere just under 2 million objects). Keep in mind, this is not per library, this is across all libraries, including OneDrive for Business (personal storage), SharePoint Document Libraries, etc.
The client has made some significant improvements in performance quickly as they refine each new build, but there are some things to be aware of before you start connecting clients to large libraries:
It. takes. forever.
The more objects in a library, the longer it's going to take for the client to build it's local cache of files on demand copies of all the items in the library. It seems that in general, the client can process about 50 objects per second, if you were connecting to a library or multiple libraries that had 1.4 million objects, it will take around 8 hours before the client is "caught up".
During the time that the content is being built out locally, Windows processes will also consume a large quantity of system resources. Specifically, explorer.exe and the Search Indexer will consume a lot of CPU and disk as they process the data that the client is building out.
The more resources you have, the better this experience will be. On a moderately powered brand new Latitude with an i5, 8GB of Memory and an SSD OS Drive, the machine's CPU was pretty heavily taxed (over 80% CPU) for over 8 hours connecting to libraries with around 1.5 million objects. On a much more powerful PC with an i7 and 16GB of memory, the strain was closer to 30% CPU, which wouldn't cripple an end user while they wait for the client and Windows to finish processing data. But, most organizations don't deploy $2000 computers to everyone, so be mindful when planning your Team-Site automount policies.
Restarts can be painful. when the OS boots back up OneDrive has to figure out what changed in the libraries in the cloud and compare that to it's local cache. I've seen this process take anywhere from 15 minutes to over an hour after restarts, depending on how many objects are in the cache.
Also, if you're connected to a large number of objects in the local cache, you can expect OneDrive to routinely use about a third of CPU on an i5 processor trying to keep itself up to date. This doesn't appear to interfere with the overall performance of the client, but it's an expensive process.
Hopefully over time this will continue to improve, especially as more organizations like mine move massive amounts of data up into SharePoint and retire on premise file servers. If I had to make a design suggestion or two:
- If SharePoint could pre-build a generic metadata file that a client could download on first connection, it would significantly reduce the time it takes to set up a client initially.
- Roll the Activity Log into an API that would allow the client to poll for changes since the last restart (this could also significantly improve the performance of migration products, as they wouldn't have to scan every object in a library when performing delta syncs, and would reduce the load on Microsoft's API endpoints when organizations perform mass migrations)
- Windows to the best of my knowledge doesn't have a mechanism to track changes on disk, i.e. "what recursively changed in this directory tree in the last x timeframe", if it were possible to do this, Windows and SharePoint could eliminate most of the overhead that the OneDrive client has to shoulder on it's own to keep itself up to date.
Speaking to OneDrive engineers at Ignite last year, support for larger libraries is high on their radar, and it's apparent in this latest production release that they are keeping their word on prioritizing iterative improvements for large libraries. If you haven't yet started mass data migrations into SharePoint, I can't stress enough the importance of deeply analyzing your data and understanding what people need access to and structuring your libraries and permissions accordingly. We used PowerBI to analyze our file server content and it was an invaluable tool in our planning.
Happy to chat with anyone struggling with similar issues and share what we did to resolve them. Happy SharePointing!
P.S., shoutout to the OneDrive Product Team, you guys are doing great, love what you've done with the OneDrive client, but for IT Pros struggling with competing product limits and business requirements, documenting behind the scenes technical data and sharing more of the roadmap would be incredibly valuable in helping our companies adopt or plan to adopt OneDrive and SharePoint.
Feb 19 2020 12:14 AM
@Jonas Back indeed.
Microsoft says that OneDrive supports 100k files max (in one library) and 300K files max synced over different libraries.
My experience tells though that you should stay well below these limits in order to have proper syncing.
Of course giving users laptops with better CPUs/more memory (i7/16GB) also helps but this is is not a proper solution.
Feb 19 2020 07:28 AM
Couple other things we've discovered / are aware of:
depending on what article you're reading, you might see the words "file" or "objects" when discussing limits. In reality, the upper limits should always be discussed as objects. From the sync clients' perspective, a folder, as an "object", carries as much weight as a file when it comes to tracking things that can change. When calculating how much pressure is being placed on the client, its important to consider the count of folders in your tree as well as files, add those two together, and that's the object count you should be considering.
For example, on a test VM I can successfully sync content that totals 98,000 files stored in a tree that contains 100,000 folders. It's not super-performant, but in this scenario that's because I'm not syncing 98,000 files, I'm syncing 200,000 'objects'.
Permissions play a huge role in the sync clients performance as well, and often in ways that aren't obvious:
One of the things the OneDrive development team has been working on (they may have fixed this already, but I tend to think not) is the notion of a "Sync Root"; essentially, at the moment, sync can only be initiated from the root of a document library. How this manifests is that even if you tell a user to sync a subfolder, if they still have permissions to content higher in the tree, the sync client will still download and track all the objects in the library the user has access to behind the scenes, even if it's not being presented on disk in Explorer. The overhead isn't as high, but it does add up. The goal is to allow SharePoint to set a per-user custom Sync Root at any level in a library so that the sync context is scoped to only the level at which the content is synced from.
You can help improve that by using the "Restricted View" SharePoint permission. This is a special permission level that has been around since the old Groove client, and is designed to prevent content sync. By setting Restricted View on a parent folder, but granting Edit on child content, you can essentially create online-only content that can't be synced, and the kicker is that the Sync Client wont download anything under that parent folder to its local manifest, helping reduce load significantly. This helps keep content in the same library for business or process continuity.
Its also helpful to enable the 'PermitDisablePermissionInheritance' GPO switch if you are using Read Only permissions and Sync, as without this the OneDrive client is exceptionally slow to sync content if Read Only is used anywhere in the tree being synced.
Feb 24 2020 06:13 PM
Feb 26 2020 10:46 AM
One thing I have noticed when people's OneDrive always says it is is processing changes is that they sometimes have .tmp files on their computers that won't sync. If they have "Hide protected operating system files" checked they won't even see them.
I have had people uncheck that box and delete all the .tmp files they find. The "processing changes" that has gone on for days then stops.
That won't solve your problem but it could well be happening with some of your users.
Feb 26 2020 09:48 PM - edited Feb 26 2020 09:50 PM
@John Twohig I will look out for tmp files thank you.
I am also in the process of synchronizing a Synology NAS with the customer's OneDrive "File Server" library using the cloud sync app installed on the NAS itself. The plan is to utilize local file shares for the users instead of syncing via the OneDrive desktop app. Then just let the NAS handle the rest. For offsite use they can use the browser. I will disable sync button on the customer's tenant/OneDrive/SharePoint admin center.
Mar 19 2020 04:57 PM
Mar 19 2020 10:39 PM
Aug 17 2020 01:12 PM
Any new updates on this topic? I have a Document Library with about 115,000 files in it and our users are having problems with the OneDrive sync getting stuck on "processing changes". Sometimes I can do a OneDrive reset, but other times it doesn't work. We're well under the 300k file limit.
Aug 17 2020 01:19 PM
Nothing major that will simply "solve" the problem unfortunately.
What Client version are your users running?
Does the library contain folders with broken inheritance?
How many users are syncing the library simultaneously?
Does the library have a high rate of change? (i.e. lots of files being modified in many different folders) or is it a lot of old static content?
Aug 17 2020 01:20 PM
@Joe McGowan - The sync limit is a pain.
Do you know if your users have a lot of files in their personal OneDrive folders that usually sync by default. Usually the personal OneDrive folders especially if the desktop protection option is enabled where documents, pictures, and the desktop get sync'ed as well.
For me the easiest way to fix stuck sync'ing is to uninstall OneDrive, hard deleted the OneDrive folder and then resync.
Hopefully this helps.
Aug 17 2020 01:54 PM
The latest Production ring client: 20.134.0705.0008
No broken inheritance.
Around 5-10 users syncing at the same time.
Yes, high rate of change.
I gave myself permissions and started syncing the library and I'm having similar issues. So I don't think its machine related. I can't even reset OneDrive now, it gives an error that is couldn't shut down OneDrive.
Aug 17 2020 02:05 PM
Its likely the rate of change that's causing you the most problems. Every time something changes (especially in a file copy scenario), OneDrive has to re-scan everything that's on disk to ensure that it hasn't lost a file somewhere (i.e. "Was this thing renamed? or was it moved? or both?"). When multiple machines are moving a lot around, it exponentially increases the local demand on each individual machine. The only way to alleviate this is to break up the content across multiple libraries. Do all 100,000+ files need to all be in the same library? do they have to all be brought down locally? if you are lucky enough to answer no to both of those questions, the solution is simple: break up the content into smaller libraries and make stale data online-only (disable sync through in the library settings).
Unfortunately, this isn't exactly OneDrive's fault, Windows still has no mechanism where an app could simply subscribe to a subsystem and say "tell me anytime something changes on disk, including all the context". That leaves it with no other alternative than to literally go searching the disk every time anything updates to make sure it hasn't lost anything... but if it gets overwhelmed with too many changes it simply cant keep up.
Aug 17 2020 02:08 PM
Ok thanks. That was the direction I was heading by either choosing folders/files not to sync or archiving/moving files into a different Document Library to cut down on the files that sync.
Aug 17 2020 02:52 PM
Aug 18 2020 07:54 AM
I think you asked two key questions: "Do all 100,000+ files need to all be in the same library? do they have to all be brought down locally?"
Even though cloud storage seems nearly free nowadays documents still need to be organized. They still have a lifecycle. I suspect that it is pretty rare that many groups in a company will need access to 100,000 documents this month or this year. If things are organized then each group only needs to sync a smaller number of current document libraries. Put the old stuff in archive libraries and use the web browser where you can use the SharePoint search.
I can think of almost no cases where people would need to bring down 100,000 files locally. You only need them locally if you aren't connected to the internet and, in today's world, most of us are seldom without internet. And, if you are somewhere where good internet isn't widely available, trying to sync 100,000 files locally is probably a lost cause.
The thing that gets me most about syncing locally is the security risk. If all these files "must" be synced locally then one would assume they are important. If you have dozens of people syncing all these files to their laptops then there is a good chance that, sooner or later, one will be lost or stolen. Once someone has physical possession of the laptop you have to assume that they will have access to all the files on it. If there is anything sensitive or juicy there you have to be prepared to see it in the news.
There are a few things I found that help:
Aug 18 2020 09:11 AM
Just one point: OneDrive has issues about syncing large libraries (whatever "large" means to you, apparently 100,000+ is when it starts getting tricky), regardless if you are downloading or "hydrating" those files locally or not.
Also it has issues with long paths, and in general the filename and filepath support gap between Sharepoint and NTFS/Windows File Explorer/Individual Apps/Etc.
I lost faith months ago, as It appears that Microsoft is not even actively developing a more robust sync engine (i.e, releasing an x64 version of OneDrive).
Aug 20 2020 05:45 PM
We have files-on-demand enforced by Group Policy on all machines. Still having random sync issues.
I tested a Document Library sync that has 600,000 objects. Took about 90 minutes to sync, then my OneDrive client got stuck at "processing changes". When I try to reset OneDrive, it gives an error that it couldn't shut down OneDrive. I can't end task the OneDrive.exe in task manager, and my sync client is broken at this point. I'm having the same experience with libraries with 100,000 objects.
I have a ticket open with support, but I'm not expecting much.
Aug 24 2020 06:02 PM
Still having issues syncing a 100,000 item Document Library after reinstalling OneDrive and cleaning up the files on the C:\ drive. The library will sync and complete then get stuck at processing changes and its dead. Starting to lose all confidence in this being a solution as a replacement for network shares.
Aug 27 2020 05:27 AM
The release notes on the latest OneDrive client mentions a change to the "processing changes" indicator. I updated my client and will keep an eye on activity.
Aug 31 2020 09:17 AM
Just a small update. I've noticed significant improvements since updating to the latest production ring version of 20.143.0716.0003. I see alot less "processing changes" notifications.