Forum Discussion
OneDrive Client, Files on Demand and Syncing large libraries
Are there a lot of broken inheritance permissions in the content that your client is syncing? or does everyone have access to everything? there is a performance switch that can be enabled in the Sync Client if it's handling read-only folders or folders with broken inheritance.
I see sharepoint online (which OneDrive is based on IIRC) recommends 300k objects max for best performance, so it's probably a case that we've exceeded that limit and things are going awry.
- Wil setting Win10 to clear OneDrive cached files every day ease the OneDrive sync issue?
John Twohig I will look out for tmp files thank you.
I am also in the process of synchronizing a Synology NAS with the customer's OneDrive "File Server" library using the cloud sync app installed on the NAS itself. The plan is to utilize local file shares for the users instead of syncing via the OneDrive desktop app. Then just let the NAS handle the rest. For offsite use they can use the browser. I will disable sync button on the customer's tenant/OneDrive/SharePoint admin center.
One thing I have noticed when people's OneDrive always says it is is processing changes is that they sometimes have .tmp files on their computers that won't sync. If they have "Hide protected operating system files" checked they won't even see them.
I have had people uncheck that box and delete all the .tmp files they find. The "processing changes" that has gone on for days then stops.
That won't solve your problem but it could well be happening with some of your users.
- Yeah! Microsoft has finally updated their documentation regarding sync limit on February 21 and now confirm our problems!
"New OneDrive sync app - For optimum performance, we recommend storing no more than 300,000 files across all synced document libraries, even if you're using Files On-Demand or choosing only some folders within the libraries to sync."
https://docs.microsoft.com/en-us/office365/servicedescriptions/sharepoint-online-service-description/sharepoint-online-limits Couple other things we've discovered / are aware of:
depending on what article you're reading, you might see the words "file" or "objects" when discussing limits. In reality, the upper limits should always be discussed as objects. From the sync clients' perspective, a folder, as an "object", carries as much weight as a file when it comes to tracking things that can change. When calculating how much pressure is being placed on the client, its important to consider the count of folders in your tree as well as files, add those two together, and that's the object count you should be considering.
For example, on a test VM I can successfully sync content that totals 98,000 files stored in a tree that contains 100,000 folders. It's not super-performant, but in this scenario that's because I'm not syncing 98,000 files, I'm syncing 200,000 'objects'.
Permissions play a huge role in the sync clients performance as well, and often in ways that aren't obvious:
One of the things the OneDrive development team has been working on (they may have fixed this already, but I tend to think not) is the notion of a "Sync Root"; essentially, at the moment, sync can only be initiated from the root of a document library. How this manifests is that even if you tell a user to sync a subfolder, if they still have permissions to content higher in the tree, the sync client will still download and track all the objects in the library the user has access to behind the scenes, even if it's not being presented on disk in Explorer. The overhead isn't as high, but it does add up. The goal is to allow SharePoint to set a per-user custom Sync Root at any level in a library so that the sync context is scoped to only the level at which the content is synced from.
You can help improve that by using the "Restricted View" SharePoint permission. This is a special permission level that has been around since the old Groove client, and is designed to prevent content sync. By setting Restricted View on a parent folder, but granting Edit on child content, you can essentially create online-only content that can't be synced, and the kicker is that the Sync Client wont download anything under that parent folder to its local manifest, helping reduce load significantly. This helps keep content in the same library for business or process continuity.
Its also helpful to enable the 'PermitDisablePermissionInheritance' GPO switch if you are using Read Only permissions and Sync, as without this the OneDrive client is exceptionally slow to sync content if Read Only is used anywhere in the tree being synced.
https://docs.microsoft.com/en-us/onedrive/use-group-policy#PermitDisablePermissionInheritance
JonasBack indeed.
Microsoft says that OneDrive supports 100k files max (in one library) and 300K files max synced over different libraries.
My experience tells though that you should stay well below these limits in order to have proper syncing.
Of course giving users laptops with better CPUs/more memory (i7/16GB) also helps but this is is not a proper solution.
Hi there. If I understand the terminology correctly... so all folders/files only exist on a single OneDrive for Business Plan 2 account. That means there's only 1 library. This plan 2 account has 9 parent folders. Unfortunately OneDrive doesn't seem to detail or report the number of files (or I just can't find where in the UI), but it should be around 600 000 - 800 000 files in total.
Each user who has O365 Business Premium license (About 20 users total) has been given shared links to certain folders in the above library, but in most cases the shared link is actually a subfolder and not the entire parent folder, see below...
So in at least 2 parent folders in the library there exists subfolders that initially contained in excess of 100 000 files. Due to this causing issues with sharing links to users and us finding out about the 100 000 limitation, we split out alphabetized variants of the subfolders and shared those instead.
Parant Folder A >
Subfolder A-F (Shared)
Subfolder G-K (Shared)
etc...
I'm now considering that despite us overcoming the shared permission issue by alphabetized split of the subfolders, because they exist in a parent folder that contains in excess of 100 000 objects then perhaps this could still cause the issue we are observing? (File on Demand sync at client end just hanging and changes taking forever to process)I'm now considering moving the subfolders out of the parent folder and re-sharing to users so they can sync to their PC's and see what happens...
UPDATE: I signed into a user's OneDrive account, accessed his "Shared With Me" section, clicked sync on a folder shared from the OneDrive Plan 2 library/account that contains 20 000 items only - unfortunately it's not syncing to my PC. Hangs on processing changes, nothing comes through - occasionally says processing 501 or 502 changes then back to nothing again. There goes that theory 😞 So moving things out a parent folder to remove the 100 000 concern probably won't help.
My next plan is to move one of the folders to another OneDrive account and try sharing/syncing that to the user's account.
eddablinHi there. If I understand the terminology correctly... so all folders/files only exist on a single OneDrive for Business Plan 2 account. That means there's only 1 library. This plan 2 account has 9 parent folders. Unfortunately OneDrive doesn't seem to detail or report the number of files (or I just can't find where in the UI), but it should be around 600 000 - 800 000 files in total.
Each user who has O365 Business Premium license (About 20 users total) has been given shared links to certain folders in the above library, but in most cases the shared link is actually a subfolder and not the entire parent folder, see below...
So in at least 2 parent folders in the library there exists subfolders that initially contained in excess of 100 000 files. Due to this causing issues with sharing links to users and us finding out about the 100 000 limitation, we split out alphabetized variants of the subfolders and shared those instead.
Parant Folder A >
Subfolder A-F (Shared)
Subfolder G-K (Shared)
etc...
I'm now considering that despite us overcoming the shared permission issue by alphabetized split of the subfolders, because they exist in a parent folder that contains in excess of 100 000 objects then perhaps this could still cause the issue we are observing? (File on Demand sync at client end just hanging and changes taking forever to process)I'm now considering moving the subfolders out of the parent folder and re-sharing to users so they can sync to their PC's and see what happens...
UPDATE: I signed into a user's OneDrive account, accessed his "Shared With Me" section, clicked sync on a folder shared from the OneDrive Plan 2 library/account that contains 20 000 items only - unfortunately it's not syncing to my PC. Hangs on processing changes, nothing comes through - occasionally says processing 501 or 502 changes then back to nothing again. There goes that theory 😞 So moving things out a parent folder to remove the 100 000 concern probably won't help.
My next plan is to move one of the folders to another OneDrive account and try sharing/syncing that to the user's account.
There's a lot of good knowledge in this thread - keep them coming.
So basically, if you have a huge library with 1 million files you will probably run into problems. If you decided to split them into 20 of them with 50k each you will be better off. But if a user decides they need to sync (even though just on-demand) all 20 of them, the user will still run into problems because the total volume of files sync'ed on this particular user's computer is 1 million?lets hope so, the product team isn't great at discussing performance roadmap objectives
dustintadam let's hope it comes even in incremental steps and not with a major release going from 100k limit to 1M files limit.
What I've heard from a friend is that there is a partner beta that tries to offload the sync client load by identefying stale files and ignoring them.
We are also struggling with sync issues and we ended up splitting large libraries to many smaller 7 months after the initial migration...
- I hear you, its a struggle, our own roll-out of OneDrive has been exceptionally limited because of the upper limits.
I know that it's on their radar, and there are a lot of customers that are struggling. Through the grapevine I hear that the internal target they are going to try and work towards is getting the sync client to handle a million objects gracefully, but they aren't going to attempt to get it to handle more than that.
Obviously, they wont provide a timeline for this officially, work on performance improvements isn't at all the same as building a new feature that's easier to hold yourself to. Could be this year? I don't have a sense of how much of a priority this is for them, and I would suspect that achieving that is going to require engineering changes beyond simply the sync client itself.
All of that is to say, yeah, it'll get significantly better and likely open up most use cases for migrating file server content to SharePoint... but don't hold your breath, it'll happen when it happens. Hi dustintadam ,
Kudos on some valuable insight here. I was wondering if there were any new updates in regards to the one drive sync limitation increase... I am dying and hoping that something will get released soon.
We have spent most of last year planning our SharePoint collaboration environment and we have re-structured our large department libraries to an active (sync'able) and archive (non-sync'able) libraries. We slimmed down the "active" libraries to about "60K" sync'able files for our large departments (from millions of files) and only recommend to sync 1 library per department.
However, we still notice a lot of sync delays, conflicts, and performance overhead.
Please let us know if there is a light at the end of the tunnel.- I would suggest, at least for the individuals who have access to a sub-set of the content, enabling the "Permit Disable Permission Inheritance" Group Policy flag for the OneDrive client:
https://docs.microsoft.com/en-us/onedrive/use-group-policy#PermitDisablePermissionInheritance
You should also consider enabling the Storage Sense group policy option in Windows 10 (on computers running at least 1903) that can automatically make synced files Online Only if they haven't been accessed in a specified time period, as this helps keep the pressure off the sync client as well.
https://support.office.com/en-us/article/use-onedrive-and-storage-sense-in-windows-10-to-manage-disk-space-de5faa9a-6108-4be1-87a6-d90688d08a48
You can enable that manually, but you can also set Storage Sense options through GPO or InTune, depending on how your client's PCs are managed.
Lastly, I find it concerning that all the content is stored in a OneDrive account as opposed to a SharePoint Document Library. While, yes, a ODfB account is in essence a SP Library, and content can be shared from it, it wasn't designed to be used in the same manner. OneDrive libraries are not given the same processing overhead as a SP Library, which is intended to serve a large number of clients. all OneDrive requests place burden on the backend SQL infrastructure to serve back responses to clients, storing the organizations content in OneDrive instead of SharePoint is bound to introduce performance issues, as OneDrive is provisioned to serve a single client primarily.
