Forum Discussion
Huge Sync Issues with Large Libary - Need Help :(
Hi, a few months we've migrated a rather large folder from a network drive to a SharePoint Online Document Library.
This all wall worked without any issues. The document library has about 300.000 files is somewhere around 300GB in size.
Now this folder tree is rather archaic and consists of many many subfolders and unfortunately none dares to start sorting this thing, which makes browsing that folder structure through the web ui rather cumbersome as the loading penalty per folder is too big, especially if you open a folder with many subfolders and sharepoint only ever loads 50 or so per scroll (some keyboard navigation would be awesome). Also the browser file previewer for .msg files isn't sufficient, since you can't view/open attachments in that .msg file.
So I decided to enable OneDrive NGSC sync for that library to alleviate many of the performance and usability issues. The users were already on Windows 10 v1709 (aka Files on Demand Feature)
Little did I know that the NGSC limits are not the same as the SPO limits, namely max. items sync limit of 100.000 for NGSC after which the sync becomes unstable.
The affected users didn't mention that problem until recently, so now I'm stuck with possibly hundereds of unsynced (on users device) files littered around that folder structure and now idea which ones and no idea on how to get that thing to sync again.
Does anyone have any ideas?
Thanks
Yes. Documentation is here.
Although SharePoint Online can store 30 million documents per library, the sync performance of OneDrive may start to decline when you store more than 100,000 files in a single OneDrive for Business site or team site library. To resolve this limitation, ensure that files are stored in multiple folders/libraries. If you have more than 100,000 files in a OneDrive for Business site, you may need to wait an extended period of time with OneDrive continuously syncing before sync can complete.There are other limits on viewing a document library using the web that may affect how you structure your files in OneDrive.We're always working to optimize OneDrive to better handle libraries with a large number of files.
9 Replies
i would start by splitting the library up into many libraries, using the top level folders as new libraries could be a good starting point. I would also want to know why users think they need so many files when they are working offline, if they need a certain set of files, those could be put into their own library for improved management flexibility.
- Hi Dean,
Agreed, splitting up the library will be the second step in this process.
As for the users need. They don't actually need those files offline, that is way we've waited until to Win10 1709 to use Files on Demand. Actually the users only has a few GB offline available. What they do need tho is easy access to their documents, which the browser view currently doesn't offer.
I'm still stuck on the first step of the recovery process. I think I can limit the out of sync clients to a single user at the moment. Which makes this a little easier to resolve.
I'm kind of stumbling over Windows 10 1709 file handling of synced libraries at the moment.
My first instinct was to sync the library with some other tool, like the new SharePoint Migration Utility. The goal is to upload the missing documents that were created on the client but haven't been uploaded yet. Unfortunately am unable to accomplish this, as the Migration Utility somehow can't read the folder structure.
Any ideas? I don't think splitting up the library before will help in any way, because of the way OneDrive/Windows handles the synced files.I ran into this issue once and I was able to use Teracopy from the original source, chose to "Skip all" of the files that already existed, and it went in and replaced all missing folders and documents.
Hi Ivan,
The wide use of NGSC with such as data volume does not bode well. Are you able to determine how many users have synched this document library (and how much data is offline)? This will also place a burden on the SharePoint infrastructure. Understanding the "size" of the problem is key to determine the best way to resolve it. I am not a fan of offline tools (also from data security + RM perspective).
The good news is: I can help you with the 2 original problems.
Browsing
My company has developed a browser-based tool named Migration Manager that shows SharePoint content as a windows explorer tree in browser. It uses REST API to retrieve the data from SharePoint. Combined with caching techniques this results in a very fast solution. Check out http://www.slimapplications.com/product/migration-manager/ for details.Viewing emails
The OOTB msg viewing capabilities of SharePoint do not expose the email attachments. Migration manager does provide this functionality in the browser: users can click on email attachments and directly open them (without any plugins). There is a 15-day full functionality trial available. I can also grant you access to our SharePoint Online demo environment if you want to see it in action.Paul | SLIM Applications http://www.slimapplications.com/
Is that sync limit for NGSC documented?
cc Stephen Rose
Yes. Documentation is here.
Although SharePoint Online can store 30 million documents per library, the sync performance of OneDrive may start to decline when you store more than 100,000 files in a single OneDrive for Business site or team site library. To resolve this limitation, ensure that files are stored in multiple folders/libraries. If you have more than 100,000 files in a OneDrive for Business site, you may need to wait an extended period of time with OneDrive continuously syncing before sync can complete.There are other limits on viewing a document library using the web that may affect how you structure your files in OneDrive.We're always working to optimize OneDrive to better handle libraries with a large number of files.- AFAIK, there is not such a limit when synchronizing files no matter if you are doing from ODFB or a SPO Doc. Library
