Huge Sync Issues with Large Libary - Need Help :(

[Ivan Unger]

Hi, a few months we've migrated a rather large folder from a network drive to a SharePoint Online Document Library.

This all wall worked without any issues. The document library has about 300.000 files is somewhere around 300GB in size.

Now this folder tree is rather archaic and consists of many many subfolders and unfortunately none dares to start sorting this thing, which makes browsing that folder structure through the web ui rather cumbersome as the loading penalty per folder is too big, especially if you open a folder with many subfolders and sharepoint only ever loads 50 or so per scroll (some keyboard navigation would be awesome). Also the browser file previewer for .msg files isn't sufficient, since you can't view/open attachments in that .msg file.

So I decided to enable OneDrive NGSC sync for that library to alleviate many of the performance and usability issues. The users were already on Windows 10 v1709 (aka Files on Demand Feature)

Little did I know that the NGSC limits are not the same as the SPO limits, namely max. items sync limit of 100.000 for NGSC after which the sync becomes unstable.

The affected users didn't mention that problem until recently, so now I'm stuck with possibly hundereds of unsynced (on users device) files littered around that folder structure and now idea which ones and no idea on how to get that thing to sync again.

Does anyone have any ideas?

Thanks

[#2]

Is that sync limit for NGSC documented?

cc @Stephen Rose

[#3]

Hi Ivan,

The wide use of NGSC with such as data volume does not bode well. Are you able to determine how many users have synched this document library (and how much data is offline)? This will also place a burden on the SharePoint infrastructure. Understanding the "size" of the problem is key to determine the best way to resolve it. I am not a fan of offline tools (also from data security + RM perspective).

The good news is: I can help you with the 2 original problems.
Browsing
My company has developed a browser-based tool named Migration Manager that shows SharePoint content as a windows explorer tree in browser. It uses REST API to retrieve the data from SharePoint. Combined with caching techniques this results in a very fast solution. Check out http://www.slimapplications.com/product/migration-manager/ for details.

Viewing emails
The OOTB msg viewing capabilities of SharePoint do not expose the email attachments. Migration manager does provide this functionality in the browser: users can click on email attachments and directly open them (without any plugins). There is a 15-day full functionality trial available. I can also grant you access to our SharePoint Online demo environment if you want to see it in action.

Paul | SLIM Applications http://www.slimapplications.com/

[#4]

AFAIK, there is not such a limit when synchronizing files no matter if you are doing from ODFB or a SPO Doc. Library

[#5 · Accepted solution]

Yes. Documentation is here.

Although SharePoint Online can store 30 million documents per library, the sync performance of OneDrive may start to decline when you store more than 100,000 files in a single OneDrive for Business site or team site library. To resolve this limitation, ensure that files are stored in multiple folders/libraries. If you have more than 100,000 files in a OneDrive for Business site, you may need to wait an extended period of time with OneDrive continuously syncing before sync can complete.

There are other limits on viewing a document library using the web that may affect how you structure your files in OneDrive.

We're always working to optimize OneDrive to better handle libraries with a large number of files.

[#6]

i would start by splitting the library up into many libraries, using the top level folders as new libraries could be a good starting point.  I would also want to know why users think they need so many files when they are working offline, if they need a certain set of files, those could be put into their own library for improved management flexibility. 

[#7]

Hi Dean,

Agreed, splitting up the library will be the second step in this process.
As for the users need. They don't actually need those files offline, that is way we've waited until to Win10 1709 to use Files on Demand. Actually the users only has a few GB offline available. What they do need tho is easy access to their documents, which the browser view currently doesn't offer.

I'm still stuck on the first step of the recovery process. I think I can limit the out of sync clients to a single user at the moment. Which makes this a little easier to resolve.
I'm kind of stumbling over Windows 10 1709 file handling of synced libraries at the moment.
My first instinct was to sync the library with some other tool, like the new SharePoint Migration Utility. The goal is to upload the missing documents that were created on the client but haven't been uploaded yet. Unfortunately am unable to accomplish this, as the Migration Utility somehow can't read the folder structure.

Any ideas? I don't think splitting up the library before will help in any way, because of the way OneDrive/Windows handles the synced files.

[#8]

Hi Ivan. I have a couple of suggestions to help resolve the issue, but they both involve using commercial tools so hope you don't mind.

Firstly to identify the files that have not yet synced you could try using Files To Go. Files To Go runs an incremental migration by default so should identify only new and modified files for migration. Just be sure your local and target root folder matches when choosing your source and destination in Files To Go then Files To Go can perform an accurate analysis.

To work with the large library after you have migrated the unsynced files across you could try using Zee Drive. Zee Drive maps network drives to OneDrive for Business and team sites. It should work fine with a large 300,000 item document library.

Disclosure: both Files To Go and Zee Drive are tools my company develops.

Regards,

Myles

[#9]

Are you able to identify and select locally (i.e. on the client) the out-of-sync files?

[#10]

I ran into this issue once and I was able to use Teracopy from the original source, chose to "Skip all" of the files that already existed, and it went in and replaced all missing folders and documents.