What's the best way to transfer 4TB of files to SharePoint Online site?

[Roger Seekell]

I have nearly 4TB of assorted files on a drive to upload to a SharePoint Online site.  What's the best/fastest way to upload all these files?  (Hopefully, it doesn't involve a paid third-party product. )  Thanks!

[#2 · Accepted solution]

You can use SharePoint Migration Tool (free): https://spmt.sharepointonline.com/install/default.htm

[#3]

@Maggan Wåhlin for a site with 4TB? That would take forever with the SMT!

Rob
Los Gallardos
Intranet, SharePoint and Power Platform Manager (and classic 1967 Morris Traveller driver)

[#4]

@Maggan Wåhlin  Thanks! I have a ~400GB folder to try first and see how it goes!

[#5]

Well, of course you will need a plan. I´m not saying uploading ALL files at once would be the best approach. It´s possible to use PowerShell (PnP for example), but the same issue will persist with this size... Do you have any suggestions?

[#6]

You might need to think about how you want to organize your files. In this case you might want to have multiple sites, which can be a good choice if you need to apply specific permissions to files in your current folders. Example architecture:

Department X folders goes into SharePoint site "Department X" - only members of Department X have access.
Department Y folders goes into SharePoint site "Department Y" - only members of Department Y have access.
Shared documents goes inte SharePoint site "Company documents" - all members of the company have access.
And so on...

Then you can connect all sites together using the Hub site feature. This allows for a shared search experience, design theme and navigation.

Read more about how to manage large lists and libraries here: https://support.microsoft.com/en-us/office/manage-large-lists-and-libraries-b8588dae-9387-48c2-9248-...

[#7]

The SharePoint Migration Tool uploaded 2TB in 24 hours! This is quite useful! Thanks!

[#8]

I'd like to second this particular point. I worked with a customer that had uploaded 4TB into a single document library in a single site. They are having so many problems as a result, particularly with syncing and they have no easy way of restructuring this architecture now the data has already been uploaded.

Whatever you do, make sure you plan out your SharePoint structure before uploading. Don't upload multiple department's files into a SharePoint Site. Create sites for each department. If a single department has approaching 100,000 files, then create multiple document libraries.

Once you've then mapped out the correct structure in your set of SharePoint Sites, create multiple migration jobs using CSV and load them into the SharePoint Migration Tool.

I have noticed that migration into SharePoint over the weekend is significantly faster than during the week, as I believe that ingestion to SharePoint is throttled during the working week.

Also, the migration tool allows you to create an initial bulk import job, then setup some incremental jobs to pick up any changes once the bulk of data has been imported.

To address RobElliott's point, I disagree. The SharePoint migration tool is very fast (subject to Microsoft throttling) as it uses the approach of uploading to Azure blob storage behind the scenes, then linking into SharePoint libraries. I've done a tonne of migrations and have been impressed with how good the out of the box SharePoint Migration Tool actually is nowadays

[#9]

This Is basically what I wrote earlier, and I do agree!

[#10]

The problem is the use of local temp storage by the SPMT. I have the smallest folder that contains 60GB of data and the available free space on the system is 50GB. So the process runs out of temp space and gets an error. Any suggestions??