Best way to migrate data between SPO site collections ("sites") in the same tenant

[Chad_V_Kealey]

We've had a few cases where departments have run into issues by cramming all of their files into one library with a hideously deep folder hierarchy and no metadata. Bad on them, yes, but now we in IT are tasked with helping them clean up the mess we told them they'd create without proper planning. 

What is the recommended solution for performing large-scale moves of files in SharePoint Online? We might be moving them from one library to another in the same site, or to a different site in the same tenant. We've tried Powershell (the Move-PnPFile method), but to use that, we need to Get those files, which is constrained by the 5,000 list view threshold (or so Powershell tells us). The "Move to" option in the Modern UI works, but is ingloriously tedious since we often need to move things in chunks (X folders at a time). 

I believe the Mover (part of the SharePoint Migration toolset, I think?) is probably our best bet, but convincing our tenant admins to invest the time to configure and allow it is going to be a challenge. However, if someone can back up my assumption that it will do the job (without costing us anything but time), we might be able to get it done.

Also, I know there are third-party options. We used Sharegate to migrate sites and content from our SP 2010 farm before shutting it down. It worked great, but we couldn't justify the ongoing license cost when we were "done" that migration. I saw something like our current situation coming, but I'm the last person who has any say over what we can spend money on.

[#2]

@Chad_V_Kealey 

I'm also looking at this as we have a number of sites on our tenant where we want to migrate the content to other sites within the same tenant.  I had a look at Mover but I'm not sure if it's going to be suitable for our needs.  When I tried to use the same 365 connector for source and destination I got the following error:

I could get around this by creating two separate 365 connectors but then when I got to the Migration setup screen it's all based around users.  The list of users includes Shared Documents libraries for Team Sites but no other libraries/lists on the site?

It seems the only options available are, as you mentioned, the native file copy/move functionality in SharePoint or a third-party tool like ShareGate.  From previous experience the copy/move functionality is fine for small migrations but for anything over a few hundred files/folders I find it tedious as you said and also there isn't full confidence that all files have been copied as no summary etc is available once completed.

For any decent sized migration I've used ShareGate in the past as it's an amazing tool for this kind of stuff, but as you've mentioned we run into issues with cost etc.

[#3]

@gjayne84 

As a Microsoft Education Tenant admin, I've been asking Microsoft to build something like this for OneNote and Teams for years. This is used by us to archive/migrate Large sets of OneNote files from Sharepoint sites (previously Teacher's OneDrives) to an archive OneDrive (deleted after 2 years of retention) and all exported to Student's as the year ends (note retention). I created a script using similar tactics to your mentioned PNP PowerShell script.

I used CSOM and Powershell Sharepoint modules to find files, group them into batches of 100 and then migrate them.

You can start with things like this: SharePoint Online: Move a File between Document Libraries using PowerShell - SharePoint Diary and SharePoint Online: Move All Files from One Folder to Another using PowerShell - SharePoint Diary, and tailor to your needs. I am looking to update some of my script since it was created using libraries from 2017/2018, and it may be a bit slow at this point.

I had to build in throttle recovery and a few other items since the start of Covid (mass Sharepoint/Teams utilization), but this has worked fine for the most part.

Microsoft has created a Sharepoint (local) to Sharepoint Online migration tool, but this isn't very useful for this case. I wish they would open this up to SPO to SPO, and they full Sharepoint library or subset of folders could be referenced for migration, but that is yet to come. Hoping.......