As part of a bad process that was setup we now have a SharePoint Online List that contains just over 10 million items (audit data). There are likely only about 200 list items that are useful and know what criteria to search to get just those items, but we can't work out how to export the data. It's also being returned in our search results but can't remove it from search (was originally set to show in search). Already set the list to not show in search results but not un-crawling existing data due to too many items issue (tried re-indexing both the site and list, but still no luck). Tried to set restricted permissions so only couple of people can access, but throws error saying can't due to too many items. And we can't delete the list until we get the relevant data exported out.
What we've tried so far with regards to exporting the data.
1. List > Export to CSV (maximum is 30,000 items)
2. List > Export to Excel (maximum rows an Excel file can have is just over 1 million items)
3. Applied filter on the list (loads in batches of around 30, so going through 10 million items isn't viable)
4. PowerShell script using Get-PnPListItem with query to only return relevant results plus set page size to 5000, but kills the machine after running for around 40 minutes.
If anyone has a great idea on how we can get the data out of the list we would be very much appreciated.
Thanks in advance :)
