Forum Discussion
Export to PST via Powershell
- Aug 10, 2017
No way to do it without going to the SCC and initializing the download via the click-one app, sorry. Perhaps you can automate it via AzCopy or some other tool that takes container/token as input - you can get those via the Result property of Get-ComplianceSearchAction.
As for the cmdlet, make sure you also use the -Format parameter!
The Format parameter specifies the format of the search results when you use the Export switch. Valid values are:
FxStream Export to PST files. This is the only option that's available when you export search results from the Security & Compliance Center.
Mime Export to .eml messsage files. This the default value when you use cmdlets to export the search results.It's most likely what causes the issue in your case.
So, I've been fighting to automate using powershell to export a PST from Office 365 off and on for a while.
AzCopy worked great for Sharepoint/OneDrive exports, but exchange data all came down as ".batch" files, which appear to need further post-processing to collect everything into a PST. Someone better than I might know how to piece them together, but I just moved on.
It's a bit of a workaround but here's what I did to automate the PST download.
When you manually download the export using the export tool, the click-once popup downloads their "Unified Export Tool" and populates it with what it needs to get it going. While it's open, you can right-click the process in task manager and click "Open file location". This will take you to the files their tool uses, in your appdata. For me, this location was "%LOCALAPPDATA%\Apps\2.0\QBKH9EZX.XMP\7R4KYC02.WJ7\micr..tool_51a5b647dacf4059_000f.0014_a4b60912e622c727\"
I'm sure Microsoft updates the tool every now and then, so this path may not work indefinitely.
Anyways, I just copied everything to a new folder (I don't recommend doing this, i provide an example to find the file automatically in the script below) and ran the "microsoft.office.client.discovery.unifiedexporttool.exe" which returned the parameters that it accepts. Much like AzCopy, it looks for the URL, Token, and a place to drop the files.
Here's what I slapped together to automatically download the export from Office 365:
The code below has been edited since this post was created to include improvements to the original code and provide a start to finish example to export and download a PST. If you want to use this as a user off-boarding script you'll need to add those tasks as well. Everyone has their own processes, but Tom Aguero's post below has some good examples too.
# Note that I'm not validating any input or providing any adequate error handling.
# This is just an example, you'll need to add these in yourself.
Param(
[Parameter(Mandatory=$True, HelpMessage='Enter the email address that you want to export')]
$Mailbox,
[Parameter(Mandatory=$True, HelpMessage='Enter the URL for the user''s OneDrive here. If you don''t enter one, this will be skipped.')]
$OneDriveURL,
[Parameter(Mandatory=$True, HelpMessage='Enter the path where you want to save the PST file. !NO TRAILING BACKSLASH!')]
$ExportLocation # = ""# you can un-comment the = "" to set a default for this parameter.
)
# Create a search name. You can change this to suit your preference
$SearchName = "$Mailbox PST"
# I'm using the Exchange Online Powershell Module v2. You can install it from an admin session with the following command: Install-Module ExchangeOnlineManagement
Write-Host "Connecting to Exchange Online. Enter your admin credentials in the pop-up (pop-under?) window."
Connect-IPPSSession
Write-Host "Creating compliance search..."
New-ComplianceSearch -Name $SearchName -ExchangeLocation $Mailbox -SharePointLocation $OneDriveURL -AllowNotFoundExchangeLocationsEnabled $true #Create a content search, including the the entire contents of the user's email and onedrive. If you didn't provide a OneDrive URL, or it wasn't valid, it will be ignored.
Write-Host "Starting compliance search..."
Start-ComplianceSearch -Identity $SearchName #Start the search created above
Write-Host "Waiting for compliance search to complete..."
for ($SearchStatus;$SearchStatus -notlike "Completed";){ #Wait then check if the search is complete, loop until complete
Start-Sleep -s 2
$SearchStatus = Get-ComplianceSearch $SearchName | Select-Object -ExpandProperty Status #Get the status of the search
Write-Host -NoNewline "." # Show some sort of status change in the terminal
}
Write-Host "Compliance search is complete!"
Write-Host "Creating export from the search..."
New-ComplianceSearchAction -SearchName $SearchName -Export -Format FxStream -ExchangeArchiveFormat PerUserPst -Scope BothIndexedAndUnindexedItems -EnableDedupe $true -SharePointArchiveFormat IndividualMessage -IncludeSharePointDocumentVersions $true
Start-Sleep -s 5 # Arbitrarily wait 5 seconds to give microsoft's side time to create the SearchAction before the next commands try to run against it. I /COULD/ do a for loop and check, but it's really not worth it.
# Check if the export tool is installed for the user, and download if not.
While (-Not ((Get-ChildItem -Path $($env:LOCALAPPDATA + "\Apps\2.0\") -Filter microsoft.office.client.discovery.unifiedexporttool.exe -Recurse).FullName | Where-Object{ $_ -notmatch "_none_" } | Select-Object -First 1)){
Write-Host "Downloading Unified Export Tool ."
Write-Host "This is installed per-user by the Click-Once installer."
# Credit to Jos Verlinde for his code in Load-ExchangeMFA in the Powershell Gallery! All I've done is update the manifest url and remove all the comments
# Ripped from https://www.powershellgallery.com/packages/Load-ExchangeMFA/1.2
# In case anyone else has any ClickOnce applications they'd like to automate the install for:
# If you're looking for where to find a manifest URL, once you have run the ClickOnce application at least once on your computer, the url for the application manifest can be found in the Windows Registry at "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Uninstall" (yes, CTR apps are installed per-user).
# Look through the keys with names that are 16 characters long hex strings. They'll have a string value (REG_SZ) named either "ShortcutAppId" or "UrlUpdateInfo" that contains the URL as the first part of the string.
$Manifest = "https://complianceclientsdf.blob.core.windows.net/v16/Microsoft.Office.Client.Discovery.UnifiedExportTool.application"
$ElevatePermissions = $true
Try {
Add-Type -AssemblyName System.Deployment
Write-Host "Starting installation of ClickOnce Application $Manifest "
$RemoteURI = [URI]::New( $Manifest , [UriKind]::Absolute)
if (-not $Manifest){
throw "Invalid ConnectionUri parameter '$ConnectionUri'"
}
$HostingManager = New-Object System.Deployment.Application.InPlaceHostingManager -ArgumentList $RemoteURI , $False
Register-ObjectEvent -InputObject $HostingManager -EventName GetManifestCompleted -Action {
new-event -SourceIdentifier "ManifestDownloadComplete"
} | Out-Null
Register-ObjectEvent -InputObject $HostingManager -EventName DownloadApplicationCompleted -Action {
new-event -SourceIdentifier "DownloadApplicationCompleted"
} | Out-Null
$HostingManager.GetManifestAsync()
$event = Wait-Event -SourceIdentifier "ManifestDownloadComplete" -Timeout 15
if ($event ) {
$event | Remove-Event
Write-Host "ClickOnce Manifest Download Completed"
$HostingManager.AssertApplicationRequirements($ElevatePermissions)
$HostingManager.DownloadApplicationAsync()
$event = Wait-Event -SourceIdentifier "DownloadApplicationCompleted" -Timeout 60
if ($event ) {
$event | Remove-Event
Write-Host "ClickOnce Application Download Completed"
}
else {
Write-error "ClickOnce Application Download did not complete in time (60s)"
}
}
else {
Write-error "ClickOnce Manifest Download did not complete in time (15s)"
}
}
finally {
Get-EventSubscriber|? {$_.SourceObject.ToString() -eq 'System.Deployment.Application.InPlaceHostingManager'} | Unregister-Event
}
}
# Find the Unified Export Tool's location and create a variable for it
$ExportExe = ((Get-ChildItem -Path $($env:LOCALAPPDATA + "\Apps\2.0\") -Filter microsoft.office.client.discovery.unifiedexporttool.exe -Recurse).FullName | Where-Object{ $_ -notmatch "_none_" } | Select-Object -First 1)
# Gather the URL and Token from the export in order to start the download
# We only need the ContainerURL and SAS Token at a minimum but we're also pulling others to help with tracking the status of the export.
$ExportName = $SearchName + "_Export"
$ExportDetails = Get-ComplianceSearchAction -Identity $ExportName -IncludeCredential -Details # Get details for the export action
# This method of splitting the Container URL and Token from $ExportDetails is thanks to schmeckendeugler from reddit: https://www.reddit.com/r/PowerShell/comments/ba4fpu/automated_download_of_o365_inbox_archive/
# I was using Convert-FromString before, which was slow and terrible. His way is MUCH better.
$ExportDetails = $ExportDetails.Results.split(";")
$ExportContainerUrl = $ExportDetails[0].trimStart("Container url: ")
$ExportSasToken = $ExportDetails[1].trimStart(" SAS token: ")
$ExportEstSize = ($ExportDetails[18].TrimStart(" Total estimated bytes: ") -as [double])
$ExportTransferred = ($ExportDetails[20].TrimStart(" Total transferred bytes: ") -as [double])
$ExportProgress = $ExportDetails[22].TrimStart(" Progress: ").TrimEnd("%")
$ExportStatus = $ExportDetails[25].TrimStart(" Export status: ")
# Download the exported files from Office 365
Write-Host "Initiating download"
Write-Host "Saving export to: " + $ExportLocation
$Arguments = "-name ""$SearchName""","-source ""$ExportContainerUrl""","-key ""$ExportSasToken""","-dest ""$ExportLocation""","-trace true"
Start-Process -FilePath "$ExportExe" -ArgumentList $Arguments
# The export is now downloading in the background. You can find it in task manager. Let's monitor the progress.
# If you want to use this as part of a user offboarding script, add your edits above here - Exports can take a lot of time...
# You can even comment this entire section and exit the script if you dont feel the need to monitor the download, it will keep downloading in the background even without the script running.
# This is only monitoring if the process exists, which means if you run multiple exports, this will stay running until they all complete.
# We could possibly utilize sysinternals handle.exe to identify the PID of the process writing to the $Exportlocation and monitor for that specifically, but I'm trying to limit external applications in this example script.
#
# Just an FYI, the export progress is how much data Microsoft has copied into PSTs from the compliance search, not how much the export tool has downloaded.
# We only know the actual size of the download after the $ExportProgress is 100% and $ExportStatus is Completed
# The actual final size of the download is then reflected in $ExportTransferred. Even then, our progress is still a bit inaccurate due to the extra log and temp files created locally, which will probably cause the progress to show over 100%
# We could make this a bit more accurate by just collecting the size of PSTs and files under the OneDrive folder, but I think this brings us close enough for most situations.
while(Get-Process microsoft.office.client.discovery.unifiedexporttool -ErrorAction SilentlyContinue){
$Downloaded = Get-ChildItem $ExportLocation\$SearchName -Recurse | Measure-Object -Property Length -Sum | Select-Object -ExpandProperty Sum
Write-Progress -Id 1 -Activity "Export in Progress" -Status "Complete..." -PercentComplete $ExportProgress
if ("Completed" -notlike $ExportStatus){Write-Progress -Id 2 -Activity "Download in Progress" -Status "Estimated Complete..." -PercentComplete ($Downloaded/$ExportEstSize*100) -CurrentOperation "$Downloaded/$ExportEstSize bytes downloaded."}
else {Write-Progress -Id 2 -Activity "Download in Progress" -Status "Complete..." -PercentComplete ($Downloaded/$ExportEstSize*100) -CurrentOperation "$Downloaded/$ExportTransferred bytes downloaded."}
Start-Sleep 60
$ExportDetails = Get-ComplianceSearchAction -Identity $ExportName -IncludeCredential -Details # Get details for the export action
$ExportDetails = $ExportDetails.Results.split(";")
$ExportEstSize = ($ExportDetails[18].TrimStart(" Total estimated bytes: ") -as [double])
$ExportTransferred = ($ExportDetails[20].TrimStart(" Total transferred bytes: ") -as [double])
$ExportProgress = $ExportDetails[22].TrimStart(" Progress: ").TrimEnd("%")
$ExportStatus = $ExportDetails[25].TrimStart(" Export status: ")
Write-Host -NoNewline " ."
}
Write-Host "Download Complete!"
pause
The nice part about this is the unified export tool includes it's own wait loop that checks the status of the export, so you don't have to create your own and worry about credential timeout, etc. Thanks to this, you can kick off the download and move on to other tasks in your off-boarding script.
In production, I've written this to execute the download on a separate terminal using Invoke-Command and email the team a transcript when the download's done.
Drawbacks:
The only way you know it's running is the process shows up in task manager.
The only way you know it's done is it's not in task manager. We can estimate how far along it is, but it isn't perfectly accurate. You definitely want to rely on the trace files to know if everything went OK.
If the export gets stopped for some reason (lost internet, restart due to windows updates, etc.) it does not resume the download when started again. I haven't been able to figure out how to get that to work.
Anyways, I wanted to give back to the thread that helped me get this far.
Is this still working for you? Have you made any changes/improvements since you posted this?
How'd you get the container URL?
Thanks!
- JChupOct 18, 2018Brass Contributor
Yes, this still worked quite well as of a week ago. I can only see this breaking if Microsoft changes the format of the "Results" property of the content search or splits it into separate properties of their own.
I haven't made any real improvements to it, but this reminds me that I should probably combine my code with Brad's so that there's a minimally functional example. I'll edit my previous post at some time for that.
The container URL comes from the "$exportdetails =" line.
It first gets the compliance search, selects the Results property that contains the container URL, then parses the URL and token from the property using the $exporttemplate
The results property is just one giant string of information, instead of separate properties of their own, so parsing it with Convert-FromString was the easiest way I could think of to get those properties.
Convert-FromString requires a template consisting of at least two lines of data, formatted so that it knows what you're looking to parse. The data in $exporttemplate is just two of our old searches, modified with the parsing tags. Sorry if that's not incredibly clear.
- Tom AgueroOct 18, 2018Copper Contributor
Awesome, I got it working and I figured out a way to monitor the process along with the percent complete. Sorry, I'm not sure how to do the fancy formatting for this post.
#Do while microsoft.office.client.discovery.unifiedexporttool.exe running
$started = $false
Do { $status = Get-Process microsoft.office.client.discovery.unifiedexporttool -ErrorAction SilentlyContinue
If (!($status)) {Write-Host 'Waiting for process to start' ; Start-Sleep -Seconds 5 }
Else {
Write-Host 'Process has started' ; $started = $true
}
}Until ( $started )
Do{
$ProcessesFound = Get-Process | ? {$_.Name -like "*unifiedexporttool*"}
If ($ProcessesFound) { $Progress = get-ComplianceSearchAction -Identity $exportname -IncludeCredential -Details | select -ExpandProperty Results | ConvertFrom-String -TemplateContent $exporttemplate | %{$_.Progress}
Write-Host "Export still downloading, progress is $Progress, waiting 60 seconds"
Start-Sleep -s 60}
}Until (!$ProcessesFound)
- Edgar GuerreroApr 12, 2019Copper Contributor
Tom Aguero Can you share the whole script to download pst physically from EXO mailboxes