Forum Discussion
Curious_Kevin16
Sep 03, 2022Brass Contributor
Upload a large list of folders in to AzureStorage area using AZCopy/Powershell
Hi Guys,
I posted this in Azure space but probably its more relevant to PowerShell hence re-posted here 🙂
Newbie to Azure/PowerShell and trying to figure how to script this task. I have a large list of folders that are named after username of people in the orgnisation (see below). I'd like to upload selected set of these folder at a given time (say 100) and that 100 will be based on a CSV file which contains the folder name (username).
Root folder path "C:\SourceFolders\"
Inside each folder there are files but no 2nd level folders. I'd like to use a CSV as the source reference to select the folders to upload (say User2FirstName.LastName, User3FirstName.LastName). Basically, although the folder root location has a large number of folders, only the ones defined in the CSV will be uploaded (serial or parellel)
.\azcopy copy "C:\SourceFolders\User1FirstName.LastName" "https://4980835d12c748f391037a5.blob.core.windows.net/ingestiondata?sv=2015-04-05&sr=c&si=IngestionSasForAzCopy202111179750&sig=uWp%2FEoJ%2Fyl3%2BGuvGsH49WRfjFzQvpI%3D&se=2022-08-31T14%3A15%3A52Z" --recursive=true
Using AZCopy tool and PowerShell, I'd be grateful if someone please can help me to create a script to run this as a batch and potentially set a single output log file for the whole batch. 🙂
Thanks very much in advance !
Kev
- raindropsdevIron Contributor
Here you go: https://github.com/Raindrops-dev/RAIN-TechCommunityContributions/blob/main/Start-AzCopyBackup.ps1
The script is kept on purpose very simple (so you can learn and develop it further yourself 🙂 ) but I integrated the log to a file in the same folder which will include all the output and the output of azcopy. There is a lot more you can do but this would be a good start.
Small consideration: ensure that you keep the SAS URL protected, ideally in an Azure Keyvault 😉