Data Copy 32gb backup

Copper Contributor

We get a nightly backup from a 3rd party provider that is 32gb in size. Our current process is to download that and get changes to update our warehouse. We are currently testing out moving to Azure and have noticed that the Copy Data step is extremely slow for moving this 32gb file from our 3rd party's SFTP to our Storage Account in Azure where we then do a restore to this database in Azure. I've played around with changing settings in the integration runtimes of the source and sink but haven't had luck in speeding this up.

I have tested using AZCopy in Powershell works much faster but that requires downloading it from the SFTP to a local server and then up to Azure as I couldn't find a way to use AZCopy to go from the SFTP directly to our storage account.

 

Any suggestions for moving this large backup file from an SFTP over to Storage Account in order to restore in Azure?

0 Replies