Azure Data Factory from AWS S3 to Azure Blob Copy

%3CLINGO-SUB%20id%3D%22lingo-sub-178229%22%20slang%3D%22en-US%22%3EAzure%20Data%20Factory%20from%20AWS%20S3%20to%20Azure%20Blob%20Copy%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-178229%22%20slang%3D%22en-US%22%3E%3CP%3EFrom%20AWS%20S3%20to%20Azure%20Blob%20Is%20there%20any%20way%20data%20movement%20allows%20us%20to%20control%20the%20container%20names%20dynamically%20change.%20and%20is%20there%20any%20custom%20code%20can%20be%20executed%20during%20the%20copy.%20Example%20%3A%20AWS%20S3%20bucket%20Container%20Name%20-%20day01222012%20in%20Azure%20blob%20container%20should%20change%20automatically.%20using%20Azure%20Data%20Factory.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-178229%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-178525%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Data%20Factory%20from%20AWS%20S3%20to%20Azure%20Blob%20Copy%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-178525%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Gowrish%2C%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou%20can%20use%20ADFv2%20to%20reach%20your%20goal.%3C%2FP%3E%0A%3CP%3EYour%20solution%20might%20look%20like%20this%3A%3C%2FP%3E%0A%3CP%3E1.%20Create%20list%20of%20AWS%20S3%20links%20to%20import%20as%20file%3C%2FP%3E%0A%3CP%3E2.%20Generate%20file%20on%20AWS%20side%20and%20download%20it%20to%20Azure%20blob%20daily%3C%2FP%3E%0A%3CP%3E3.%20Create%20lookup%20activity%20for%20the%20file%20contents%3C%2FP%3E%0A%3CP%3E4.%20For%20each%20line%20-%20invoke%20copy%20activity%20with%20source%20url%20and%20generate%20destination%20folder%2Ffile%20name%20in%20ADF%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E
Deleted
Not applicable

From AWS S3 to Azure Blob Is there any way data movement allows us to control the container names dynamically change. and is there any custom code can be executed during the copy. Example : AWS S3 bucket Container Name - day01222012 in Azure blob container should change automatically. using Azure Data Factory.

1 Reply

Hi Gowrish,

 

You can use ADFv2 to reach your goal.

Your solution might look like this:

1. Create list of AWS S3 links to import as file

2. Generate file on AWS side and download it to Azure blob daily

3. Create lookup activity for the file contents

4. For each line - invoke copy activity with source url and generate destination folder/file name in ADF