Many times we have a requirement to migrate azure storage tables from one storage account to another. For a small number of tables, you can choose the following options:
For storage accounts containing large number of tables, we can also use Azure Data Factory (ADF). This blog demostrates how we can use ADF for copying storage tables across 2 storage accounts.
In this blob, we will be moving storage tables from a source account to destination storage account. In the source storage account we currently have 10 tables:
These tables are populated with some sample data:
In the destination, there are no tables.
As we need to make REST API calls to both the storage accounts, we will be using SAS tokens to authorize the calls.
We will create a service SAS token for accessing both storage accounts. You can fine tune the permissions if required. I will be creating SAS using following permissions for both accounts:
We will now create an Azure Data Factory (ADF) resource. If you are new to ADF, please refer this link on how to create one: https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-portal#create-a-d...
Once created we can access the Azure Data Factory UI page using the link on Overview as shown below:
We will create our pipeline that we can use to setup our copy process.
We will add two parameters in this pipeline, to save our SAS tokens. We can use these parameters in the pipeline. Once expired, we can update the parameter values.
Now that the pipeline is created we will add the activities for the copy process. The flow for this process is as follows:
We will use a Web Activity for this operation. Using this activity we will make a REST API call to storage service with SAS token to fetch the list of tables. The API being used is Query Tables.
In the URL we are providing the source storage endpoint as per the API specification, concatenating with the sourceSAS parameter we saved earlier.
@concat('https://SOURCE.table.core.windows.net/Tables', pipeline().parameters.sourceSAS)
Once configured, we can publish the changes.
Add a ForEach activity, which will be used to loop through list of tables received from REST API.
In the Items setting, we will use the output of the previous activity for this loop. We have added ".value' as this is property that contains the array of tables returned by REST API.
The rest of the activities will be created inside ForEach activity.
We will use another Web Activity to call the Create Table REST API on the destination storage account. Configure the Web Activity according to Create Table REST API specification.
We are passing the table name in Body dynamically using the 'item' object from ForEach activity.
The next item is 'Copy data' activity, which will peform the data copy between both storage accounts. Copy data activity will require a Dataset and Linked Service to connect to storage account. For datasets, click on on 'New' for both source & sink(destination) and select 'Azure Table Storage' to continue. Provide a relevant name.
You can follow the steps shared in this documents to create linked service for both of your storage accounts: https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-table-storage?tabs=data-factory#...
Once you have created the datasets & linked service for both source & sink account, we need to configure the datasets to pass the Table names dynamically. Use the "item" object of ForEach activity to get the table name, for both source and destination(sink) datasets as shown below:
Create a parameter in both datasets:
Use the parameter as table name in dataset:
Pass the current item to the dataset from Copy data activity:
Tip: You can further optimize your ADF to use storage account name, SAS tokens etc. dynamically using parameters and variables. To learn more on the same refer this documentation: https://azure.microsoft.com/en-in/resources/azure-data-factory-passing-parameters/
We can now run the activity to start the copy process. The pipeline will complete depending on the number of tables and data in the source storage account.
We can see that tables are now created in the destination storage account:
And the data is copied too.
Tip: You can also add a Delete Table API request after the Copy data activity to delete the table in source storage account.
Update:
To copy source tables PartitionKey and RowKey, use the below configuration in Copy Data activity:
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.