Introduction
In this article, we will build together a pipeline that will allow us to send a dataset as an input to a PUT request using the Copy activity.
Main Idea
In this example, we will modify a sample JSON document and send a change via a PUT request. To observe how this works, we will change the title in the JSON string (https://jsonplaceholder.typicode.com/posts/1) where the ID is 1.
{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nostrum rerum est autem sunt rem eveniet architecto"
}
Pre-requisites
You will require:
- Basic knowledge of ADF, including how to create a new pipeline and add activities to a pipeline, etc.
- How to use JsonPlaceHolder API, for the sake of the demo that we will build, we will execute a PUT request.
please read carefully "Updating a resource" in the documentation here: https://jsonplaceholder.typicode.com/guide/
Services:
You will need:
- An Azure Storage Account (Blob storage):
You will need to create a blob storage account. In this demo, I created an account and saved the metadata.json file in the blob storage. - ADF:
Added a linked dataset that points to the file (metadata.json) we created in the Blob Linked Service connection with the ADF instance; this will be our dynamic dataset.
dataset file name: metadata.json
Read more here:
- Datasets - Azure Data Factory & Azure Synapse | Microsoft Docs
- Azure Blob Storage documentation | Microsoft Docs
ADF Pipeline
- Drag a Copy Activity to the pipeline.
- Specify the path to the JSON document that we saved in the Blob storage as a dataset like so:
- Copy Activity: Add the above dataset as a source, in Sink:
Sink DataSet :
New -> Rest -> Create new linkedService like so:
- Sink Request Method: PUT
Output:
Here we can see that the request was successfully executed, and the JSON object is modified.
Updated Aug 15, 2022
Version 1.0Sally_Dabbah
Microsoft
Joined July 10, 2022
FastTrack for Azure
Follow this blog board to get notified when there's new activity