How To Send Dataset Dynamically to A REST Service Using ADF using Copy Activity

[Sally_Dabbah]

Introduction

In this article, we will build together a pipeline that will allow us to send a dataset as an input to a PUT request using the Copy activity. 

Main Idea

In this example, we will modify a sample JSON document and send a change via a PUT request.  To observe how this works, we will change the title in the JSON string (https://jsonplaceholder.typicode.com/posts/1) where the ID is 1.

{
    "userId": 1,
    "id": 1,
    "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
    "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nostrum rerum est autem sunt rem eveniet architecto"
}

Pre-requisites

You will require:

• Basic knowledge of ADF, including how to create a new pipeline and add activities to a pipeline, etc.
• How to use JsonPlaceHolder API, for the sake of the demo that we will build, we will execute a PUT request.
  please read carefully "Updating a resource" in the documentation here:  https://jsonplaceholder.typicode.com/guide/
  
  

Services:
You will need:

• An Azure Storage Account (Blob storage):
  You will need to create a blob storage account. In this demo, I created an account and saved the metadata.json file in the blob storage. 
• ADF:
  Added a linked dataset that points to the file (metadata.json) we created in the Blob Linked Service connection with the ADF instance; this will be our dynamic dataset.
  dataset file name: metadata.json

Read more here:

• Datasets - Azure Data Factory & Azure Synapse | Microsoft Docs
• Azure Blob Storage documentation | Microsoft Docs  

ADF Pipeline

• Drag a Copy Activity to the pipeline.
• Specify the path to the JSON document that we saved in the Blob storage as a dataset like so:

•  Copy Activity: Add the above dataset as a source, in Sink:
  Sink DataSet : 
  New -> Rest -> Create new linkedService like so:
  
  

  

• Sink Request Method: PUT
  

 Output:
Here we can see that the request was successfully executed, and the JSON object is modified.