Moving data from SAP API to Rest API using ADF

[Rdadheech]

Hi Team,

i am asking few questions about ADF which i have created but need suggestion for improvement

Requirement: Using ADF for transferring records from Source (SAP ECC) to Destination (Rest API)

Approach used : 

1. Create Copy Activity with appropriate connector and applied proper mapping of fields , but surprised even output of pipeline showing records are written on destination , but there were no records refer below SS. [ Any comment or suggestion for this issue ]

2. In next approach i created a new pipeline with below chaining components :

      A. Web Activity [for fetching records from SAP ECC connector ]  which returns many fields but few of needed for next processing 

      B. Store output of A into Variable (Array) using set variable 

      C. Then add Foreach activity to process each record of B one by one through another Web Activity (inside for each loop) although destination api have option to put many records in one request. 

Now problem area is that second approach take so long time as it iterates for each element of array, but in my case , it would need to be speedup (in case of need to process 20000 records Ex.) 

Constraints need to follow : Without saving data records anywhere like BLOB storage .

                                 Any suggestion for overcoming this problem.

[#2]

@Rdadheech 

Finally, i got the solution by implementing pipeline with help of several activities.Just posting here so that it can be help others if have same requirement (i just temporary stored data on blob storage and deleted on execution of pipelines get completed) 

1. Copy Data activity - Fetch and store data into blob storage (SINK) in CSV format (not json because csv option provide way of specifying max rows which help to store files in chunks of data inside a temporary created folder ex: Input_1234 that can be operated easily by metadata activity as used in next steps )
2. Used metadata activity with child items value in fields list option for fetching detail of CSV files as generated by step1.
3. Used foreach loop on basis on copy activity output.

   @if(greater(activity('Copyactivity1').output.rowsCopied,0) , activity('MetadataActivity').output.childItems,variables('varEmptyArr'))
   
   *varEmptyArr is a declared variable contain blank value , used in case of no record received form source

1. Inside foreach i linked step3 to new copy activity for converting CSV data into Json and restored in blob storage on different folder (ex: output_1234)
2. Then next activity added after step4 is lookup, which pick records from Json file and that can be used in next Web activity body option.
3. Add Web activity for posting data into rest Api, used output of step5 inside body option.
4. Use delete activity for deleting first Json file whose records has been pushed to rest Api in step 6
5. Outside of foreach finally again used deleted activity for deleting csv all files from blob.
6. It's done , but used couple of variables for holding values of parameters which are up to you how to use it in pipelines.
7. So finally, i could say that using data factory you can transfer data from one SAP API/REST api to another restapi . :)