ADF - REST API Copy Data Activity - Best Practices

Don't parallelize just because ADF supports it. First, determine whether the API is stateless or stateful.

Case 1: API is Stateless

Every request contains all the information needed to process it. For example:

GET /reports?franchiseId=101&page=1 GET /reports?franchiseId=102&page=1

or:

Header: FranchiseId: 101

Each request is completely independent, so one request doesn't affect another.

Recommended approach:

• Use a parallel ForEach activity.
• Pass the franchise ID using @item().FranchiseId instead of pipeline variables.
• Write each franchise's output to a separate file or folder.
• For better isolation and error handling, consider using a child pipeline per franchise.

If the API and infrastructure can handle the load, this can reduce a 30-minute pipeline to just a few minutes.

Case 2: API is Stateful

The API maintains server-side state. For example:

SetCurrentFranchise(101) RunReport() SetCurrentFranchise(102) RunReport()

Now imagine two parallel executions:

Pipeline A SetCurrentFranchise(101) Pipeline B SetCurrentFranchise(102) Pipeline A RunReport()

Pipeline A expects data for franchise 101, but instead receives data for 102 because Pipeline B changed the server's current franchise.

In this scenario, parallel execution can produce incorrect or inconsistent results.

Recommended approach:

• Process requests sequentially.
• Avoid shared pipeline variables or shared server-side state.
• The bottleneck is the API design, not Azure Data Factory.

Conclusion

Before deciding whether to use parallelism, determine whether the API is stateless or stateful.