Forum Discussion
Need ADF pipeline suggestion
I have an ADF pipeline that copies Files from source to destination. Both Source and destinations are different folders within adls only. My pipeline design is as follows 1.) Lookup activity- A ...
If the batch count controls parallel execution, have you tried increasing it to 30? I believe ADF allows up to 50 concurrent executions.
Additionally, if there is a pattern in the file names, you could split them into chunks and run multiple pipelines simultaneously instead of relying solely on batch count for parallelism.
It's unclear whether the 1 million files are new each time or not. If they are not, consider copying only the new or updated files based on the modified date