Forum Discussion
Foreach activity parallel runs turns into serial run from parallel after an hour+
For each activity parallel tasks are scheduled at the beginning of the execution. This means that if a task in a queue take longer will delay the whole queue, and even if there is slots available in other queue ADF will not re-shuffle the tasks
Ok. practically the runtime of activities inside 'foreach' like COPY/LOAD depends on various factors like
size of data, stuff around target database on that day here etc, but existing design choice in adf scheduling in 'foreach' do gives performance impact in production, do we see it is reasonable to flag product improvement needed in this area? if so whom/where to loop this?
Here is chart from today prod run (foreach): if scheduling is dynamic and aggressive job would have finished around 20 minutes earlier though in today's run. some days it goes for hour because of this.
Total scheduled task inside 'foreach' = 30 and how it breaks it up,
Queue1: Total task ran = 11
Queue2: Total task ran = 6
Queue3: Total task ran = 7
Queue4: Total task ran = 5
Queue5: Total task ran = 2