Forum Discussion

shashankgwl's avatar
shashankgwl
Copper Contributor
May 12, 2023

dataflow taking 3 minutes to run but data insertion in last 10 seconds

Hello,

I have a ADF dataflow ran through a pipeline, this dataflow copies data from Dataverse to azure cosmos DB, there are ONLY 5 records which we are talking about so it's not millions of records, so the azure runtime details shouldn't matter I believe, but it's the default runtime with compute type "general purpose, 4 cores".

 

The problem I'm facing is that whether it's 5 records or 20 records, it's taking static 3 minutes to run, but I'm guessing that all the work happening in the last 10 seconds, because during 90% of the time time when the pipeline is "in progress", there are no records available in Cosmos DB, but the moment the pipeline finishes I see data in COSMOS. 

 

Thanks for your help!!

No RepliesBe the first to reply

Resources