How to calculate the cost of copy data from AzureBlob to Azure Data Explorer with Azure Data Factory

Published 07-08-2019 03:32 PM 2,947 Views

Azure Data Factory is a fully managed cloud-based data integration service. You can use the service to populate your Azure Data Explorer database with data from various locations and save time when building your analytics solutions. To estimate the total time and E2E cost to ingest Azure blob using Azure Data Factory to Azure Data Explorer, see below:

 

  • Azure Data Explorer estimated cost: please use ADX calculator http://aka.ms/adx.cost to gain insight of the cluster size and cost.
  • Learn about ADF pricing. Here is some guidelines to estimate ADF cost:
  • The general speed equation of ADF is XdataMB/YMBps/3600s = Yh.
    The speed to copy Azure Blob to Azure Data Explorer cluster using a single copy activity is around 11MB/s speed. 
  • To achieve greater throughput, you can run multiple copy activities in parallel and achieve 11MBps * M in aggregated throughput.
    Therefore, copy duration for copying Azure Blob to Azure Data Explorer using ADF is XdataMB/11MBps/Z/3600s = Yh
  • Cost equation for cloud copy by default with 4 DIUs (Data Integration Units) and $0.25 per DIU per hour, copy cost estimation: Yh * Z*$0.25 * 4 DIU = $$$.

 

Example of ADF costs:

A customer wants to upload one blob a day with 250 GB of CSV data.

250GB * 1024 / 11 MB / 3600 = 6.5h

6.5* 0.25$ * 4 DIUs = 6.5$ per 250 GB.

 

Costs may include:

  • Additional orchestration and operation cost (usually much less than data movement).
  • Failure/retries.

Those numbers can be impacted by the following list:

  • Locations of your data/ADF/ADX
    • Reduce cost by having source, ADF, and target cluster in the same region.
  • ADX cluster – number of nodes (used the smaller cluster of 2 nodes)
  • Number of DIUs (4~32 DIUs)
    • Additional DIUs are helpful when more than one file/blob copied in parallel
  • Data format (used CSV)

Azure Data Factory billing Deck:

https://microsoft.sharepoint.com/:p:/t/admsteam/EbV6yJMo4NZBnoNgZUsV88QBsQqcPpv9Fk0WTdgUPfMkoA?e=grV...

2 Comments
Occasional Visitor

can any one help me how to calculate DIUs (Data Integration Units) in Azure data factory?

in ADF, i have 60 pipeline created and each pipeline runs every 3 mins in a day. i am seeing resource cost is high for the month.  pipe line runs is more than 18000 times. From pipeline monitor, i am seeing data moment in DIU but can not search for 18000 times.. so looking how to calculate DIU?

@Vasu1983 it seems that you are asking a general question on ADF, so please redirect your question to ADF channel 

%3CLINGO-SUB%20id%3D%22lingo-sub-743780%22%20slang%3D%22en-US%22%3EHow%20to%20calculate%20the%20cost%20of%20copy%20data%20from%20AzureBlob%20to%20Azure%20Data%20Explorer%20with%20Azure%20Data%20Factory%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-743780%22%20slang%3D%22en-US%22%3E%3CP%3EAzure%20Data%20Factory%20is%20a%20fully%20managed%20cloud-based%20data%20integration%20service.%20You%20can%20use%20the%20service%20to%20populate%20your%20Azure%20Data%20Explorer%20database%20with%20data%20from%20various%20locations%20and%20save%20time%20when%20building%20your%20analytics%20solutions.%20To%20estimate%20the%20total%20time%20and%20E2E%20cost%20to%20ingest%20Azure%20blob%20using%20Azure%20Data%20Factory%20to%20Azure%20Data%20Explorer%2C%20see%20below%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3EAzure%20Data%20Explorer%20estimated%20cost%3A%20please%20use%20ADX%20calculator%20%3CA%20href%3D%22http%3A%2F%2Faka.ms%2Fadx.cost%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttp%3A%2F%2Faka.ms%2Fadx.cost%3C%2FA%3Eto%20gain%20insight%20of%20the%20cluster%20size%20and%20cost.%3C%2FLI%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3ELearn%20about%20%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fazure.microsoft.com%252Fen-us%252Fpricing%252Fdetails%252Fdata-factory%252Fdata-pipeline%252F%26amp%3Bdata%3D02%257C01%257Ctzgitlin%2540microsoft.com%257Ca877908fb93c42da510a08d703b29689%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C636981937895998743%26amp%3Bsdata%3D0ZgNPVGNn6iXq6e9GYoXe%252FkscF3UsERlXEStdDprgW8%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EADF%20pricing%3C%2FA%3E.%20Here%20is%20some%20guidelines%20to%20estimate%20ADF%20cost%3A%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CUL%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3EThe%20general%20speed%20equation%20of%20ADF%20is%20XdataMB%2FYMBps%2F3600s%20%3D%20Yh.%20%3CBR%20%2F%3EThe%20speed%20to%20copy%20Azure%20Blob%20to%20Azure%20Data%20Explorer%20cluster%20using%20a%20single%20copy%20activity%20is%20around%2011MB%2Fs%20speed.%26nbsp%3B%3C%2FLI%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3ETo%20achieve%20greater%20throughput%2C%20you%20can%20run%20multiple%20copy%20activities%20in%20parallel%20and%20achieve%2011MBps%20*%20M%20in%20aggregated%20throughput.%3CBR%20%2F%3ETherefore%2C%20copy%20duration%20for%20copying%20Azure%20Blob%20to%20Azure%20Data%20Explorer%20using%20ADF%20is%20XdataMB%2F%3CSTRONG%3E11%3C%2FSTRONG%3EMBps%2FZ%2F3600s%20%3D%20%3CSTRONG%3EY%3C%2FSTRONG%3Eh%3C%2FLI%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3ECost%20equation%20for%20cloud%20copy%20by%20default%20with%204%20DIUs%20(Data%20Integration%20Units)%20and%20%240.25%20per%20DIU%20per%20hour%2C%20copy%20cost%20estimation%3A%20%3CSTRONG%3EY%3C%2FSTRONG%3Eh%20*%20Z*%240.25%20*%204%20DIU%20%3D%20%3CSTRONG%3E%24%24%24%3C%2FSTRONG%3E.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EExample%20of%20ADF%20costs%3A%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EA%20customer%20wants%20to%20upload%20one%20blob%20a%20day%20with%20250%20GB%20of%20CSV%20data.%3C%2FP%3E%0A%3CP%3E250GB%20*%201024%20%2F%2011%20MB%20%2F%203600%20%3D%206.5h%3C%2FP%3E%0A%3CP%3E6.5*%200.25%24%20*%204%20DIUs%20%3D%206.5%24%20per%20250%20GB.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ECosts%20may%20include%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3EAdditional%20orchestration%20and%20operation%20cost%20(usually%20much%20less%20than%20data%20movement).%3C%2FLI%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3EFailure%2Fretries.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EThose%20numbers%20can%20be%20impacted%20by%20the%20following%20list%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3ELocations%20of%20your%20data%2FADF%2FADX%3C%2FLI%3E%0A%3CUL%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3EReduce%20cost%20by%20having%20source%2C%20ADF%2C%20and%20target%20cluster%20in%20the%20same%20region.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3EADX%20cluster%20%E2%80%93%20number%20of%20nodes%20(used%20the%20smaller%20cluster%20of%202%20nodes)%3C%2FLI%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3ENumber%20of%20DIUs%20(4~32%20DIUs)%3C%2FLI%3E%0A%3CUL%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3EAdditional%20DIUs%20are%20helpful%20when%20more%20than%20one%20file%2Fblob%20copied%20in%20parallel%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CLI%20style%3D%22font-weight%3A%20400%3B%22%3EData%20format%20(used%20CSV)%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EAzure%20Data%20Factory%20billing%20Deck%3A%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2Fap%2Fp-59584e83%2F%3Furl%3Dhttps%253A%252F%252Fmicrosoft.sharepoint.com%252F%253Ap%253A%252Ft%252Fadmsteam%252FEbV6yJMo4NZBnoNgZUsV88QBsQqcPpv9Fk0WTdgUPfMkoA%253Fe%253DgrV1Qf%26amp%3Bdata%3D02%257C01%257Ctzgitlin%2540microsoft.com%257C1277c410d24846f26bc308d70fdfddeb%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C636995326499291488%26amp%3Bsdata%3D%252B2dD7NMg078HAR8K8ZRTOWbcASf9eIR9GVha5%252FptT4o%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fmicrosoft.sharepoint.com%2F%3Ap%3A%2Ft%2Fadmsteam%2FEbV6yJMo4NZBnoNgZUsV88QBsQqcPpv9Fk0WTdgUPfMkoA%3Fe%3DgrV1Qf%3C%2FA%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-743780%22%20slang%3D%22en-US%22%3E%3CP%3EAzure%20Data%20Factory%20is%20a%20fully%20managed%20cloud-based%20data%20integration%20service.%20You%20can%20use%20the%20service%20to%20populate%20your%20Azure%20Data%20Explorer%20database%20with%20data%20from%20various%20locations%20and%20save%20time%20when%20building%20your%20analytics%20solutions.%20To%20estimate%20the%20total%20time%20and%20E2E%20cost%20to%20ingest%20Azure%20blob%20using%20Azure%20Data%20Factory%20to%20Azure%20Data%20Explorer%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F122635i0EC0C360A6DF9852%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20alt%3D%22ADX_Tips.png%22%20title%3D%22ADX_Tips.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2047943%22%20slang%3D%22en-US%22%3ERe%3A%20How%20to%20calculate%20the%20cost%20of%20copy%20data%20from%20AzureBlob%20to%20Azure%20Data%20Explorer%20with%20Azure%20Data%20Fac%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2047943%22%20slang%3D%22en-US%22%3E%3CP%3Ecan%20any%20one%20help%20me%20how%20to%20calculate%26nbsp%3B%3CSPAN%3EDIUs%20(Data%20Integration%20Units)%20in%20Azure%20data%20factory%3F%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3Ein%20ADF%2C%20i%20have%2060%20pipeline%20created%20and%20each%20pipeline%20runs%20every%203%20mins%20in%20a%20day.%20i%20am%20seeing%20resource%20cost%20is%20high%20for%20the%20month.%26nbsp%3B%20pipe%20line%20runs%20is%20more%20than%2018000%20times.%20From%20pipeline%20monitor%2C%20i%20am%20seeing%20data%20moment%20in%20DIU%20but%20can%20not%20search%20for%2018000%20times..%20so%20looking%20how%20to%20calculate%20DIU%3F%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2081604%22%20slang%3D%22en-US%22%3ERe%3A%20How%20to%20calculate%20the%20cost%20of%20copy%20data%20from%20AzureBlob%20to%20Azure%20Data%20Explorer%20with%20Azure%20Data%20Fac%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2081604%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F924405%22%20target%3D%22_blank%22%3E%40Vasu1983%3C%2FA%3E%26nbsp%3Bit%20seems%20that%20you%20are%20asking%20a%20general%20question%20on%20ADF%2C%20so%20please%20redirect%20your%20question%20to%20ADF%20channel%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Jul 24 2019 04:13 AM
Updated by: