Blog Post

Azure Data Factory Blog
2 MIN READ

New ADF pipeline consumption report

Mark Kromer's avatar
Mark Kromer
Icon for Microsoft rankMicrosoft
May 16, 2020

Data Factory adds a new easy way to view estimated consumption of your pipelines. Use it to estimate the number of units consumed by activities while debugging your pipeline and post-execution runs.

 

While designing and debugging your pipelines, click on the "View debug run consumption" link in the bottom monitoring output.

 

 

ADF will now pop-up a report showing the number of units consumed for each activity in that pipeline. You can then click the Pricing Calculator link and plug in the consumed units to the pricing calculator to estimate your usage for that run.

 

If you are in design or debug mode, you should note the data sizes and volumes that you are using for your testing. You can then extrapolate the units consumed to the production workload sizes that you will use for scheduled pipelines. You can perform this from your Git branch without publishing your factory.

 

The ADF consumption report is only surfacing ADF related units. There may be additional units billed from other services that you are using and accessing which are not accounted for here including Azure SQL Database, Synapse Analytics, CosmosDB, ADLS, etc.

 

Once you have completed debugging, you can then view the estimated consumption from a triggered run. Before setting a schedule for the pipeline, you can now publish your pipeline and test it with larger datasets. After publishing, click Trigger Now. The results of the pipeline execution will appear in the ADF monitoring pane. There, you can click on the consumption report icon to view the units consumed for that triggered execution.

 

 

 

Once you are satisfied with your pipeline and apply an operational schedule to trigger your pipeline, you can view the consumption report again from the monitoring view.

 

Updated May 20, 2020
Version 2.0
  • NewAzure618 That note is referring to the total "debug session time", which is not indicated in the consumption report output. The consumption report is intended to give you an estimate of how many ADF units will be billed for that execution. But if you still have your debug session running, you will be billed for the entirety of the time that your debug session is active, which is not indicated in the consumption report estimator.

  • Marks4Java's avatar
    Marks4Java
    Copper Contributor

    How can I get that Pipeline Run Consumption data for all pipeline runs over a period of time? Using the GUI for all of that is tedious. Have been exploring the PowerShell Commands for ADF V2, and still not seeing how I could get all of that data. Thanks

     

  • edjaja-hicv's avatar
    edjaja-hicv
    Copper Contributor

    Is it still in preview? I just try it, but I don't see the run consumption button. 

  • azdev's avatar
    azdev
    Copper Contributor

    Does the Consumption Report show the overall consumption for all nested pipelines within a master pipeline? Does the data flow consumption include TTL? It will also be beneficial to be able to see the count of IR clusters that spun up for a pipeline that includes data flows in nested pipelines or in a loop.

  • I notice that there is a notification upon the Report: "There will be additional charges for the total time of your debug session." What does it mean? Because when I use other activities, there will be notifications like this one, will it charge more regardless of normal activity expenses?

  • This is great thank you however many customer require this breakdown in the billing portal. Perhaps this could be implemented via tags (bit like annotations) at the pipeline level which could be used to breakdown costs in the billing page?