New ADF pipeline consumption report
Published May 15 2020 09:09 PM 9,920 Views
Microsoft

Data Factory adds a new easy way to view estimated consumption of your pipelines. Use it to estimate the number of units consumed by activities while debugging your pipeline and post-execution runs.

 

While designing and debugging your pipelines, click on the "View debug run consumption" link in the bottom monitoring output.

 

consumption2.png

 

ADF will now pop-up a report showing the number of units consumed for each activity in that pipeline. You can then click the Pricing Calculator link and plug in the consumed units to the pricing calculator to estimate your usage for that run.

 

consumption1.png

If you are in design or debug mode, you should note the data sizes and volumes that you are using for your testing. You can then extrapolate the units consumed to the production workload sizes that you will use for scheduled pipelines. You can perform this from your Git branch without publishing your factory.

 

The ADF consumption report is only surfacing ADF related units. There may be additional units billed from other services that you are using and accessing which are not accounted for here including Azure SQL Database, Synapse Analytics, CosmosDB, ADLS, etc.

 

Once you have completed debugging, you can then view the estimated consumption from a triggered run. Before setting a schedule for the pipeline, you can now publish your pipeline and test it with larger datasets. After publishing, click Trigger Now. The results of the pipeline execution will appear in the ADF monitoring pane. There, you can click on the consumption report icon to view the units consumed for that triggered execution.

 

monitor-consumption-1.png

 

monitor-consumption-2.png

 

Once you are satisfied with your pipeline and apply an operational schedule to trigger your pipeline, you can view the consumption report again from the monitoring view.

 

9 Comments
Copper Contributor

Is it still in preview? I just try it, but I don't see the run consumption button. 

Microsoft

@edjaja-hicv You should see the feature light-up on your ADF UI next week

Brass Contributor

This is great, thanks! 

Copper Contributor

@Mark Kromer Is there also an API available for this?

Copper Contributor

Does the Consumption Report show the overall consumption for all nested pipelines within a master pipeline? Does the data flow consumption include TTL? It will also be beneficial to be able to see the count of IR clusters that spun up for a pipeline that includes data flows in nested pipelines or in a loop.

Microsoft

I notice that there is a notification upon the Report: "There will be additional charges for the total time of your debug session." What does it mean? Because when I use other activities, there will be notifications like this one, will it charge more regardless of normal activity expenses?

Microsoft

@NewAzure618 That note is referring to the total "debug session time", which is not indicated in the consumption report output. The consumption report is intended to give you an estimate of how many ADF units will be billed for that execution. But if you still have your debug session running, you will be billed for the entirety of the time that your debug session is active, which is not indicated in the consumption report estimator.

Microsoft

This is great thank you however many customer require this breakdown in the billing portal. Perhaps this could be implemented via tags (bit like annotations) at the pipeline level which could be used to breakdown costs in the billing page?

Copper Contributor

How can I get that Pipeline Run Consumption data for all pipeline runs over a period of time? Using the GUI for all of that is tedious. Have been exploring the PowerShell Commands for ADF V2, and still not seeing how I could get all of that data. Thanks

 

Version history
Last update:
‎May 19 2020 09:36 PM
Updated by: