%3CLINGO-SUB%20id%3D%22lingo-sub-1397741%22%20slang%3D%22en-US%22%3ERe%3A%20New%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1397741%22%20slang%3D%22en-US%22%3E%3CP%3EIs%20it%20still%20in%20preview%3F%20I%20just%20try%20it%2C%20but%20I%20don't%20see%20the%20run%20consumption%20button.%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1397976%22%20slang%3D%22en-US%22%3ERe%3A%20New%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1397976%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F670611%22%20target%3D%22_blank%22%3E%40edjaja-hicv%3C%2FA%3E%26nbsp%3BYou%20should%20see%20the%20feature%20light-up%20on%20your%20ADF%20UI%20next%20week%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1394671%22%20slang%3D%22en-US%22%3ENew%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1394671%22%20slang%3D%22en-US%22%3E%3CP%3EData%20Factory%20adds%20a%20new%20easy%20way%20to%20view%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fplan-manage-costs%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eestimated%20consumption%20of%20your%20pipelines%3C%2FA%3E.%20Use%20it%20to%20estimate%20the%20number%20of%20units%20consumed%20by%20activities%20while%20debugging%20your%20pipeline%20and%20post-execution%20runs.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWhile%20designing%20and%20debugging%20your%20pipelines%2C%20click%20on%20the%20%22View%20debug%20run%20consumption%22%20link%20in%20the%20bottom%20monitoring%20output.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22consumption2.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F192253iFBCF8B5B8C4CA658%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22consumption2.png%22%20alt%3D%22consumption2.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EADF%20will%20now%20pop-up%20a%20report%20showing%20the%20number%20of%20units%20consumed%20for%20each%20activity%20in%20that%20pipeline.%20You%20can%20then%20click%20the%20Pricing%20Calculator%20link%20and%20plug%20in%20the%20consumed%20units%20to%20the%20pricing%20calculator%20to%20estimate%20your%20usage%20for%20that%20run.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22consumption1.png%22%20style%3D%22width%3A%20480px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F192250i24F81624AB2B5C2A%2Fimage-dimensions%2F480x500%3Fv%3D1.0%22%20width%3D%22480%22%20height%3D%22500%22%20title%3D%22consumption1.png%22%20alt%3D%22consumption1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EIf%20you%20are%20in%20design%20or%20debug%20mode%2C%20you%20should%20note%20the%20data%20sizes%20and%20volumes%20that%20you%20are%20using%20for%20your%20testing.%20You%20can%20then%20extrapolate%20the%20units%20consumed%20to%20the%20production%20workload%20sizes%20that%20you%20will%20use%20for%20scheduled%20pipelines.%20You%20can%20perform%20this%20from%20your%20Git%20branch%20without%20publishing%20your%20factory.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20ADF%20consumption%20report%20is%20only%20surfacing%20ADF%20related%20units.%20There%20may%20be%20additional%20units%20billed%20from%20other%20services%20that%20you%20are%20using%20and%20accessing%20which%20are%20not%20accounted%20for%20here%20including%20Azure%20SQL%20Database%2C%20Synapse%20Analytics%2C%20CosmosDB%2C%20ADLS%2C%20etc.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20you%20have%20completed%20debugging%2C%20you%20can%20then%20view%20the%20estimated%20consumption%20from%20a%20triggered%20run.%20Before%20setting%20a%20schedule%20for%20the%20pipeline%2C%20you%20can%20now%20publish%20your%20pipeline%20and%20test%20it%20with%20larger%20datasets.%20After%20publishing%2C%20click%20Trigger%20Now.%20The%20results%20of%20the%20pipeline%20execution%20will%20appear%20in%20the%20ADF%20monitoring%20pane.%20There%2C%20you%20can%20click%20on%20the%20consumption%20report%20icon%20to%20view%20the%20units%20consumed%20for%20that%20triggered%20execution.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22monitor-consumption-1.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F192251i5455F9E07C39422D%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22monitor-consumption-1.png%22%20alt%3D%22monitor-consumption-1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22monitor-consumption-2.png%22%20style%3D%22width%3A%20894px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F192252i11CB7DD0621F1EF7%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22monitor-consumption-2.png%22%20alt%3D%22monitor-consumption-2.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20you%20are%20satisfied%20with%20your%20pipeline%20and%20apply%20an%20operational%20schedule%20to%20trigger%20your%20pipeline%2C%20you%20can%20view%20the%20consumption%20report%20again%20from%20the%20monitoring%20view.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1394671%22%20slang%3D%22en-US%22%3E%3CP%3EData%20Factory%20adds%20new%20easy%20way%20to%20view%20estimated%20consumption%20of%20your%20pipelines.%20Use%20it%20to%20estimate%20the%20number%20of%20units%20consumed%20by%20activities%20while%20debugging%20your%20pipeline%20and%20post-execution%20runs.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22consumption1.png%22%20style%3D%22width%3A%20633px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F192250i24F81624AB2B5C2A%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22consumption1.png%22%20alt%3D%22consumption1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1394671%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1446144%22%20slang%3D%22en-US%22%3ERe%3A%20New%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1446144%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20is%20great%2C%20thanks!%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1498049%22%20slang%3D%22en-US%22%3ERe%3A%20New%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1498049%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F175989%22%20target%3D%22_blank%22%3E%40Mark%20Kromer%3C%2FA%3E%26nbsp%3BIs%20there%20also%20an%20API%20available%20for%20this%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1509344%22%20slang%3D%22en-US%22%3ERe%3A%20New%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1509344%22%20slang%3D%22en-US%22%3E%3CP%3EDoes%20the%20Consumption%20Report%20show%20the%20overall%20consumption%20for%20all%20nested%20pipelines%20within%20a%20master%20pipeline%3F%20Does%20the%20data%20flow%20consumption%20include%20TTL%3F%20It%20will%20also%20be%20beneficial%20to%20be%20able%20to%20see%20the%20count%20of%20IR%20clusters%20that%20spun%20up%20for%20a%20pipeline%20that%20includes%20data%20flows%20in%20nested%20pipelines%20or%20in%20a%20loop.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1607076%22%20slang%3D%22en-US%22%3ERe%3A%20New%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1607076%22%20slang%3D%22en-US%22%3E%3CP%3EI%20notice%20that%20there%20is%20a%20notification%20upon%20the%20Report%3A%26nbsp%3B%22There%20will%20be%20additional%20charges%20for%20the%20total%20time%20of%20your%20debug%20session.%22%20What%20does%20it%20mean%3F%20Because%20when%20I%20use%20other%20activities%2C%20there%20will%20be%20notifications%20like%20this%20one%2C%20will%20it%20charge%20more%20regardless%20of%20normal%20activity%20expenses%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1607077%22%20slang%3D%22en-US%22%3ERe%3A%20New%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1607077%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F768510%22%20target%3D%22_blank%22%3E%40NewAzure618%3C%2FA%3E%26nbsp%3BThat%20note%20is%20referring%20to%20the%20total%20%22debug%20session%20time%22%2C%20which%20is%20not%20indicated%20in%20the%20consumption%20report%20output.%20The%20consumption%20report%20is%20intended%20to%20give%20you%20an%20estimate%20of%20how%20many%20ADF%20units%20will%20be%20billed%20for%20that%20execution.%20But%20if%20you%20still%20have%20your%20debug%20session%20running%2C%20you%20will%20be%20billed%20for%20the%20entirety%20of%20the%20time%20that%20your%20debug%20session%20is%20active%2C%20which%20is%20not%20indicated%20in%20the%20consumption%20report%20estimator.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Microsoft

Data Factory adds a new easy way to view estimated consumption of your pipelines. Use it to estimate the number of units consumed by activities while debugging your pipeline and post-execution runs.

 

While designing and debugging your pipelines, click on the "View debug run consumption" link in the bottom monitoring output.

 

consumption2.png

 

ADF will now pop-up a report showing the number of units consumed for each activity in that pipeline. You can then click the Pricing Calculator link and plug in the consumed units to the pricing calculator to estimate your usage for that run.

 

consumption1.png

If you are in design or debug mode, you should note the data sizes and volumes that you are using for your testing. You can then extrapolate the units consumed to the production workload sizes that you will use for scheduled pipelines. You can perform this from your Git branch without publishing your factory.

 

The ADF consumption report is only surfacing ADF related units. There may be additional units billed from other services that you are using and accessing which are not accounted for here including Azure SQL Database, Synapse Analytics, CosmosDB, ADLS, etc.

 

Once you have completed debugging, you can then view the estimated consumption from a triggered run. Before setting a schedule for the pipeline, you can now publish your pipeline and test it with larger datasets. After publishing, click Trigger Now. The results of the pipeline execution will appear in the ADF monitoring pane. There, you can click on the consumption report icon to view the units consumed for that triggered execution.

 

monitor-consumption-1.png

 

monitor-consumption-2.png

 

Once you are satisfied with your pipeline and apply an operational schedule to trigger your pipeline, you can view the consumption report again from the monitoring view.

 

7 Comments
Occasional Visitor

Is it still in preview? I just try it, but I don't see the run consumption button. 

Microsoft

@edjaja-hicv You should see the feature light-up on your ADF UI next week

New Contributor

This is great, thanks! 

Visitor

@Mark Kromer Is there also an API available for this?

Occasional Visitor

Does the Consumption Report show the overall consumption for all nested pipelines within a master pipeline? Does the data flow consumption include TTL? It will also be beneficial to be able to see the count of IR clusters that spun up for a pipeline that includes data flows in nested pipelines or in a loop.

Microsoft

I notice that there is a notification upon the Report: "There will be additional charges for the total time of your debug session." What does it mean? Because when I use other activities, there will be notifications like this one, will it charge more regardless of normal activity expenses?

Microsoft

@NewAzure618 That note is referring to the total "debug session time", which is not indicated in the consumption report output. The consumption report is intended to give you an estimate of how many ADF units will be billed for that execution. But if you still have your debug session running, you will be billed for the entirety of the time that your debug session is active, which is not indicated in the consumption report estimator.