%3CLINGO-SUB%20id%3D%22lingo-sub-1397741%22%20slang%3D%22en-US%22%3ERe%3A%20New%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1397741%22%20slang%3D%22en-US%22%3E%3CP%3EIs%20it%20still%20in%20preview%3F%20I%20just%20try%20it%2C%20but%20I%20don't%20see%20the%20run%20consumption%20button.%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1397976%22%20slang%3D%22en-US%22%3ERe%3A%20New%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1397976%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F670611%22%20target%3D%22_blank%22%3E%40edjaja-hicv%3C%2FA%3E%26nbsp%3BYou%20should%20see%20the%20feature%20light-up%20on%20your%20ADF%20UI%20next%20week%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1394671%22%20slang%3D%22en-US%22%3ENew%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1394671%22%20slang%3D%22en-US%22%3E%3CP%3EData%20Factory%20adds%20a%20new%20easy%20way%20to%20view%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fplan-manage-costs%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eestimated%20consumption%20of%20your%20pipelines%3C%2FA%3E.%20Use%20it%20to%20estimate%20the%20number%20of%20units%20consumed%20by%20activities%20while%20debugging%20your%20pipeline%20and%20post-execution%20runs.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWhile%20designing%20and%20debugging%20your%20pipelines%2C%20click%20on%20the%20%22View%20debug%20run%20consumption%22%20link%20in%20the%20bottom%20monitoring%20output.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22consumption2.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F192253iFBCF8B5B8C4CA658%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22consumption2.png%22%20alt%3D%22consumption2.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EADF%20will%20now%20pop-up%20a%20report%20showing%20the%20number%20of%20units%20consumed%20for%20each%20activity%20in%20that%20pipeline.%20You%20can%20then%20click%20the%20Pricing%20Calculator%20link%20and%20plug%20in%20the%20consumed%20units%20to%20the%20pricing%20calculator%20to%20estimate%20your%20usage%20for%20that%20run.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22consumption1.png%22%20style%3D%22width%3A%20480px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F192250i24F81624AB2B5C2A%2Fimage-dimensions%2F480x500%3Fv%3D1.0%22%20width%3D%22480%22%20height%3D%22500%22%20title%3D%22consumption1.png%22%20alt%3D%22consumption1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EIf%20you%20are%20in%20design%20or%20debug%20mode%2C%20you%20should%20note%20the%20data%20sizes%20and%20volumes%20that%20you%20are%20using%20for%20your%20testing.%20You%20can%20then%20extrapolate%20the%20units%20consumed%20to%20the%20production%20workload%20sizes%20that%20you%20will%20use%20for%20scheduled%20pipelines.%20You%20can%20perform%20this%20from%20your%20Git%20branch%20without%20publishing%20your%20factory.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20ADF%20consumption%20report%20is%20only%20surfacing%20ADF%20related%20units.%20There%20may%20be%20additional%20units%20billed%20from%20other%20services%20that%20you%20are%20using%20and%20accessing%20which%20are%20not%20accounted%20for%20here%20including%20Azure%20SQL%20Database%2C%20Synapse%20Analytics%2C%20CosmosDB%2C%20ADLS%2C%20etc.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20you%20have%20completed%20debugging%2C%20you%20can%20then%20view%20the%20estimated%20consumption%20from%20a%20triggered%20run.%20Before%20setting%20a%20schedule%20for%20the%20pipeline%2C%20you%20can%20now%20publish%20your%20pipeline%20and%20test%20it%20with%20larger%20datasets.%20After%20publishing%2C%20click%20Trigger%20Now.%20The%20results%20of%20the%20pipeline%20execution%20will%20appear%20in%20the%20ADF%20monitoring%20pane.%20There%2C%20you%20can%20click%20on%20the%20consumption%20report%20icon%20to%20view%20the%20units%20consumed%20for%20that%20triggered%20execution.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22monitor-consumption-1.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F192251i5455F9E07C39422D%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22monitor-consumption-1.png%22%20alt%3D%22monitor-consumption-1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22monitor-consumption-2.png%22%20style%3D%22width%3A%20894px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F192252i11CB7DD0621F1EF7%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22monitor-consumption-2.png%22%20alt%3D%22monitor-consumption-2.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20you%20are%20satisfied%20with%20your%20pipeline%20and%20apply%20an%20operational%20schedule%20to%20trigger%20your%20pipeline%2C%20you%20can%20view%20the%20consumption%20report%20again%20from%20the%20monitoring%20view.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1394671%22%20slang%3D%22en-US%22%3E%3CP%3EData%20Factory%20adds%20new%20easy%20way%20to%20view%20estimated%20consumption%20of%20your%20pipelines.%20Use%20it%20to%20estimate%20the%20number%20of%20units%20consumed%20by%20activities%20while%20debugging%20your%20pipeline%20and%20post-execution%20runs.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22consumption1.png%22%20style%3D%22width%3A%20633px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F192250i24F81624AB2B5C2A%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22consumption1.png%22%20alt%3D%22consumption1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1394671%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1446144%22%20slang%3D%22en-US%22%3ERe%3A%20New%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1446144%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20is%20great%2C%20thanks!%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1498049%22%20slang%3D%22en-US%22%3ERe%3A%20New%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1498049%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F175989%22%20target%3D%22_blank%22%3E%40Mark%20Kromer%3C%2FA%3E%26nbsp%3BIs%20there%20also%20an%20API%20available%20for%20this%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1509344%22%20slang%3D%22en-US%22%3ERe%3A%20New%20ADF%20pipeline%20consumption%20report%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1509344%22%20slang%3D%22en-US%22%3E%3CP%3EDoes%20the%20Consumption%20Report%20show%20the%20overall%20consumption%20for%20all%20nested%20pipelines%20within%20a%20master%20pipeline%3F%20Does%20the%20data%20flow%20consumption%20include%20TTL%3F%20It%20will%20also%20be%20beneficial%20to%20be%20able%20to%20see%20the%20count%20of%20IR%20clusters%20that%20spun%20up%20for%20a%20pipeline%20that%20includes%20data%20flows%20in%20nested%20pipelines%20or%20in%20a%20loop.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Microsoft

Data Factory adds a new easy way to view estimated consumption of your pipelines. Use it to estimate the number of units consumed by activities while debugging your pipeline and post-execution runs.

 

While designing and debugging your pipelines, click on the "View debug run consumption" link in the bottom monitoring output.

 

consumption2.png

 

ADF will now pop-up a report showing the number of units consumed for each activity in that pipeline. You can then click the Pricing Calculator link and plug in the consumed units to the pricing calculator to estimate your usage for that run.

 

consumption1.png

If you are in design or debug mode, you should note the data sizes and volumes that you are using for your testing. You can then extrapolate the units consumed to the production workload sizes that you will use for scheduled pipelines. You can perform this from your Git branch without publishing your factory.

 

The ADF consumption report is only surfacing ADF related units. There may be additional units billed from other services that you are using and accessing which are not accounted for here including Azure SQL Database, Synapse Analytics, CosmosDB, ADLS, etc.

 

Once you have completed debugging, you can then view the estimated consumption from a triggered run. Before setting a schedule for the pipeline, you can now publish your pipeline and test it with larger datasets. After publishing, click Trigger Now. The results of the pipeline execution will appear in the ADF monitoring pane. There, you can click on the consumption report icon to view the units consumed for that triggered execution.

 

monitor-consumption-1.png

 

monitor-consumption-2.png

 

Once you are satisfied with your pipeline and apply an operational schedule to trigger your pipeline, you can view the consumption report again from the monitoring view.

 

5 Comments
Occasional Visitor

Is it still in preview? I just try it, but I don't see the run consumption button. 

Microsoft

@edjaja-hicv You should see the feature light-up on your ADF UI next week

New Contributor

This is great, thanks! 

Occasional Visitor

@Mark Kromer Is there also an API available for this?

Occasional Visitor

Does the Consumption Report show the overall consumption for all nested pipelines within a master pipeline? Does the data flow consumption include TTL? It will also be beneficial to be able to see the count of IR clusters that spun up for a pipeline that includes data flows in nested pipelines or in a loop.