ADF Data Flows Script Intro: Copy, Paste, Snippets

ADF Data Flows has a low-code graph-based UI to design and develop data transformations at scale. But the script behind those graphs is very powerful. This article introduces you to the Data Flow Script and how to use it to copy, paste, and share data transformation code.

Below is a sample data flow where I have a fairly complex Aggregate transformation that I wish to use in another data flow. To do that, I'm going to click on the Script button the ADF Data Flow design UI to view the script behind.

Details of Data Flow Script are here at the ADF doc page.

Each data flow starts with a source transformation. Each subsequent transformation in the script will start with the name of the incoming stream. We want to share the aggregate called SummaryStats. To do that, click Script, then Ctrl-F to enter into Find/Replace. Search for "SummaryStats". To share this transformation, we copy from the aggregate() function to the end of the expression, just before the transformation name. The name of the transformation uses the syntax ~> and then the name of transformation. When I share this, I do not copy the transformation name, assuming that I can set a new name when I paste it.

Here is the code that gets copied:

aggregate(each(match(true()), $$+'_NotNull' = countIf(!isNull($$)), $$ + '_Null' = countIf(isNull($$))),each(match(type=='double'||type=='integer'||type=='short'||type=='decimal'), $$+'_stddev' = round(stddev($$),2), $$ + '_min' = min ($$), $$ + '_max' = max($$), $$ + '_average' = round(avg($$),2), $$ + '_variance' = round(variance($$),2)), each(match(type=='string'), $$+'_maxLength' = max(length($$))))

This includes all of my expression function language work, so I can share that code with other data flows. I could also use the script-behind to rename my transformations using find/replace.

I've started including shareable data flow script snippets in the script docs here. You can copy those snippet blocks and paste them into your data flows. Watch the video above toward the end for an example of how to use the code snippets.