Azure Data Factory July 2022 Monthly Update

Published Aug 02 2022 10:46 PM 1,341 Views
Microsoft

Welcome to Azure Data Factory’s July monthly update! Here, we’ll share the latest updates on what’s new in Azure Data Factory. You can also find all our updates in our What's New in ADF page

 

We’ll also be hosting our monthly livestream next week on August 9th at 4pm PST! Join us to see some live demos and to ask us your ADF questions! 

Join the livestream here.

 

 

Table of Contents

Continuous Integration/Continuous Delivery

Data flow

 

Continuous Integration/Continuous Delivery

CI/CD improvement by supporting Global Parameters

We’ve added a new mechanism to include Global Parameters in the ARM templates. This helps to solve an earlier issue, which overrode some configurations during deployments when users included global parameters in ARM templates.

Noelle_Li_0-1659408796133.png

 

To read more information on this update, read CICD improvements while deploying global parameters (microsoft.com).

 

Data flow

Asana REST-based connector added as Source [Preview]

We’ve added a new REST-based connector to mapping data flows! Users can now read their tables from Asana. Note: This connector is only available when using inline datasets.

Noelle_Li_1-1659408796138.png

 

To read more information on the Asana connector [Preview], read Transform data in Asana (Preview) - Azure Data Factory & Azure Synapse | Microsoft Docs.

 

3 new dataflow transformation functions added

3 new data transformation functions have been added to mapping data flows in Azure Data Factory and Azure Synapse Analytics. Now, users are able to use collectUnique(), to create a new collection of unique values in an array, substringIndex(), to extract the substring before n occurrences of a delimiter, and topN(), to return the top n results after sorting your data.

 

To learn more, read 3 New Data Transformation Functions in ADF (microsoft.com).

 

Refetch preview data from source

When building and debugging a data flow, your source data can change. There is now a new easy way to refetch the latest updated source data from the refresh button in the data preview pane.

Noelle_Li_2-1659408796147.png

To learn more, read Mapping data flow Debug Mode.

 

User defined functions [Generally Available]

With the new user-defined functions feature, you can create customized expressions that can be reused across multiple mapping data flows. This will keep you from having to building complex logic over and over. User-defined functions will be grouped in libraries to help developers group common sets of functions.  

 

Once you’ve created a data flow library, you can add in your user-defined functions. You can even add in multiple arguments to make your function more reusable. 

 

Noelle_Li_3-1659408796153.png

 

 

To learn more about user-defined functions, read User defined functions in mapping data flows.

 

Easier configuration of data flow runtime with preset compute sizes

Azure Data Factory has made it easier for users to configure Azure Integration Runtime for mapping data flows by adding new pre-configured categories for Spark Compute. You can still set your own custom configurations.

 

Noelle_Li_4-1659408796156.png

 

For more information on this UI update, read ADF Makes it Easy to Select Azure IR Size for Data Flows (microsoft.com).

 

We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in these blogs or livestreams. We love hearing your feedback!

 

Co-Authors
Version history
Last update:
‎Aug 08 2022 03:48 PM
Updated by: