Welcome to Azure Data Factory’s July monthly update! Here, we’ll share the latest updates on what’s new in Azure Data Factory. You can also find all our updates in our What's New in ADF page.
We’ll also be hosting our monthly livestream next week on August 9th at 4pm PST! Join us to see some live demos and to ask us your ADF questions!
Join the livestream here.
Table of Contents
Continuous Integration/Continuous Delivery
Data flow
- Asana REST-based connector added as Source [Preview]
- 3 new dataflow transformation functions added
- Refetch preview data from source
- User defined functions [Generally Available]
- Easier configuration of data flow runtime with preset compute sizes
Continuous Integration/Continuous Delivery
CI/CD improvement by supporting Global Parameters
We’ve added a new mechanism to include Global Parameters in the ARM templates. This helps to solve an earlier issue, which overrode some configurations during deployments when users included global parameters in ARM templates.
To read more information on this update, read CICD improvements while deploying global parameters (microsoft.com).
Data flow
Asana REST-based connector added as Source [Preview]
We’ve added a new REST-based connector to mapping data flows! Users can now read their tables from Asana. Note: This connector is only available when using inline datasets.
To read more information on the Asana connector [Preview], read Transform data in Asana (Preview) - Azure Data Factory & Azure Synapse | Microsoft Docs.
3 new dataflow transformation functions added
3 new data transformation functions have been added to mapping data flows in Azure Data Factory and Azure Synapse Analytics. Now, users are able to use collectUnique(), to create a new collection of unique values in an array, substringIndex(), to extract the substring before n occurrences of a delimiter, and topN(), to return the top n results after sorting your data.
To learn more, read 3 New Data Transformation Functions in ADF (microsoft.com).
Refetch preview data from source
When building and debugging a data flow, your source data can change. There is now a new easy way to refetch the latest updated source data from the refresh button in the data preview pane.
To learn more, read Mapping data flow Debug Mode.
User defined functions [Generally Available]
With the new user-defined functions feature, you can create customized expressions that can be reused across multiple mapping data flows. This will keep you from having to building complex logic over and over. User-defined functions will be grouped in libraries to help developers group common sets of functions.
Once you’ve created a data flow library, you can add in your user-defined functions. You can even add in multiple arguments to make your function more reusable.
To learn more about user-defined functions, read User defined functions in mapping data flows.
Easier configuration of data flow runtime with preset compute sizes
Azure Data Factory has made it easier for users to configure Azure Integration Runtime for mapping data flows by adding new pre-configured categories for Spark Compute. You can still set your own custom configurations.
For more information on this UI update, read ADF Makes it Easy to Select Azure IR Size for Data Flows (microsoft.com).
We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in these blogs or livestreams. We love hearing your feedback!