Azure Data Factory August 2022 Monthly Update
Published Sep 12 2022 01:54 PM 2,766 Views
Microsoft

Welcome to Azure Data Factory’s August monthly update! Here, we’ll share the latest updates on what’s new in Azure Data Factory. You can also find all our updates in our What's New in ADF page

 

We’ll also be hosting our monthly livestream THIS week on September 15th at 11am PST / 2pm EST! Join us to see some live demos and to ask us your ADF questions! 

 

Join the livestream here.

 

 

Table of Contents

Continuous integration and continuous delivery (CI/CD)

Data flow

Data movement

User interface

 

Continuous integration and continuous delivery

 

Exclude pipeline triggers that did not change in deployments

We’ve improved the CI/CD process by only requiring the triggers that are changed or updated that will be stopped and started during deployments using the updated PowerShell. This will help reduce overall deployment time and the need to stop all triggers during deployment!

 

Noelle_Li_0-1662767163226.png

 

To learn more about this update, read CI/CD trigger deployment improvement in ADF (microsoft.com).

 

Data flow

Appfigures REST-based connector added (Preview)

We’ve added a new REST-based connector to mapping data flows! Users can now read their tables from Appfigures. Note: This connector is only available when using inline datasets.

 

Noelle_Li_1-1662767163234.png

 

 

For more information, read Transform data in AppFigures (Preview) - Azure Data Factory & Azure Synapse | Microsoft Docs.

 

Cast transformation added – visually convert data types

It is very common to transform data types into data integration workloads. For example, you might need to convert strings to dates. Previously to do this in mapping data flows, you had to use Expression Builder and remember the correct syntax or keywords to convert data types in mapping data flows.

 

For example, the syntax to convert a string ‘Jul-01-2022’ to date 2022-07-01 is toDate(‘Jul-01-2022’, ‘MMM-dd-yyyy’, ‘en-US’).

 

Now, you can use the cast transformation to quickly transform data types visually!

 

Noelle_Li_2-1662767163237.png

 

To learn more about this transformation in mapping dataflows, read Cast transformation in mapping data flow.

 

New UI for inline datasets - categories added to easily find data sources

We’ve updated our data flow source UI to make it easier to find your inline dataset type. Previously, you would have to scroll through the list or filter to find your inline dataset type. Now, we have categories that group your dataset types, making it easier to find what you’re looking for.

 

Noelle_Li_3-1662767163241.png

 

To learn more about this update, read Mapping data flows - Inline Datasets.

 

Data movement

Default activity time-out changed from 7 days to 12 hours

We heard your feedback, and it was clear that the default pipeline activity timeout of 7 days was too long! While you can still customize the timeout, the default activity time out for Azure Data Factory and Azure Synapse Analytics is now 12 hours.

 

Noelle_Li_4-1662767163243.png

 

 

To learn more about this change, read Azure Data Factory Changing Default Pipeline Activity.

 

Service principal authentication type added for Blob storage

Now, users can use the Service Principal authentication type for Azure Blob Storage! This allows you to use Azure role-based access control (Azure RBAC) to grant permissions to a security principal, which may be a user, group, or application service principal, who can then authenticate with Azure AD to authorize a request.

 

Noelle_Li_0-1663008609583.png

 

To learn more, read Copy and transform data in Azure Blob storage - Azure Data Factory & Azure Synapse | Microsoft Docs.

 

 

User interface

New data factory creation experience - 1 click to have your factory ready within seconds

A new user experience is ready in the landing page of adf.azure.com or the "Switch Data Factory" tab of Data Factory Studio portal to enable users to create a Data Factory with only one click! All you need is a factory name, location, and subscription and the Data Factory created will be available for use within seconds. The new creation experience will not only empower new users to quickly use and learn more about Data Factory, but also let experienced users easily create a new factory for testing. 

 

Noelle_Li_5-1662767163288.png

 

To learn more about this new experience, read New Experience for Creating Data Factory within Seconds (microsoft.com).

 

Expression builder UI update - categorical tabs added for easier use

We’ve updated our expression builder UI to make pipeline designing easier. Our pipeline expression builder used to list out all components in a single pane.

 

Now, to make it easier to work with different pipeline components, we’ve created new content category tabs to make it easier to find what you’re looking for.

 

Noelle_Li_0-1663015609175.png

 

To learn more, read More Pipeline Expression Builder Ease-of-Use Enhancements!

 

We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in these blogs or livestreams. We love hearing your feedback!

 

2 Comments
Co-Authors
Version history
Last update:
‎Sep 15 2022 12:18 PM
Updated by: