Welcome to Azure Data Factory’s first monthly update blog! Here we’ll share the latest updates on what’s new in Azure Data Factory. You can also find all of our updates at What's new in Azure Data Factory - Azure Data Factory | Microsoft Docs.
We’ll also be hosting our monthly livestream on Wednesday July 20th at 1:30 PM EST! Let us know what’d you like to see and join us to see live demos and to ask us your ADF questions!
Join the livestream here.
Table of Contents
Data flow
- Assert error handling
- Editing capabilities in source projection
- Fuzzy Join option in Join Transformation
Data movement
- Parameterization natively supported in additional 4 connectors
- SAP Change Data Capture (CDC) in new SAP ODP connector [Public Preview]
Integration runtime
Monitoring
Orchestration
User interface
Data flow
Assert Error Handling
Error handling has now been added to sinks following an assert transformation. Assert transformations enable you to build custom rules for data quality and data validation and now, you can log rows that fail your own custom assertions!
This example shows which rows failed your assertions:
To learn more about error handling, read Assert data transformation in mapping data flow.
Editing capabilities in source projection
New UI updates have been made to source projection editing in mapping data flows. You can now update source projection column names and column types with the click of a button.
To learn more about source projection editing, read Source transformation in mapping data flow.
Fuzzy Join option in Join Transformation
Fuzzy matching with a sliding similarity score option has been added to the Join transformation in Mapping Data Flows. You can create inner and outer joins on data values that are similar rather than exact matches! The sliding scale value goes from 60% to 100%, making it easy to adjust the similarity threshold of the match.
To learn more about fuzzy joins, read Join transformation in mapping data flow.
Data Movement
Parameterization natively supported in 4 additional connectors
We’ve added native UI support of parameterization for the following linked services:
- Azure Data Lake Gen1
- Azure PostgreSQL
- Google AdWords
- PostgreSQL
To learn more, read Parameterize linked services - Azure Data Factory & Azure Synapse | Microsoft Docs.
SAP Change Data Capture (CDC) in new SAP ODP connector [Public Preview]
SAP change data capture (CDC) in ADF is now in public preview! The new SAP ODP data connector can connect to all SAP systems that support ODP and can be used with predefined templates to help streamline the integration of SAP data within Azure services.
To learn more, see Announcing the Public Preview of the SAP CDC solution in Azure Data Factory (microsoft.com).
Integration Runtime
Time-To-Live in managed VNET [Public Preview]
We’ve introduced the public preview of time-to-live (TTL) in managed virtual network. With this, users can reserve computes for the TTL period, saving time and improving efficiency.
To learn more, read Announcing public preview of Time-To-Live (TTL) in managed virtual network - Microsoft Tech Community.
Monitoring
Rerun pipeline with new parameters
You can now change pipeline parameters when re-running a pipeline from the Monitoring page without having to return to the pipeline editor. After running a pipeline with new parameters, you can easily monitor the new run against the old ones without having to toggle between pages.
Note that by re-running a pipeline with new parameters, it will be considered a new pipeline run and will not show under re-run groupings.
To learn more about rerunning pipelines with new parameters, read Rerun pipelines and activities.
Orchestration
‘turnOffAsync' property available in Web activity
Web activity supports an async request-reply pattern that invokes HTTP GET on the Location field in the response header of an HTTP 202 Response. It helps web activity automatically poll the monitoring end-point till the job runs. ‘turnOffAsync' property is now supported to disable this behavior in cases where polling isn't needed.
To learn more, read Web Activity - Azure Data Factory & Azure Synapse | Microsoft Docs.
User Interface (UI)
Studio preview experience
If you are interested in trying new UI updates, then the Azure Data Factory studio preview experience is for you! When you enable the studio preview experience, you’ll be able to test UI changes in the ADF pipeline and mapping data flow experience and provide feedback to the team.
To learn more about the experience, read Managing Azure Data Factory Studio preview updates - Azure Data Factory | Microsoft Docs.
Let us know what your favorites are in the comments below!
Microsoft
