Welcome to Azure Data Factory’s January monthly update! Here we’ll share the latest updates on what’s new in Azure Data Factory. You can find all our updates here.
We’ll also be hosting our monthly livestream next week on February 16th, 2023! Join us to see some live demos and to ask us your ADF questions!
Join the livestream here.
Table of Contents
Data flow
- Flowlets now supports schema drift [Learn more]
- SQL CDC incremental extract now supports numeric columns [Learn more]
Data movement
Orchestration
Region expansion
- Continued region expansion – China North 3 now supported [Learn more]
- Continued region expansion – Switzerland West now supported [Learn more]
SQL Server Integration Services (SSIS)
Developer Productivity
- Added support for favoriting resources on home page
- Save column width for Pipeline monitoring
- Monitor Filter Updates for faster searches
- Directly access Pipeline Template Gallery through Azure portal [Learn more]
Data flow
Flowlets now supports schema drift
Allow schema drift is now supported for flowlets. Flowlets are reusuable activities that can be created from scratch or from an existing mapping dataflow. Now with schema drift support, users can use their flowlets for datasets that have changing source columns, allowing for more flexible flowlet use across many mapping dataflows.
For more information about schema drift, read Schema drift in mapping data flow - Azure Data Factory & Azure Synapse | Microsoft Learn.
SQL CDC incremental extract now supports numeric columns
Enabling incremental extract in SQL sources in mapping dataflows allows for users to only process rows that have changed since the last time that pipeline was executed. Previously when enabling incremental extract, users were only allowed to select a date/time column as a watermark. Now, supported incremental column types include date/time and numeric columns.
For more information about change data capture, read Change Data Capture - Azure Data Factory & Azure Synapse | Microsoft Learn.
Data movement
New top-level CDC resource for easier native CDC configuration
One of the most difficult aspects to building enterprise-ready data integration pipelines is capturing just the latest changed data from disparate sources quickly. Change Data Capture (CDC) is a set of technologies and techniques that enables data engineers to discover changed records as they occur at the source. We are excited to share that a new CDC top-level resource is now available in Public Preview that will allow you to automatically read changes from your data sources in 3 simple steps!
For more information, read Announcing the Public Preview of a new CDC top-level resource in ADF (microsoft.com).
Orchestration
Orchestrate Synapse notebooks and Synapse spark job definitions natively
Azure Data Factory now supports orchestrating Synapse notebooks and Synapse spark job definitions (SJD) natively. This will help users who have invested in both ADF and Synapse Spark without having to switch to Synapse pipelines to orchestrate Synapse notebooks and spark job definitions. This also allows users more flexibility to use Spark SQL and PySpark to perform data transformations, from simple data cleaning tasks to more complex data processing tasks.
To learn more, read Orchestrate and operationalize Synapse Notebooks and Spark Job Definitions from Azure Data Factory (....
Region expansion
Continued region expansion – China North 3 now supported
Azure Data Factory is now available in China North 3. You can co-locate your ETL workflow in this new region if you are utilizing the region for storing and managing your modern data warehouse.
To learn more about supported regions, read Azure Products by Region | Microsoft Azure.
Continued region expansion – Switzerland West now supported
Azure Data Factory is now available in Switzerland West. You can co-locate your ETL workflow in this new region if you are utilizing the region for storing and managing your modern data warehouse.
To learn more about supported regions, read Azure Products by Region | Microsoft Azure.
SQL Server Integration Services (SSIS)
Express virtual network injection for SSIS is now generally available
Express virtual network injection for SSIS is now Generally Available. This method of injecting your SSIS integration runtime into a virtual network allows for your SSIS integration runtime provisions and starts faster. Inbound traffic is also not needed to meet Enterprise security compliance requirements.
To learn more about express virtual network injection for SSIS, read General Availability of Express Virtual Network injection for SSIS in Azure Data Factory.
Developer Productivity
Added support for favoriting resources on home page
In ADF, users can now pin their favorite resources on the ADF home page. This will allow for users to quickly and easily access most frequently used resources.
Save column width for Pipeline monitoring
When users change the width of columns in Pipeline Monitoring, the changes will now be saved. This allows for users to focus on monitoring without having to constantly update their monitoring view.
Monitor Filter updates for faster searches
In Pipeline Monitoring, improvements have been made to support faster filtering so that users can more easily find what they’re searching for. Users can also use autocomplete when searching through triggered pipeline runs to make searching quicker.
Directly access Pipeline Template Gallery through Azure portal
In the Azure portal, users can now directly access the Pipeline template gallery when launching their data factory. This will allow for easy access to the Azure Data Factory studio and help users quickly create pipelines from predefined pipeline scenarios.
To learn more about this, readDirectly Launch ADF Pipeline Template Gallery through the Azure Portal (microsoft.com).
We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in our blogs or livestreams. We love hearing your feedback!