Azure Data Factory October 2022 Monthly Update
Published Nov 04 2022 11:48 AM 3,217 Views
Microsoft

Welcome to Azure Data Factory’s October monthly update! Here we’ll share the latest updates on what’s new in Azure Data Factory. You can also find all our updates on our What's New in ADF page. 

 

We’ll also be hosting our monthly livestream next week on November 10th at 10:30am PST/ 1:30pm EST! Join us to see some live demos and to ask us your ADF questions! 

 

Join the livestream here 


 

Table of Contents 

Continuous Integration and continuous deployment (CI/CD)

Data flow 

 Data movement 

 Developer productivity

Orchestration 

Region expansion 

 

Continuous integration and continuous delivery (CI/CD)

 

Exclude pipeline triggers that did not change in deployments – now generally available 

We’ve improved the CI/CD process by only requiring the triggers that are changed or updated to be stopped and started during deployments using the updated PowerShell. It will help reduce overall deployment time and the need to stop all triggers during deployment. 

 

Noelle_Li_9-1667416331332.png

 

To learn more about this update, read CI/CD trigger deployment improvement in ADF (microsoft.com). 

 

Data flow 

 

Export up to 1000 rows from data flow preview 

When designing a mapping data flow, users can view their data at each step using Data preview. Currently, the Data preview pane is limited to a view containing 100 rows, but by using the Export CSV option, users will be able to view up to 1000 rows from the transformation results.   

 

To learn more about this, read Mapping data flow Debug Mode - Azure Data Factory & Azure Synapse | Microsoft Learn. 

 

SQL CDC in Mapping Data Flows now available (Public Preview) 

Azure Data Factory has expanded its native change data capture capabilities to include SQL Server, Azure SQL Database, and Azure SQL Managed Instance (MI) connectors. Changed data in SQL stores can be automatically detected and extracted by ADF mapping data flows. 

 

To learn more about how you leverage these new native CDC capabilities, read Copy and transform data to and from SQL Server - Azure Data Factory & Azure Synapse | Microsoft Lear.... 

 

Unlock advanced analytics with Microsoft 365 Mapping Data Flow Connector (Preview) 

We have launched a limited public preview for a new M365 Mapping Data Flows Connector in Azure Data Factory and Azure Synapse Analytics. With this connector, you will be able to quickly clean, normalize, and flatten your data from parquet to multiple data sinks, reducing costs and accelerating analytics. A new pipeline template is also available to simplify the configuration experience, saving time and effort. 

 

Noelle_Li_10-1667416748006.png

 

 

To read more about this, read Scale access to Microsoft 365 data with Microsoft Graph Data Connect - Microsoft 365 Developer Blog. 

 

Data movement 

 

SAP Change Data Capture (CDC) connector now generally available 

The SAP CDC connector is now generally available in Azure Data Factory and Azure Synapse Analytics, providing a simple, integrated experience to extract and transform the raw change data coming from a variety of SAP source systems, including SAP ECC, SAP S/4HANA, SAP BW, and SAP BW/4HANA.  

 

To read more information about this update, read Transform data from an SAP ODP source with the SAP CDC connector in Azure Data Factory or Azure Syna... 

 

Developer Productivity

 

Learning Center now available in the Azure Data Factory Studio 

We've launched the Azure Data Factory (ADF) Learning Center, a new central location where users can find learning resources like the latest features updates, getting started tutorials, best practice videos, training modules, product links, and more. With the Learning Center, we aim to reduce the learning curve and empower each user to achieve success with self-directed learning that is integrated into the ADF studio. 

 

Noelle_Li_16-1667417103604.png

 

For more information, read Introducing the Learning Center to Azure Data Factory Studio (microsoft.com).

 

Now accepting community contributions to Template Gallery  

We are now accepting pipeline template submissions from the community, allowing you the opportunity to contribute to the existing Azure Data Factory pipeline template gallery.  

 

Noelle_Li_14-1667417050435.png

 

To learn more about how to contribute, read Introducing Azure Data Factory Community Templates (microsoft.com).

 

New design in Azure portal – easily discover how to launch ADF Studio 

Based on user feedback, we have re-designed the user interface (UI) for launching of the ADF Studio to make it easier for users to discover how to launch the ADF studio from the Azure portal. Users will also be able to locate training content and access quick starts and tutorials before jumping into the studio.   

 

Noelle_Li_15-1667417080851.png

 

For more information on this update, read Improved UI for Launching Azure Data Factory Studio (microsoft.com). 

 

Updated quick-start experience - one-click to try Azure Data Factory

We’ve updated our new quick start experience to make it easier to try Azure Data Factory. Now, users can get started quickly with one-click and create a new data factory and pipeline with sample data to explore.

 

Noelle_Li_0-1667585399099.png

 

 

To learn more and try it out, read Get started and try out your first data factory pipeline - Azure Data Factory | Microsoft Learn.

 

Orchestration 

 

Granular billing view available for ADF – see detailed billing information by pipeline (Public Preview) 

To provide more clarity and transparency, a granular billing view is now available for Azure Data Factory pipelines. With the built-in per pipeline detailed billing view, users can see exactly how much each pipeline costs to operate. By opting into the feature, a separate line item will be shown for each pipeline and charges associated with a pipeline will be grouped together under the pipeline name, giving a clear view of the cost of operations. Aggregate views for factories are also available, as charges can be filtered by factory name in the Azure billing report.  

 

Noelle_Li_11-1667416873431.png

 

To learn more about this, read Granular Billing for Azure Data Factory (microsoft.com).

 

Script activity execution timeout now configurable 

You can now configure the wait time for a script block execution operation to complete before the it times out. Previously this was set to 2 hours, but now, it can be personalized to a value that you set.   

 

Noelle_Li_13-1667417013296.png

 

To learn more, read Transform data by using the Script activity - Azure Data Factory & Azure Synapse | Microsoft Learn. 

 

Region expansion 

 

Continued region expansion – Qatar Central now supported 

Azure Data Factory is now available in Qatar Central. You can co-locate your ETL workflow in this new region if you are utilizing the region for storing and managing your modern data warehouse. 

 

Noelle_Li_0-1667532005540.png

 

To learn more about supported regions, read Azure Products by Region | Microsoft Azure. 

 

 

We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in our blogs or livestreams. We love hearing your feedback!

 

Co-Authors
Version history
Last update:
‎Nov 04 2022 11:43 AM
Updated by: