Welcome to Azure Data Factory’s October monthly update! Here we’ll share the latest updates on what’s new in Azure Data Factory. You can also find all our updates on our What's New in ADF page.
We’ll also be hosting our monthly livestream next week on November 10th at 10:30am PST/ 1:30pm EST! Join us to see some live demos and to ask us your ADF questions!
Join the livestream here.
We’ve improved the CI/CD process by only requiring the triggers that are changed or updated to be stopped and started during deployments using the updated PowerShell. It will help reduce overall deployment time and the need to stop all triggers during deployment.
To learn more about this update, read CI/CD trigger deployment improvement in ADF (microsoft.com).
When designing a mapping data flow, users can view their data at each step using Data preview. Currently, the Data preview pane is limited to a view containing 100 rows, but by using the Export CSV option, users will be able to view up to 1000 rows from the transformation results.
To learn more about this, read Mapping data flow Debug Mode - Azure Data Factory & Azure Synapse | Microsoft Learn.
Azure Data Factory has expanded its native change data capture capabilities to include SQL Server, Azure SQL Database, and Azure SQL Managed Instance (MI) connectors. Changed data in SQL stores can be automatically detected and extracted by ADF mapping data flows.
To learn more about how you leverage these new native CDC capabilities, read Copy and transform data to and from SQL Server - Azure Data Factory & Azure Synapse | Microsoft Lear....
We have launched a limited public preview for a new M365 Mapping Data Flows Connector in Azure Data Factory and Azure Synapse Analytics. With this connector, you will be able to quickly clean, normalize, and flatten your data from parquet to multiple data sinks, reducing costs and accelerating analytics. A new pipeline template is also available to simplify the configuration experience, saving time and effort.
To read more about this, read Scale access to Microsoft 365 data with Microsoft Graph Data Connect - Microsoft 365 Developer Blog.
The SAP CDC connector is now generally available in Azure Data Factory and Azure Synapse Analytics, providing a simple, integrated experience to extract and transform the raw change data coming from a variety of SAP source systems, including SAP ECC, SAP S/4HANA, SAP BW, and SAP BW/4HANA.
To read more information about this update, read Transform data from an SAP ODP source with the SAP CDC connector in Azure Data Factory or Azure Syna...
We've launched the Azure Data Factory (ADF) Learning Center, a new central location where users can find learning resources like the latest features updates, getting started tutorials, best practice videos, training modules, product links, and more. With the Learning Center, we aim to reduce the learning curve and empower each user to achieve success with self-directed learning that is integrated into the ADF studio.
For more information, read Introducing the Learning Center to Azure Data Factory Studio (microsoft.com).
We are now accepting pipeline template submissions from the community, allowing you the opportunity to contribute to the existing Azure Data Factory pipeline template gallery.
To learn more about how to contribute, read Introducing Azure Data Factory Community Templates (microsoft.com).
Based on user feedback, we have re-designed the user interface (UI) for launching of the ADF Studio to make it easier for users to discover how to launch the ADF studio from the Azure portal. Users will also be able to locate training content and access quick starts and tutorials before jumping into the studio.
For more information on this update, read Improved UI for Launching Azure Data Factory Studio (microsoft.com).
We’ve updated our new quick start experience to make it easier to try Azure Data Factory. Now, users can get started quickly with one-click and create a new data factory and pipeline with sample data to explore.
To learn more and try it out, read Get started and try out your first data factory pipeline - Azure Data Factory | Microsoft Learn.
To provide more clarity and transparency, a granular billing view is now available for Azure Data Factory pipelines. With the built-in per pipeline detailed billing view, users can see exactly how much each pipeline costs to operate. By opting into the feature, a separate line item will be shown for each pipeline and charges associated with a pipeline will be grouped together under the pipeline name, giving a clear view of the cost of operations. Aggregate views for factories are also available, as charges can be filtered by factory name in the Azure billing report.
To learn more about this, read Granular Billing for Azure Data Factory (microsoft.com).
You can now configure the wait time for a script block execution operation to complete before the it times out. Previously this was set to 2 hours, but now, it can be personalized to a value that you set.
To learn more, read Transform data by using the Script activity - Azure Data Factory & Azure Synapse | Microsoft Learn.
Azure Data Factory is now available in Qatar Central. You can co-locate your ETL workflow in this new region if you are utilizing the region for storing and managing your modern data warehouse.
To learn more about supported regions, read Azure Products by Region | Microsoft Azure.
We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in our blogs or livestreams. We love hearing your feedback!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.