Welcome to Azure Data Factory’s November monthly update! Here we’ll share the latest updates on What’s New in Azure Data Factory. You can find all our updates here.
We’ll also be hosting our monthly livestream next week on December 15th at 9:00am PST/ 12:00pm EST! Join us to see some live demos and to ask us your ADF questions!
Join the livestream here.
Table of Contents
Continuous Integration and continuous deployment (CI/CD)
Data flow
Data movement
- Azure Databricks now supported through private link from a Data Factory managed virtual network [Doc]
Developer Productivity
Continuous integration and continuous delivery (CI/CD)
Disable publish button in ADF studio now available
We have added the ability to disable the publish button from the ADF studio when source control is enabled. Once this is enabled, the Publish button will be grayed out in the ADF Studio. This will help to avoid overwriting the last automated publish deployment.
To learn more about this update, read Source control - Azure Data Factory | Microsoft Learn.
Data flow
New improvements to SAP Change Data Capture (CDC)
We added some new updates to the SAP CDC connector in mapping data flows, adding new capabilities:
Incremental only is now available
You can get changes only from SAP system without initial full load. With this feature, you have the flexibility to focus on incremental data process without worrying about big data volume movement on an initial full data load.
Performance improvements have been made - source partitions in initial full data load
With a simple radio button, you can now connect your SAP system to multiple sessions in parallel to extract data, which will greatly reduce the time to perform an initial full data load from your SAP system.
Pipeline template added to Template Gallery
There is a new pipeline template in the Template Gallery to help speed up your journey on loading big amounts of SAP objects by a parameterized pipeline with SAP CDC enabled in ADF. You can get more information on the template here.
To learn more about this, read Transform data from an SAP ODP source with the SAP CDC connector in Azure Data Factory or Azure Syna....
Data movement
Azure Databricks now supported through private link from a Data Factory managed virtual network
Azure Databricks is now a supported data service that can be connected to through a private link from a Data Factory managed virtual network.
To learn more about this update, read Managed virtual network and managed private endpoints - Azure Data Factory | Microsoft Learn.
Developer Productivity
Pipeline designer enhancements added to ADF Studio preview experience
We have added three new UI updates to the ADF Studio preview experience:
Dynamic Content Flyout
A new dynamic content flyout has been added to make it easier to set dynamic content in your pipeline activities without having to use the expression builder.
Error Messaging
Error messages have been relocated to the Status column in Debug and Pipeline monitoring. This will make it easier to view errors whenever a pipeline run fails.
Container view for monitoring
A new monitoring view is available, adding the option to enable the container view. This view provides a more consolidated monitoring view of activities that were run. A big change that you will see is that nested activities are now grouped under the parent activity. To be able to view more activities on one page, you can choose to hide the list of nested activities under the parent activity.
To learn more about these enhancements, read Managing Azure Data Factory studio preview experience - Azure Data Factory | Microsoft Learn.
We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in our blogs or livestreams. We love hearing your feedback!