Azure Data Factory November 2022 Monthly Update
Published Dec 02 2022 02:19 PM 2,409 Views
Microsoft

Welcome to Azure Data Factory’s November monthly update! Here we’ll share the latest updates on What’s New in Azure Data Factory. You can find all our updates here.

 

We’ll also be hosting our monthly livestream next week on December 15th at 9:00am PST/ 12:00pm EST! Join us to see some live demos and to ask us your ADF questions! 

 

Join the livestream here.

 

 

Table of Contents

Continuous Integration and continuous deployment (CI/CD)

Data flow

Data movement

Developer Productivity

 

Continuous integration and continuous delivery (CI/CD)

 

Disable publish button in ADF studio now available

We have added the ability to disable the publish button from the ADF studio when source control is enabled. Once this is enabled, the Publish button will be grayed out in the ADF Studio. This will help to avoid overwriting the last automated publish deployment.

 

Noelle_Li_0-1670017635045.png

 

To learn more about this update, read Source control - Azure Data Factory | Microsoft Learn.

 

Data flow

 

New improvements to SAP Change Data Capture (CDC)

We added some new updates to the SAP CDC connector in mapping data flows, adding new capabilities:

Incremental only is now available

You can get changes only from SAP system without initial full load. With this feature, you have the flexibility to focus on incremental data process without worrying about big data volume movement on an initial full data load.

 
 

sap-change-data-capture-mapping-data-flow-run-mode.png

 

Performance improvements have been made - source partitions in initial full data load

With a simple radio button, you can now connect your SAP system to multiple sessions in parallel to extract data, which will greatly reduce the time to perform an initial full data load from your SAP system.

 

sap-change-data-capture-mapping-data-flow-optimize-partition.png

 

Pipeline template added to Template Gallery

There is a new pipeline template in the Template Gallery to help speed up your journey on loading big amounts of SAP objects by a parameterized pipeline with SAP CDC enabled in ADF. You can get more information on the template here.

 

To learn more about this, read Transform data from an SAP ODP source with the SAP CDC connector in Azure Data Factory or Azure Syna....

 

Data movement

 

Azure Databricks now supported through private link from a Data Factory managed virtual network

Azure Databricks is now a supported data service that can be connected to through a private link from a Data Factory managed virtual network. 

 

To learn more about this update, read Managed virtual network and managed private endpoints - Azure Data Factory | Microsoft Learn.

 

Developer Productivity

 

Pipeline designer enhancements added to ADF Studio preview experience

We have added three new UI updates to the ADF Studio preview experience:

Dynamic Content Flyout

A new dynamic content flyout has been added to make it easier to set dynamic content in your pipeline activities without having to use the expression builder.

 

Noelle_Li_1-1670017635052.png

 

Error Messaging

Error messages have been relocated to the Status column in Debug and Pipeline monitoring. This will make it easier to view errors whenever a pipeline run fails.

 

Noelle_Li_2-1670017635057.png

 

Container view for monitoring

A new monitoring view is available, adding the option to enable the container view. This view provides a more consolidated monitoring view of activities that were run. A big change that you will see is that nested activities are now grouped under the parent activity. To be able to view more activities on one page, you can choose to hide the list of nested activities under the parent activity.  

 

Noelle_Li_3-1670017635064.png

 

To learn more about these enhancements, read Managing Azure Data Factory studio preview experience - Azure Data Factory | Microsoft Learn.

 

We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in our blogs or livestreams. We love hearing your feedback!

 

1 Comment
Co-Authors
Version history
Last update:
‎Dec 13 2022 07:08 PM
Updated by: