Welcome to Azure Data Factory’s September monthly update! Here, we’ll share the latest updates on what’s new in Azure Data Factory. You can also find all our updates on our What's New in ADF page.
We’ll also be hosting our monthly livestream next week on October 11th at 11:30am PST/ 2:30pm EST! Join us to see some live demos and to ask us your ADF questions!
Join the livestream here.
Table of Contents
Data flow
- Amazon S3 source connector added [Doc]
- Google Sheets REST-based connector added as Source (Preview) [Doc]
- Maximum column optimization in dataflow [Doc]
- SAP Change Data Capture capabilities in Mapping Data Flow (Preview) - Extract and transform data changes from SAP systems for a more efficient data refresh [Doc]
- Writing data to a lookup field via alternative keys supported in Dynamics 365/CRM connectors for mapping data flows [Doc]
Data movement
Monitoring
- Additional monitoring improvements in Azure Data Factory [Blog]
- Additional sorting columns in monitoring – sorting added for Pipeline name, Run End, and Status
- Monitoring consumption report improvement – loading icon added to know when consumption report is fully calculated
- Monitoring loading improvements - pipeline re-run groupings data fetched only when expanded
- Pagination added to pipeline activity runs view to show all activity records in pipeline run
- Time-zone settings now saved in monitoring
- Gantt chart view now supported in IR monitoring [Doc]
Orchestration
User interface
- Native UI support of parameterization added for 6 additional linked services – SAP ODP, ODBC, Microsoft Access, Informix, Snowflake, and DB2 [Doc]
- Pipeline designer enhancements added to Default studio experience from Preview experience [Blog]
Data flow
Amazon S3 source connector added
We’ve added a new source connector to mapping data flows! Users can now read their data from Amazon Simple Storage Service (S3). Note: This connector is only available for the Dataset Source Type.
To read more information on this new source connector, read Copy and transform data in Amazon Simple Storage Service (S3) - Azure Data Factory & Azure Synapse | Microsoft Learn.
Google Sheets REST-based connector added (Preview)
We’ve added a new REST-based connector to mapping data flows! Users can now read their data from Google Sheets. Note: This connector is only available when using inline datasets.
To read more information on the Google Sheets connector (Preview), read Transform data in Google Sheets (Preview) - Azure Data Factory & Azure Synapse | Microsoft Learn.
Maximum column optimization in mapping dataflow
For CSV data sources, we’ve added a new setting that allows you to set the maximum number of columns. The default is 20480 so if you have more than 20480 columns, you can customize this value and optimize it for your dataset.
To learn more about this setting, refer to this document, Mapping data flows - delimited text format in Azure Data Factory.
SAP Change Data Capture capabilities in Mapping Data Flow (Preview) - Extract and transform data changes from SAP systems for a more efficient data refresh
We made an exciting step in our SAP CDC roadmap! The SAP CDC connector is now available in ADF mapping data flows, providing a simple, integrated experience to extract and transform the raw change data coming from SAP.
To read more information about these new capabilities, read Transform data from an SAP ODP source with the SAP CDC connector in Azure Data Factory or Azure Synapse Analytics - Azure Data Factory & Azure Synapse | Microsoft Learn
Writing data to a lookup field via alternative keys supported in Dynamics 365/CRM connectors for mapping data flows
Now, you can write data to a lookup field when using Dynamics 365/CRM connectors in mapping dataflows. With alternate keys, you can define an attribute in your entity to correspond to a unique identifier, which can be used to uniquely identify a record in place of the primary key. In mapping data flows, writing data to a lookup field via alternative keys is quick and simple – just update your sink mappings! Note: This is currently supported only when you use an inline dataset for your mapping data flow sink transformation.
To learn more about this update, read Copy and transform data in Dynamics 365 (Microsoft Dataverse) or Dynamics CRM - Azure Data Factory & Azure Synapse | Microsoft Learn.
Data movement
Support to convert Oracle NUMBER type to corresponding integer in source
We have added new support to convert Oracle NUMBER type to corresponding integer in source. You will see a new property convertDecimalToInteger, which will allow you to convert Oracle NUMBER type from zero or an unspecified scale to the corresponding integer.
To learn more about this update, read Copy data to and from Oracle by using Azure Data Factory or Azure Synapse Analytics.
Monitoring
Additional monitoring improvements in Azure Data Factory
We’ve released a new bundle of improvements to the monitoring experience based on community feedback. These improvements aim to provide a smoother and more comprehensive monitoring experience.
Additional sorting columns in monitoring – sorting added for Pipeline name, Run End, and Status
We now have more columns available for sorting. Previously, you could only sort by Run start. Now, you can sort by Pipeline Name, Run End, and Status in addition to the Trigger Name and Status columns.
Monitoring consumption report improvement – loading icon added to know when consumption report is fully calculated
The pipeline consumption report now shows a loading icon when there are many activities in a pipeline to let you know when the consumption information has been fully calculated.
Monitoring loading improvements - pipeline re-run groupings data fetched only when expanded
Previously, when pipelines were re-run, all re-runs were retrieved for the monitoring UI, even when the re-runs were not viewed. Now, pipeline groupings for re-runs will only fetch data when the re-run grouping is expanded. This will improve the performance of monitoring pages that have multiple pipeline re-run groupings.
Pagination added to pipeline activity runs view to show all activity records in pipeline run
Previously, in pipelines with many activities, the list of records might have timed-out or returned an incomplete list of records. Now, when monitoring pipeline runs, all records for pipelines will be retrieved and can be viewed in different pages to help you navigate through all the activity records.
Time-zone settings now saved in monitoring
Monitoring time zone settings are now saved in your browser cache with your configured columns so that you don’t have to reconfigure your settings each time!
To learn more about these monitoring updates, read Azure Data Factory monitoring improvements (microsoft.com).
Gantt chart view now supported in IR monitoring
You now have the option to view your activity runs with a Gantt chart in Integration Runtime monitoring. A Gantt chart is a view that allows you to see the run history over a time range. This will give you additional monitoring views of your integration runtimes.
For more information, read Monitor integration runtime in Azure Data Factory - Azure Data Factory | Microsoft Learn.
Orchestration
DELETE method in the Web activity now supports sending a body with HTTP request
We have added support for sending a body (optional) when using the DELETE method in Web activity. This is useful in the case where your web endpoint expects a body to be passed with the HTTP request DELETE method.
To learn more about this update, read Web Activity - Azure Data Factory & Azure Synapse | Microsoft Learn.
User interface
Native UI support of parameterization added for 6 additional linked services – SAP ODP, ODBC, Microsoft Access, Informix, Snowflake, and DB2
We have added native UI support of parameterization for 6 additional linked services! These linked services include SAP ODP, ODBC, Microsoft Access, Informix, Snowflake, and DB2. This provides a built-in parameterization experience with parameter options and dynamic content support when authoring linked services.
For more information, read Parameterize linked services - Azure Data Factory & Azure Synapse | Microsoft Learn.
Pipeline designer enhancements added to Default studio experience from Preview experience
We are moving some of our exciting preview UI updates from Preview Experience to the default UI designer!
You will see a new floating toolbox when adding activities to your pipeline. This will allow you to add the next activity in your workflow sequence without needing to drag in an activity from the mounted toolbox.
In addition to this update to adding pipeline activities, you will also see activity connectors on the right-hand side of the pipeline activity, allowing you to easily choose what kind of activity connector to add.
Lastly, you will have the ability to view iteration and conditional activities with the container view. This allows you to adjust your activity size to be able to see the activities contained within your iteration and conditional activities.
For more information on the UI updates, read Azure Data Factory Updated Pipeline Designer (microsoft.com).
We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in these blogs or livestreams. We love hearing your feedback!
Microsoft
