We are constantly inspired by all of you and how you use Azure Data Factory for all your data integration needs. Thank you for continuing to share your feedback with us. We love hearing from you and continue to use your input to improve Azure Data Factory.
Many exciting announcements are landing at the Microsoft Ignite 2022 conference, and, in this blog post, we’re thrilled to share the latest Azure Data Factory news.
Azure Data Factory (ADF) is a fully managed cloud service that's built for petabyte-scale data ingestion, data transformation, and orchestration at scale. Azure Data Factory provides data engineers with a familiar and productive environment for authoring their data integration pipelines and data flows for code-free transformations at scale. Users have sophisticated data pipeline and data transformation capabilities to author robust data integration tasks that operate over large amounts of data. Hundreds of connectors enable data-source-specific connectivity to ingest data from diverse data sources (e.g. multi-cloud, SaaS, on-premises), transform data at scale, and more.
Related: Microsoft named a Leader in 2022 Gartner® Magic Quadrant™ for Data Integration Tools
For Ignite 2022, we’ve focused on delivering new features and innovations which enable data teams to deliver insights to business decision-makers faster than ever.
SAP Change Data Capture Connector is Now Generally Available
Customers use SAP systems for their business-critical operations. Today, customers want to be able to combine their SAP data with non-SAP data for their analytics needs.
Azure Data Factory works seamlessly to combine data and prepare it at cloud-scale. Customers are using ADF to ingest data from different SAP data sources (e.g., SAP ECC, SAP Hana, SAP Table, SAP BW Open Hub, SAP BW via MDX, and SAP Cloud for Customers), and combine the SAP data with data from other operational stores (e.g., Azure Cosmos DB, the Azure SQL family, and more), enabling customers to gain deeper insights from both SAP and non-SAP data.
At Ignite this week, we’re announcing the general availability of the new SAP CDC connector in Azure Data Factory and Azure Synapse Analytics, enabling incremental load capabilities for a variety of SAP source systems, including SAP ECC, SAP S/4HANA, SAP BW, and SAP BW/4HANA based on SAP’s Operational Data Provisioning (ODP) framework. This is one of the first services supporting full fidelity change data capture (CDC) integrated as a source into ADF mapping data flows, and this new connector makes processing of change data from SAP sources simple and efficient.
To learn how you can leverage this new connector to transform your SAP data, read the documentation.
SQL CDC in Mapping Data Flows Launches in Public Preview
In a data integration solution, incrementally (or delta) loading data after an initial full data load is widely used for greatly reducing the time for reprocessing data.
Azure Data Factory is expanding its native change data capture capabilities to more connectors, including SQL Server, Azure SQL Database, and Azure SQL Managed Instance (MI). Changed data (including row inserts, updates, and deletions) in SQL stores can be automatically detected and extracted by ADF mapping data flows.
With the no-code experience in mapping data flows, users can easily implement data replication scenarios from SQL stores by appending a database as a destination store. Additionally, users can compose any data transformation logic to achieve an incremental ETL scenario from SQL stores.
To learn more about how you leverage these new native CDC capabilities, read the documentation.
Transform data at scale from Microsoft 365 (M365) with Azure Data Factory
Businesses that run on Microsoft 365 (M365) services need to use available data to guide decision making. When analyzed at scale, Microsoft 365 data can help you uncover insights to improve employee productivity, optimize business processes, safeguard your sensitive intellectual property, better serve your customers, and much more.
We are excited to announce a limited public preview for a new M365 Mapping Data Flows Connector in Azure Data Factory and Azure Synapse Analytics. Mapping Data Flows will clean, normalize, and flatten the data (Parquet format) into multiple data sinks. This accelerates access to Microsoft 365 data for analysis:
- Flattened data (Parquet format) is much cheaper and faster to process with big data processing (such as Spark or Hadoop), which reduces cost and accelerates data analytics (lower latency) compared to JSON.
- Additional data sink support (for Azure Cosmos DB and SQL DW) gives customers flexibility and greater efficiency in constructing data pipelines that optimize workflows.
This new capability will enable scale access to Microsoft 365 data for analytics to solve rich business problems. Sign-up is available here.
To learn more about this new connector, read the documentation.
Community Templates
The Azure Data Factory pipeline template gallery empowers users to get started quickly with building data factory pipelines, improving developer productivity while reducing development time for repeat processes. The template gallery contains use-case based templates, data movement templates, SSIS templates, and transformation templates that you can use to get hands-on with building your data factory pipelines.
Today, we are announcing support for community templates in Azure Data Factory to allow our community members to share their own pipeline templates for various use cases.
For instructions on how to contribute to the template gallery, read the documentation.
… and a BIG thank you to our community!
Thank you for continuing to share your feedback with us, especially on our community forums! We love hearing from you and your input is invaluable!
Some recent changes that we’ve made based on your feedback include:
Stay Involved
Stay up to date with our latest innovations on our What's New in ADF page and by reading our monthly blogs. We host monthly livestreams on YouTube, where we demo new features and answer your questions live!
Find our most recent blog at What’s New in ADF – Sept 2022 and our most recent livestream below:
For more ways to get involved, refer to our community resources:
We look forward to hearing your thoughts and seeing what you can do with these latest innovations in Azure Data Factory.
