General availability of SAP CDC capabilities for Azure Data Factory and Azure Synapse Analytics
Customers use SAP systems for their business-critical operations. Today, customers want to be able to combine their SAP data with non-SAP data for their analytics needs. Azure Data Factory (ADF) is an industry-leading data integration service which enables customers to ingest data from diverse data sources (e.g., multi-cloud, SaaS, on-premises), transform data at scale, and more. Azure Data Factory (ADF) works seamlessly to combine data and prepare it at cloud-scale. Customers are using ADF to ingest data from different SAP data sources (e.g., SAP ECC, SAP Hana, SAP Table, SAP BW Open Hub, SAP BW via MDX, SAP Cloud for Customers), and combining them with data from other operational stores (e.g., Cosmos DB, Azure SQL family, and more). This enables customers to gain deep insights from both SAP and non-SAP data. Today, we are excited to announce the General Availability of SAP CDC support in Azure Data Factory and Azure Synapse Analytics.
Optimizing ETL Workflows: A Guide to Azure Integration and Authentication with Batch and Storage
Unlock the Power of Azure: Transform Your ETL Pipelines Dive into the world of data transformation and discover how to build a solid foundation for your ETL pipelines with Azure’s powerful trio: Data Factory, Batch, and Storage. Learn to navigate the complexities of data authentication and harness the full potential of Synapse Pipeline for seamless integration and advanced data processing. Ready to revolutionize your data strategy? This guide is your key to mastering Azure’s services and optimizing your workflows.
Rerun from failed activity inside conditional block - Azure Data Factory
I have a pipeline that includes an if activity. In the true block of the if condition, there are three activities (let’s call them A, B, and C) in sequence. A and B were successful, but C failed. When I rerun the pipeline from the point of failure (POF), it executes A and B again, even though they were already successful. My expectation was that only the C activity would be triggered, and the previously successful activities would be skipped. Is there any workaround to achieve this?Announcing Public Preview of the SAP CDC solution in Azure Data Factory and Azure Synapse Analytics
UPDATE! We've launched the new SAP connector in ADF today, June 30, 2022, and updated the blog post below accordingly. For decades, companies have relied on Microsoft and SAP software to run their most mission-critical operations. Today, we’re excited to launch public preview of SAP Change Data Capture (CDC) in Azure Data Factory (ADF). Combining a new data connector with predefined data flow templates, this solution streamlines the integration of SAP data within core Azure services like Azure Synapse Analytics and Azure Machine Learning. The new SAP ODP connector leverages SAP Operational Data Provisioning (ODP) framework, which is an established best practice for data integration within SAP landscapes. ODP provides access to a wide range of sources across all major SAP applications and comes with built-in CDC capabilities. In combination with the predefined data flow templates to process and update the changed records to any sink, this makes SAP data integration into Azure very much straight forward.Orchestrate and operationalize Synapse Notebooks and Spark Job Definitions from Azure Data Factory
Today, we are introducing support for orchestrating Synapse notebooks and Synapse spark job definitions (SJD) natively from Azure Data Factory pipelines. These new native activities make it easy to run Synapse notebooks and SJD from an ADF pipeline!Implement Fill Down in ADF and Synapse Data Flows
"Fill Down" is an operation common in data prep and data cleansing meant to solve the problem with data sets when you want to replace NULL values with the value from the previous non-NULL value in the sequence. Here is how to implement this in ADF and Synapse data flows.Map Data tool in Synapse Analytics now generally available
Mapping data flows, in Azure Synapse Analytics, is the scale-out data transformation feature that allows data engineers to build low-code ETL jobs visually and execute using serverless Spark compute. Now with the Map Data tool, you can quickly build Mapping Data Flows in a wizard-like approach to get started with transformation at scale even faster.Introducing User Defined Functions preview for mapping data flows in ADF and Synapse
Mapping data flows, in Azure Data Factory and Synapse Analytics, is the scale-out data transformation feature that allows data engineers to build low-code ETL jobs visually and execute using serverless Spark compute. Now with User Defined Functions (preview), you can quickly build reusable function logic to share across multiple mapping data flows.
