Improve Spark pool utilization with Synapse Genie
Synapse Genie Framework improves Spark pool utilization by executing multiple Synapse notebooks on the same Spark pool instance. It considers the sequence and dependencies between notebook activities in an ETL pipeline, which results in higher usage of a full cluster for resources available in a Spark pool.Announcing Public Preview of the SAP CDC solution in Azure Data Factory and Azure Synapse Analytics
UPDATE! We've launched the new SAP connector in ADF today, June 30, 2022, and updated the blog post below accordingly. For decades, companies have relied on Microsoft and SAP software to run their most mission-critical operations. Today, we’re excited to launch public preview of SAP Change Data Capture (CDC) in Azure Data Factory (ADF). Combining a new data connector with predefined data flow templates, this solution streamlines the integration of SAP data within core Azure services like Azure Synapse Analytics and Azure Machine Learning. The new SAP ODP connector leverages SAP Operational Data Provisioning (ODP) framework, which is an established best practice for data integration within SAP landscapes. ODP provides access to a wide range of sources across all major SAP applications and comes with built-in CDC capabilities. In combination with the predefined data flow templates to process and update the changed records to any sink, this makes SAP data integration into Azure very much straight forward.Process your data in seconds with new ADF real-time CDC
In January, we announced that we've elevated our Change Data Capture features front-and-center in ADF. Up until just today, the lowest latency we were allowing for CDC processing was 15 minutes. But today, I am super-excited to announce that we have enabled the real-time option!Introducing 'Workflow Orchestration Manager' powered by Apache Airflow in Azure Data Factory
Today, we are excited to announce the capability to run Apache Airflow DAGs (Directed Acyclic Graph) within Azure Data Factory, adding a key Open-Source integration that provides extensibility for orchestrating python-based workflows at scale on Azure.Announcing the Public Preview of a new top-level CDC resource in ADF
Azure Data Factory (ADF) has recently added many new CDC-enabled connectors to process change data from SQL, Storage, Cosmos DB, and many other sources. Much of the feedback that we received from our users about this has been centered around making it easy to configure and to continuously detect changes at the source. We heard your feedback and are super excited to announce the immediate release of a new top-level ADF resource that is now available in public preview in your ADF resource explorer!Granular Billing for Azure Data Factory
In this blog, Charlie Zhu walks you through the new granular billing option for Azure Data Factory and helps you better understand pipeline costs. We are bringing clarity and transparency into data pipelines operations with built-in per pipeline billing report so that you will know exactly how much each pipeline costs you.
SharePoint Online Multiple Files (Folder) Copy with Http Connector
This blog shows how to copy multiple files from a folder from SharePoint Online using ADF. Go through this public documentation on how to copy a single file - Copy data from SharePoint Online List by using Azure Data Factory - Azure Data Factory | Microsoft DocsIntroducing the Management Hub to Azure Data Factory
The management hub, accessed by the Manage tab in the Azure Data Factory UX, is a portal that hosts global management actions for your data factory. You can manage your connections to data stores and external computes, source control configuration, and trigger settings.
