Introducing 'Workflow Orchestration Manager' powered by Apache Airflow in Azure Data Factory
Today, we are excited to announce the capability to run Apache Airflow DAGs (Directed Acyclic Graph) within Azure Data Factory, adding a key Open-Source integration that provides extensibility for orchestrating python-based workflows at scale on Azure.Orchestrating ADX queries using managed Airflow
Apache Airflow is a widely used task orchestration framework, which gained its popularity due to Python-based programmatic interface - the language of first choice by Data engineers and Data ops. The framework allows defining complex pipelines that move data around different parts, potentially implemented using different technologies. The following article shows how to setup managed instance of Apache Airflow and define a very simple DAG (direct acyclic graph) of tasks that does the following: Uses Azure registered application to authenticate with the ADX cluster. Schedules daily execution of a simple KQL query that calculates HTTP errors statistics based on Web log records for the last day. Pre-requisites for completing the steps of this tutorial are: Microsoft Azure account. Running Azure Data Explorer (ADX) cluster.
