Data Factory Increases Maximum Activities Per Pipeline to 80
This week we have doubled the limit on number of activities you may define in a pipeline, from 40 to 80. With more freedom to develop, we want to empower you to create more powerful, versatile, and resilient data pipelines for all your business needs. We are excited to see what you come up with, harnessing the power of 40 more activities per pipeline!Orchestrating ADX queries using managed Airflow
Apache Airflow is a widely used task orchestration framework, which gained its popularity due to Python-based programmatic interface - the language of first choice by Data engineers and Data ops. The framework allows defining complex pipelines that move data around different parts, potentially implemented using different technologies. The following article shows how to setup managed instance of Apache Airflow and define a very simple DAG (direct acyclic graph) of tasks that does the following: Uses Azure registered application to authenticate with the ADX cluster. Schedules daily execution of a simple KQL query that calculates HTTP errors statistics based on Web log records for the last day. Pre-requisites for completing the steps of this tutorial are: Microsoft Azure account. Running Azure Data Explorer (ADX) cluster.Pipeline Logic 3: Error Handling and Try Catch
In this series on Orchestration, we will dive deep to understand conditional executions in ADF, and build complex logic such as how to execute a shared error handling step of any failures in the pipeline, how to add informative logging with best effort attempt, and how to ensure all dependencies succeed before proceeding to next steps.Pipeline Logic 1: Error Handling and Best Effort Step
In this series on Orchestration, we will dive deep to better understand conditional executions in ADF, and build complex logic such as: how to execute a shared error handling step of any failures in the pipeline, how to add informative logging with best effort attempt, and how to ensure all dependencies succeed before proceeding to next steps.Pipeline Logic 2: OR (at least 1 activity succeeded or failed)
In this series on Orchestration, we will dive deep to understand conditional executions in ADF, and build complex logic such as how to execute a shared error handling step of any failures in the pipeline, how to add informative logging with best effort attempt, and how to ensure all dependencies succeed before proceeding to next steps.
