Orchestrate Azure Container Apps Jobs with Apache Airflow

Azure Container Apps (ACA) Jobs are a great way to run work that starts, does something, and finishes: nightly batch, data processing, ETL, ML scoring, report generation. They scale to zero, bill per execution, and run any container you give them.

But the moment your "one job" becomes "a set of jobs that depend on each other," a gap appears:

How do I run twenty jobs in parallel, wait for all of them, then run one more job only if they all succeeded — and retry just the one that failed?

A single ACA Job can't express that on its own. What you're describing is an orchestrator, and the most widely adopted one in the data world is Apache Airflow.

This post introduces two open-source templates that connect the two, so Airflow becomes the brain and ACA Jobs become the muscle. Pick the one that matches what you already run:

• airflow-on-aca-jobs: you already have Airflow. Drop in an operator and point it at ACA Jobs. Host nothing new.
• airflow-hosted-on-aca: you don't have Airflow. Get a full one running on Azure Container Apps with one command.

Both use the same operator and the same DAGs, so you can start with one and move to the other later without rewriting your workflows.

See Airflow orchestrate real ACA Job executions with parallel fan-out, dependency ordering, and automatic retries.

Why ACA Jobs need an orchestrator

A plain ACA Job is great at one thing: run this container to completion, then stop. That covers a scheduled job or a one-off task perfectly.

Real pipelines need more than that:

• Dependency ordering: step B runs only after step A succeeds.
• Parallel fan-out: launch one execution per file, per store, or per partition, all at once, then wait for the whole batch.
• Per-task retries: if one execution in a batch of fifty fails, retry just that one, not the other forty-nine.
• Backfills and scheduling: re-run yesterday's pipeline, or run every night with a full history of what happened.

These are the problems an orchestrator solves. Instead of building that logic yourself, you let Airflow handle the graph, the scheduling, and the retries, while ACA Jobs run the compute. You get serverless, scale-to-zero workers, and you didn't have to stand up a scheduler to get them.

The operator that ties them together

Both templates ship the same small plugin: an Airflow operator called AzureContainerAppsJobOperator. In a DAG it looks like any other task:

report_sales = AzureContainerAppsJobOperator(
    task_id="report_store_sales",
    subscription_id="{{ var.value.azure_subscription_id }}",
    resource_group="{{ var.value.aca_resource_group }}",
    job_name="{{ var.value.aca_job_name }}",
    image="python:3.12-slim",
    command=["python", "-c", MY_PROGRAM],
    env_vars={"STORE_NAME": "Seattle"},
    deferrable=True,
)

A few things make this operator easy to work with:

• Per-execution overrides. It takes the ACA Job you point it at and overrides the image, command, args, and env_vars for that run. You can drive many different workloads from a single ACA Job definition, and you don't need to build or push a custom image just to try something. The example above runs the stock python:3.12-slim image with an inline program.
• Deferrable by default. With deferrable=True, Airflow frees its worker slot while the ACA Job runs and resumes when it finishes. That means your fan-out width is bounded by ACA, not by how many Airflow workers you have. You can launch dozens of parallel executions cheaply.
• No secrets required. Authentication resolves in a sensible order: an Airflow Connection if you set one, otherwise an AZURE_ACCESS_TOKEN environment variable, otherwise DefaultAzureCredential (managed identity). In Azure, the hosted template uses a managed identity so nothing sensitive is stored in Airflow at all.

Because both templates share this operator, a DAG written for one runs unchanged on the other.

Option 1: Bring your own Airflow (host nothing)

Choose airflow-on-aca-jobs if you already run Airflow: Azure Managed Airflow, MWAA, Astronomer, or your own deployment. You keep that Airflow exactly as it is and simply teach it to talk to ACA Jobs.

+------------------------------------------+
|  Your Airflow  (you host it, unchanged)  |
|  runs AzureContainerAppsJobOperator      |
+------------------------------------------+
                     |
                     |  ACA Jobs REST API
                     v
+------------------------------------------+
|  ACA Job  (Azure Container Apps)         |
|                                          |
|  store 1 | store 2 | ... | store N       |
|  parallel executions -> scale to zero    |
+------------------------------------------+

Your existing Airflow runs the operator; ACA Jobs run the work. You host nothing new.

Adoption is three small steps:

1. Copy the operator into your Airflow's plugins/ folder.
2. Add a DAG that uses AzureContainerAppsJobOperator.
3. Set three Airflow Variables so the operator knows which job to drive:

That's the whole integration. Nothing new to host, no extra scheduler or database, no custom image. ACA Jobs just become another task type Airflow can call.

If you want a job to point at first, the template includes an Azure Developer CLI (azd) deployment that stands up a sample ACA Job for you:

git clone https://github.com/hetvip2/airflow-on-aca-jobs
cd airflow-on-aca-jobs
azd up          # deploys a sample ACA Job, prints its resource group + name

Then copy airflow/plugins/ and airflow/dags/ into your Airflow, set the three Variables, and trigger the DAG.

Option 2: Airflow hosted on ACA (turnkey)

Choose airflow-hosted-on-aca if you don't already have an orchestrator and want one running next to your jobs. One command provisions the whole thing on Azure Container Apps:

                   azd up
                     |
                     v
+------------------------------------------+
|  Airflow control plane on ACA            |
|  web  |  scheduler  |  triggerer         |
|  Postgres (metadata) + Azure Files (dags)|
|  Managed Identity - no secrets stored    |
+------------------------------------------+
                     |
                     |  ACA Jobs REST API
                     v
+------------------------------------------+
|  ACA Job  (Azure Container Apps)         |
|                                          |
|  store 1 | store 2 | ... | store N       |
|  parallel executions -> scale to zero    |
+------------------------------------------+

One command deploys the whole Airflow control plane on ACA, right next to the jobs it drives.

git clone https://github.com/hetvip2/airflow-hosted-on-aca
cd airflow-hosted-on-aca
azd env new my-airflow
azd up          # prints your Airflow URL when it finishes

azd up deploys a complete, working Airflow control plane on ACA:

• airflow-web, airflow-scheduler, and airflow-triggerer running as Container Apps on LocalExecutor, so there's no Celery or Redis to operate.
• A Postgres metadata database.
• A user-assigned managed identity with permission to call the ACA Jobs API, so the operator authenticates with no secrets stored in Airflow.
• A sample ACA Job for Airflow to drive out of the box.

Your DAGs and plugins live on a mounted Azure Files share, so you ship new workflows by re-uploading files rather than rebuilding an image:

cp my_dag.py airflow/dags/
azd hooks run postprovision   # uploads dags + plugins to the share

Airflow picks up the change within a minute. You now own a real orchestrator, hosted serverlessly on the same platform as your jobs.

Which one should you pick?

The important part: the workload never changes. The same DAG and the same operator drive the same ACA Job executions in both. Start wherever you are today, and switch later with zero changes to your pipelines.

See it end to end

Picture a retailer that wants one number every night: total sales across all stores. Each store reports its own sales as a separate ACA Job execution, all running in parallel. When every store is in, a final job adds them into the company total.

That one workflow exercises exactly what a plain Job can't do alone:

• parallel fan-out: one ACA Job execution per store, all at once
• dependency ordering: the roll-up runs only after every store reports
• per-task retries: if a store's execution fails, Airflow retries just that store, and the nightly total still lands

In Airflow's Graph view you watch the store tasks light up together, then the roll-up run last. In the Azure portal you watch real executions appear under your ACA Job and scale back to zero when they finish. Same job, same DAG, whichever template you chose.

Call to action

If you run batch, ETL, or any multi-step work on Azure Container Apps Jobs, give one of these templates a try:

• Already have Airflow? Start with airflow-on-aca-jobs.
• Need an orchestrator? Start with airflow-hosted-on-aca.

Both are open source, deploy with azd up, and share the same operator so you can move between them freely. Try them out and let us know what you orchestrate.