Forum Discussion
Multiple Triggers in Azure Databricks Lakeflow Jobs: Simplifying Data Orchestration
A new feature that simplifies event-driven pipeline execution in Azure Databricks.
On October 5, 2026, Databricks announced support for Multiple Triggers in Lakeflow Jobs, currently available in Beta.
Until now, a Job could have only one configured trigger. This created an interesting limitation in data engineering projects: what happens when the same pipeline needs to run under different conditions?
Imagine a data ingestion pipeline that needs to run when a new file arrives in storage, when a Delta table is updated, or every day at a specific time, regardless of those events.
Previously, a common solution was to create separate Jobs, each with its own trigger, while reusing or duplicating the processing logic. Another option was to create intermediate Jobs responsible for calling the main pipeline.
Both approaches work, but they introduce additional components and increase maintenance effort.
With Multiple Triggers, we can now configure different execution conditions directly within the same Job.
1. How did it work before?
Let's consider a pipeline responsible for processing sales data.
This pipeline has three requirements:
- Process new files arriving in a Unity Catalog Volume.
- Reprocess data whenever a source Delta table is updated.
- Run every day at 8 AM to ensure scheduled processing.
Before Multiple Triggers, each Job supported only one trigger configuration.
One possible approach was to create three separate Jobs, each responsible for a specific trigger type, pointing to the same processing logic.
Although notebooks and tasks could be shared across these Jobs, we still needed to manage separate Job definitions, permissions, parameters, and execution settings.
In enterprise environments, this complexity can grow as the number of pipelines and dependencies increases.
2. What changed with Multiple Triggers?
Azure Databricks now allows us to configure up to five triggers within a single Lakeflow Job, including combinations of different trigger types.
For example:
Scheduled Trigger: Runs the Job at a predefined time.
File Arrival Trigger: Runs when new files are detected.
Table Update Trigger: Runs when specific tables are updated.
Model Update Trigger: Runs based on updates to models registered in Unity Catalog.
The execution logic is straightforward: when any active trigger meets its condition, it can initiate a Job run.
This means we no longer need to maintain separate Jobs just to support multiple execution conditions.
Another useful feature is the ability to pause or resume each trigger independently.
For example, during maintenance, we can temporarily pause the Table Update Trigger while keeping the daily schedule active.
3. Practical example: Configuring Multiple Triggers
Let's create a Job responsible for processing sales data.
In this example, we will configure three execution conditions:
Trigger 1: New files arriving in a Unity Catalog Volume.
Trigger 2: Updates to the bronze.sales.orders table.
Trigger 3: Daily execution at 8 AM, using the São Paulo time zone.
We can configure these triggers through the Azure Databricks interface by opening the Job and adding triggers under Schedules & Triggers.
We can also use the Jobs API to automate this configuration.
Example using the Jobs API
The following JSON illustrates a Job configuration using the triggers array.
{
"name": "sales_processing_job",
"max_concurrent_runs": 1,
"tasks": [
{
"task_key": "process_sales",
"notebook_task": {
"notebook_path": "/Shared/pipelines/process_sales"
},
"existing_cluster_id": "<CLUSTER_ID>"
}
],
"triggers": [
{
"file_arrival": {
"url": "/Volumes/bronze/sales/landing"
},
"pause_status": "UNPAUSED"
},
{
"table_update": {
"table_names": [
"bronze.sales.orders"
],
"condition": "ANY_UPDATED"
},
"pause_status": "UNPAUSED"
},
{
"schedule": {
"quartz_cron_expression": "0 0 8 * * ?",
"timezone_id": "America/Sao_Paulo"
},
"pause_status": "UNPAUSED"
}
]
}In this example, we have a single Job with three independent execution conditions.
The triggers field contains an array of trigger configurations, allowing multiple triggers to be defined without relying on the legacy top-level schedule, trigger, or continuous fields.
The example uses an existing cluster for simplicity. In production environments, the compute configuration should be selected according to workload requirements and organizational policies.
Creating the Job with Python
We can submit the configuration using Python and the requests library.
import os
import json
import requests
workspace_url = os.environ["DATABRICKS_HOST"].rstrip("/")
token = os.environ["DATABRICKS_TOKEN"]
with open("sales_job.json", encoding="utf-8") as file:
job_settings = json.load(file)
response = requests.post(
f"{workspace_url}/api/2.1/jobs/create",
headers={
"Authorization": f"Bearer {token}",
"Content-Type": "application/json"
},
json=job_settings,
timeout=30
)
response.raise_for_status()
print("Job created:", response.json()["job_id"])The sales_job.json file contains the JSON configuration shown earlier.
For enterprise implementations, I recommend using service principals and secure authentication mechanisms instead of embedding credentials directly in the code.
4. What happens when two triggers fire at the same time?
This is an important architectural consideration.
Imagine a new file arriving at exactly 8 AM, when the scheduled execution is also expected to run.
Both triggers may request an execution of the same Job.
However, this does not necessarily mean that two runs will execute simultaneously. The behavior depends on the Job's concurrency settings.
By default, a Job allows only one active run at a time. Therefore, we need to consider concurrency limits and how additional runs are handled.
Another important consideration is idempotency.
In simple terms, running the same pipeline multiple times against the same data should not produce duplicate records or inconsistent results.
In Azure Databricks, we can use Delta Lake features and MERGE operations to implement idempotent processing strategies.
Multiple Triggers improves execution flexibility, but it does not replace proper pipeline design.
5. Monitoring Jobs using System Tables
One capability I find particularly useful is the possibility of monitoring trigger configurations through Databricks System Tables.
The system.lakeflow.jobs table provides information about Jobs, including the trigger_type and triggers fields.
When a Job has multiple triggers, the trigger_type field can contain the value MULTIPLE.
We can identify these Jobs using SQL:
WITH current_jobs AS (
SELECT
workspace_id,
job_id,
name,
trigger_type,
triggers,
ROW_NUMBER() OVER (
PARTITION BY workspace_id, job_id
ORDER BY change_time DESC
) AS rn
FROM system.lakeflow.jobs
)We can also expand the triggers array to inspect the individual configurations associated with each Job.
WITH current_jobs AS ( SELECT workspace_id, job_id, name, triggers, ROW_NUMBER() OVER ( PARTITION BY workspace_id, job_id ORDER BY change_time DESC ) AS rn FROM system.lakeflow.jobs ) SELECT workspace_id, job_id, name, trigger FROM current_jobs LATERAL VIEW EXPLODE(triggers) AS trigger WHERE rn = 1;
This approach can support governance and observability dashboards, especially in environments with hundreds or thousands of Jobs.
6. Limitations and considerations
Since Multiple Triggers is currently in Beta, there are some important limitations and requirements:
- Each Job can have up to five triggers.
- The Multiple Triggers feature must be enabled in the workspace Preview settings.
- Each trigger can be paused or resumed independently.
- Continuous triggers cannot be combined with other trigger types.
- When updating the triggers array through the Jobs API, the entire existing trigger list is replaced.
The last point is particularly important when working with infrastructure automation or CI/CD pipelines.
When updating trigger configurations through the API, we need to make sure that all triggers we want to preserve are included in the new definition.
7. My perspective on this change
Based on my experience working with data architecture and data engineering projects, one recurring challenge is avoiding unnecessary complexity in orchestration.
The problem is not always related to data processing itself. Sometimes, it comes from the number of components we need to maintain just to execute relatively simple logic.
Multiple Triggers does not change how we build tasks within a Job, but it improves how we define when those tasks should run.
I see significant value in this feature for incremental ingestion, event-driven processing, and pipelines that need to combine data updates with scheduled execution.
I also appreciate the ability to manage triggers independently, which can make maintenance and operational control easier.
It may seem like a relatively small configuration change, but it can lead to meaningful architectural simplification in certain scenarios.
Conclusion
With Multiple Triggers support, Azure Databricks Lakeflow Jobs provides greater flexibility for configuring different execution conditions within a single Job.
This feature reduces the need for auxiliary Jobs or duplicated Job definitions, making orchestration architectures simpler and easier to maintain.
We still need to consider concurrency, idempotency, governance, and observability, but we now have another option for building event-driven pipelines without introducing unnecessary components.
For anyone working with Azure Databricks, this feature is worth exploring to identify opportunities to simplify existing orchestration workflows.
Official references
Databricks: Automate jobs with schedules and triggers
Databricks Release Notes, October 2026
Databricks Jobs API
Databricks System Tables: Jobs