The Durable Task Scheduler is a fully managed backend for Azure Durable Functions that enhances performance, reliability, and ease of monitoring of stateful orchestrations.
Today, we are excited to introduce an advancement in the durable execution space – the Durable Task Scheduler for Azure Durable Functions.
Sign up be considered for the limited early access preview: https://aka.ms/dts-ignite-signup.
Durable Functions lets developers write stateful orchestrations (sometimes referred to as workflows) within Azure Functions, Azure's serverless compute. Orchestrations allow developers to define sequential and/or parallel tasks for arbitrarily long-running business processes, something that is otherwise challenging to do in serverless. The orchestration state is durably persisted so that it may be recovered in the event of errors, which means developers do not have to architect for fault tolerance themselves. Examples of scenarios where Durable Functions are useful include AI/intelligent apps (i.e, RAG), distributed transactions, large-scale data processing, ETL (extract, transform, and load), and implementing asynchronous APIs.
Durable Functions rely on a backend storage component to handle orchestration and task scheduling, persist orchestration state, manage failures/handle retries, and load balance orchestration execution at scale. The existing storage providers for Durable Functions are bring-your-own (BYO) and are self-managed by development teams. This can lead to significant operational overhead, especially when it comes to the debugging and management of orchestrations since developers are required to sift through endless logs to understand the lifecycle of their orchestrations just to identify a potential failure. Also, setting up data backup and recovery is complex, as they require manual configuration of storage accounts. Lastly, the performance of these existing storage providers is suboptimal for some scenarios since they were not originally designed for durable execution purposes. These factors can pose significant challenges for developers, diverting their attention from delivering true business value...
To address these challenges and other experience gaps, the Azure Functions product group is launching a new storage provider, the Durable Task Scheduler. Fully managed by Azure, the Durable Task Scheduler offers several additional enhancements over the existing BYO storage options. These enhancements include superior performance in scheduling orchestrations, making it ideal for high-throughput applications; an out-of-the-box debugging and management dashboard that provides insights into the progress and execution history of your orchestrations, making it easy to diagnose and solve unexpected issues; security-related features like RBAC integrations, and more.
Azure Managed
Unlike existing storage providers for Durable Functions, the Durable Task Scheduler offers dedicated resources and is fully managed by Azure, ensuring optimal performance and reliability. This means tasks such as cleaning up execution history data, or handling failovers are executed for you by Azure. Ensuring your apps are resilient and you avoid creeping storage costs.
Please note that the Durable Task Scheduler is a paid offering, and pricing details will be shared in the near future.
Performance
The Durable Task Scheduler is designed to be the most performant backend for Durable Functions. Benchmarks show that the Durable Task Schduler can process approximately 200 orchestrations per second (1 worker on P2V3 compute), significantly surpassing the 26 orchestrations per second handled by the most common storage provider, Azure Storage. The Durable Task Scheduler is the recommended choice for applications requiring high throughput and scalability in scheduling orchestrations and tasks.
Debugging and Management Dashboard
Today, understanding the state of an orchestration and identifying if it has failed or become ‘stuck’ is no easy task. Often, developers must navigate through thousands or even millions of logs, manually correlating them to piece together the history of an orchestration. Debugging and managing your orchestrations should be simpler and be provided out of the box, without the need to set up and configure a custom dashboard.
That’s why the Durable Task Scheduler comes with an out-of-the-box, user-friendly dashboard for debugging and managing orchestrations, all secured with Role-Based Access Control (RBAC).
You can use the dashboard to review execution history with fine-grained filters that enable you to filter by orchestration property, including status. You can view inputs and outputs from each orchestration execution in the “Actions” column, as well as drill into specific instances to view task execution history both through tables and visual references, like gantt and sequence charts. This makes it extremely easy to understand the lifecycle of your orchestrations and identify any unexpected behavior, for example, a failure that resulted in numerous retries or an orchestration that may appear “stuck”.
Lastly, the dashboard provides management controls that allow you to manage orchestrations ad hoc. These controls offer options to pause/resume in-flight orchestrations, terminate them, or raise an event if an orchestration requires human interaction to proceed. These features make it simple to interact with orchestrations using a UI, eliminating the need for coding or other mechanisms.
Not using Azure Functions? The Durable Task Scheduler is compatible with the Durable Task Framework
Users of the Durable Task Framework can also easily leverage the Durable Task Scheduler as the storage backend for their apps. The Durable Task Framework is an open-source library for writing stateful orchestrations and is the underlying framework used by Durable Functions.
Roadmap
This is just the beginning for the Durable Task Scheduler. Over time, the Durable Task Scheduler will evolve to include more capabilities, such as:
- Multiple versioning strategies to ensure a seamless migration of new workflow schemas while legacy workflows are still in flight.
- Encryption of sensitive data with customer-managed keys
- Full feature availability and resiliency through zone availability and automatic failover
- The ability to interact with orchestrations across teams/networks/regions
- SLAs for availability and latency guarantees.
While integration into Durable Functions is the initial focus, we are also evaluating the expansion of integrating the Durable Task Scheduler into other compute environments through alternative developer workflow services like Dapr workflow or through introducing a Durable Task SDK. The goal is to ensure that, regardless of where your orchestrations are deployed, the Durable Task Schduler can be leveraged as the backend.
If you want to discuss the roadmap or have any other questions and/or suggestions, please drop us an email at DurableTaskScheduler@microsoft.com.
Sign up for limited early access
The Durable Task Schduler has first-class integration with Durable Functions, meaning existing apps can simply make a few configuration changes to start leveraging it – no code changes required.
If you’re interested in participating in the early access preview for the Durable Task Scheduler, consider signing up using the form linked below. If selected, the Azure Functions product group will reach out to you with onboarding instructions.
Sign up using the link below to be considered for the limited early access preview for the Durable Task Scheduler: https://aka.ms/dts-ignite-signup
Demo
Microsoft
