We’re excited to announce that support for the new Azure Durable Functions storage providers, Netherite and Microsoft SQL Server (MSSQL), is now generally available! Some of you may have learned about these new storage providers during the annual Build conference and/or from this preview announcement blog post.
Durable Functions enables you to write long-running, reliable, event-driven, and stateful logic on the serverless Azure Functions platform using everyday imperative code. We believe the new storage providers will enable a host of new scenarios for apps using Durable Functions because they allow apps to run at a higher scale, with greater price-performance efficiency, and with more portability compared to the default option Azure Storage.
Why we developed the new storage providers
Durable Functions requires a storage provider, the default of which is Azure Storage, to persist orchestration and entity states. Azure Storage is the lowest-cost and most convenient option because it requires no additional configuration. As such, it will remain the default storage provider for Durable Functions and will be under active development just like the new providers. Azure Storage, however, does have some notable limitations that prevent it from enabling certain scenarios for apps using Durable Functions:
These limitations motivated investments in the new storage providers.
Netherite for maximum orchestration throughput
Netherite may sound familiar to you if you’re a Minecraft fan. It refers to a rare material that is more durable than diamond and is resistant to fire and lava. The Netherite storage provider aspires to have similar qualities in the context of Durable Functions.
Designed and developed by Microsoft Research, the Netherite provider is optimized for high throughput scenarios. It uses Azure Event Hubs and the FASTER database technology on top of Azure Page Blobs to store state information. Its better throughput performance is achieved by representing states differently than Azure Storage. For a more detailed explanation of how this is done and to see how Netherite’s architecture is like, see this paper “Netherite: Efficient Execution of Serverless Workflows”.
To illustrate Netherite’s throughput performance, we show the results of an example benchmark called Hello5. This benchmark performs many independent orchestrations concurrently, each of which calls 5 activities in sequence. We made the activities do very little CPU work so we can measure how efficient orchestration progress is persisted to storage. Read more about this benchmark.
Single-Node Throughput |
Multi-Node Throughput |
|
|
Key observations for the Hello5 benchmark:
For more information and numbers, see the article on multi-node throughputs in the documentation.
Note: Results may vary due to many factors. We’re simply presenting the observations of a benchmark we did and are not claiming to guarantee specific numbers. We encourage you to run these tests yourself and tweak them for your purposes.
Microsoft SQL for maximum control and portability
The Microsoft SQL (MSSQL) storage provider for Durable Functions was designed to fulfill enterprise needs, including the ability to decouple from the Azure cloud.
MSSQL can run anywhere, including on-premises servers, Edge devices, Linux Docker containers, on the Azure SQL Database serverless tier, and even on competitor cloud providers like AWS and GCP. With the support of this storage provider, you can run Durable Functions anywhere that Azure Functions can run, including your own Azure Arc-enabled Kubernetes clusters. In fact, the Azure Functions Core Tools and the Azure Arc App Service extension have been updated to support automatically configuring Durable Function apps on a Kubernetes cluster with the MSSQL KEDA scaler for elastic scale-out.
In addition to the above, the MSSQL provider offers the following benefits:
How to choose?
Here’s an image that highlights the key strengths of each storage provider:
For a comprehensive list of tradeoffs between the three providers, see “Comparing storage providers”.
Try it out!
It’s easy to configure Durable Functions to use the Netherite or MSSQL storage provider for local development or for apps running on Azure. See the following quickstarts for instructions:
Resources
Detailed documentation on architecture, scaling, performance, and more can be found on our github.io pages for Netherite and MSSQL.
Development of these new storage providers happens in the open just like other features of Durable Functions. You can find their source code in the durabletask-netherite and durabletask-mssql GitHub repos. You’re welcome to submit PRs if you’d like to contribute to the development of these new providers, as well as to the default Azure Storage provider.
If you have suggestions, questions, and/or bugs to report, you can file them as issues in the azure-functions-durable-extension repo.
Thank you for your interest in Azure Durable Functions!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.