How to Refresh Expired Keys in Redis using Azure Functions

Microsoft

May 06, 2024

Introduction

Redis is an extremely popular in-memory data store that is used as a cache, session store, message broker, and more. One of the features that makes Redis really useful is the ability to set a time-to-live (TTL) value for any key. A key with a TTL value will be automatically deleted from Redis when the TTL expires. This helps reduce memory usage and ensure data freshness.

For instance, you might be storing information like inventory or pricing for your e-commerce site in Redis. This is a perfect application for caching in Redis. This type of data will be queried constantly and serving it directly from a database will either slow down your application or make your database implementation very expensive to handle the load! Using a CDN is out of the question because pricing and inventory data isn’t static. Redis can handle the throughput at low latencies, boosting the performance of your site.

There’s one problem, however—how can you be sure the price or inventory data in the cache is up to date with the official value in the database? This is a classic cache invalidation problem. TTL functionality in Redis solves half of the problem. By setting a TTL, you can ensure that stale values are periodically purged. But the other half of the problem remains—how do you gracefully update the cache with the latest value? That’s where serverless functions come in. This blog post will show you how to use the Redis bindings for Azure Functions to detect key expirations and automatically refresh the cache with the latest values.

Key Expiration and TTL in Redis

Redis has native support for several key expiration mechanisms, which enables you to have precise control over the TTL of data. Expiration commands include:

EXPIRE, which sets the TTL of a Redis key in seconds.
PEXPIRE, which sets the TLL of a Redis key in milliseconds.
EXPIREAT, which sets keys to expire at an absolute time in the future, using UNIX timestamps.
PEXPIREAT, which is the same as EXPIREAT, but using milliseconds.

Redis also has several commands that can be used to return the remaining time to live for a key, including TTL, PTTL, EXPIRETIME, and PEXPIRETIME.

So, for example, if you have a key named “apple” that is storing the apple’s price, you could use the following command to expire the price value after 60 seconds:

EXPIRE apple 60

Or this command to expire the price on December 25^th, 2024:

EXPIREAT apple 1735138800

Keyspace Notifications

Expiration of stale keys is a great feature of Redis, but it still means you must check for expired keys and periodically refresh them. That’s where keyspace notifications can be used. This nifty feature will generate a notification each time an expiration event occurs. Specifically, it uses a special pub/sub channel within your cache to publish these events. This is a great feature because you can monitor the pub/sub channel and know when your keys are expiring, then proactively update them. There are a couple of things to bear in mind when using keyspace notifications:

First, you need to explicitly enable keyspace notifications since they are not enabled by default. Second, there is a performance impact from keyspace notifications, especially if your cache is firing off a lot of events. Third, keyspace notifications are fire-and-forget, which means that Redis does not guarantee the delivery or the order of the messages. You should not rely on keyspace notifications for critical or sensitive operations. Finally, keyspace notifications are asynchronous, which means that there may be some delay or inconsistency between the actual event and the notification.

Redis Triggers and Bindings for Azure Functions

Keyspace notifications are a powerful feature, but monitoring the notification channel can be a challenge. Fortunately, serverless functions offer an excellent (and low-cost) way to monitor and act upon these events. Even better, there are now triggers and bindings that allow Azure Cache for Redis to be used seamlessly with Azure Functions. This means you can trigger and automatically execute an Azure Function based on activity on your Redis cache. And, among other things, you can trigger on keyspace notifications! That means that, every time a key expires on your cache, you can trigger a Function that will pull in the latest value and update the expired key.

Putting it all Together

Ultimately, the architecture we’re looking for is something like this:

In this blog post, we’ll focus on three components using a very simple e-commerce use-case:

A Redis instance that will host a copy of price data for grocery items (using Azure Cache for Redis)
A primary database that will serve as the “source of truth” for the price of each item (using Azure Cosmos DB)
A serverless function that will detect key expirations in Redis, query the latest value in the database, and update the Redis instance (using Azure Functions)

Sample Code

To make things easier, the code for this sample is available on GitHub. To run the example, you’ll need:

The Azure Developer CLI makes setting up the example simple. Follow these steps to get the demo up and running on Azure:

Open a terminal/shell
Copy the repo to a location of your choice (e.g. git clone https://github.com/MSFTeegarden/ExpirationTrigger)
In the terminal, change the directory to the project folder
Run:
```
azd up
```
Follow the instructions to enter an environment name, select an Azure subscription, and choose an Azure region.

The Azure Developer CLI automation will kick in and start provisioning the resources you need in a new resource group. The following Azure resources will be created:

Azure Cache for Redis
Azure Cosmos DB
App Service plan
Function app
Storage Account
Application Insights

It will take 15-25 min for provisioning to finish. Pricing for these resources will depend on your region and usage, of course. But for me, it was around $2-$4 per day.

The automation will also enable keyspace notifications on your Redis instance, add a new database and container to your Azure Cosmos DB instance, and add connection strings for both Azure Cosmos DB and Azure Cache for Redis to your Function app environment variables. Finally, it deploys sample code to the functions instance. This is what the code looks like:

using Microsoft.Extensions.Logging;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Extensions.Redis;
using Microsoft.Azure.Cosmos;
using Microsoft.Azure.Cosmos.Linq;

public class RedisTrigger
{
    private readonly ILogger<RedisTrigger> logger;

    public RedisTrigger(ILogger<RedisTrigger> logger)
    {
        this.logger = logger;
    }

    // This function will be invoked when a key expires in Redis. It then queries Cosmos DB for the key and its value. Finally, it uses an Azure Function output binding to write the updated key and value back to Redis .
   [Function("ExpireTrigger")]
   [RedisOutput(Common.connectionString, "SET")] //Redis output binding. This specifies that the returned value will be written to Redis using the SET command.

    //Trigger on Redis key expiration, and use an CosmosDB input binding to query the key and value from Cosmos DB.
    public static async Task<String> ExpireTrigger(
        [RedisPubSubTrigger(Common.connectionString, "__keyevent@0__:expired")] Common.ChannelMessage channelMessage,
        FunctionContext context,
        [CosmosDBInput(
            "myDatabase", //Cosmos DB database name
            "Inventory", //Cosmos DB container name
            Connection = "CosmosDbConnection" //Parameter name for the connection string in a environmental variable or local.settings.json
            )] Container container
        )    
        {
            var logger = context.GetLogger("RedisTrigger");
            var redisKey = channelMessage.Message; //The key that has expired in Redis
            logger.LogInformation($"Key '{redisKey}' has expired.");

            //Query Cosmos DB for the key and value
            IOrderedQueryable<CosmosData> queryable = container.GetItemLinqQueryable<CosmosData>();
            using FeedIterator<CosmosData> feed = queryable
                .Where(b => b.item == redisKey) //item name must be the same as the key in Redis
                .ToFeedIterator<CosmosData>();
            FeedResponse<CosmosData> response = await feed.ReadNextAsync();

            CosmosData data = response.FirstOrDefault(defaultValue: null);
            if (data != null)
            {
                logger.LogInformation($"Key: \"{data.item}\", Value: \"{data.price}\" added to Redis.");
                return $"{redisKey} {data.price}"; //Return the key and value to be written to Redis
            }
            else
            {
                logger.LogInformation($"Key not found");
                return $"{redisKey} false"; //set the value of the key to "false" if not found in Cosmos DB
            }
        }
}

With data models contained in the Common.cs file:

public class Common
{
    public const string connectionString = "redisConnectionString";

    public class ChannelMessage
    {
        public string SubscriptionChannel { get; set; }
        public string Channel { get; set; }
        public string Message { get; set; }
    }
}

public record CosmosData(
        string id,
        string item,
        string price
);

The example has several key elements:

A Redis pub/sub trigger is used to monitor expiration key events, using the “__keyevent@0:expired” keyspace notification channel. This will execute the function each time an expiration event is detected.
A Cosmos DB Input binding is used to connect to the database—an easy way to manage the connection and read data on the database.
A Redis output binding is used to write new data to your Redis instance, saving the hassle of importing a Redis library and establishing a connection.

Each time the function is executed, the example will record the name of the key that expired, search for the value of this key within a specified Cosmos DB container, and then write the value from Cosmos DB back to Redis.

Running the Example

To finish setting up the example, you need to add some items to the Cosmos database.

Navigate to your Cosmos DB resource in the Azure Portal, and go to the Data Explorer menu. You’ll see a database named “myDatabase” and a container named “Inventory” which have already been created.

Select the Items entry under “Inventory” and then select New Item.

Enter in an id of your choosing, and then add entries for item and price, then select Save.

You can add as many items as you’d like—this will provide the source of truth price for entries in your Redis cache.

Next, navigate to your functions app resource in the portal and select the Log stream menu. This will allow you to monitor the execution of your functions trigger.

Finally, open a connection to your Redis instance. You can use the redis-cli command line tool, the built-in console in the Azure portal, or a standalone Redis GUI like Redis Insights.

When you have a connection to Redis open, first the value of a key, for example: