Redis is a successful open source in-memory data structure store first released in 2009. It is most commonly used as a database, cache, and message broker. Developers enjoy Redis for its versatility and simplicity. Low cognitive load makes development fast and efficient.
Language support for Redis is very mature with clients across every major language one might expect, with multiple community-supported implementations per language (including Go, Java, Python and Rust), as well as support for advanced functionality such as Redis Cluster and Redis Sentinel, which enable sharding and master-replica topologies respectively.
Many other open source projects integrate with Redis. Just two examples in the cloud native ecosystem that we contribute to in the CNCF are Dapr (Distributed Application Runtime) and KEDA (Kubernetes event-driven autoscaling). Dapr supports Redis for both its state store and pub/sub broker components, and Redis is used by default for local development. KEDA has scalers for Redis that support Redis Lists and Redis Streams, including support for Redis Cluster and Redis Sentinel.
Redis is fully supported by managed cloud services such as Azure Cache for Redis which are often the first choice for customers deploying Redis open-source and Redis Enterprise in the cloud.
However, there are many valid reasons developers choose to deploy their own instances of Redis. Linux is the recommended deployment platform, and Docker is a seamless, cross-platform, option to run Redis locally for development purposes. Redis is especially popular on container-oriented cloud native platforms such as Kubernetes, including managed platforms such as Azure Kubernetes Service. Common reasons to deploy Redis using Docker or Kubernetes include:
- Portability – developers may want a truly vendor agnostic solution. An open source application, for example, can be packaged into a helm chart and be a single helm install away, on any Kubernetes cluster, anywhere.
- Local development – Whether using Docker Compose, or KIND (Kubernetes IN Docker), the benefits of a local, containerized, development environment which closely mirrors a production environment in the cloud can be significant.
- Scalability – Rather than remain static, it may make sense for Redis to scale with your application. Whether as a side-car cache to specific components of your application, or scaled in-cluster with the Pod Autoscaler, Kubernetes is often a great choice for scalable compute.
- Monitoring – Kubernetes has a rich observability ecosystem and the ability to monitor Redis with tools like Prometheus and Grafana is no exception.
Resilient and highly-available Redis
While Kubernetes can help make Redis more resilient – even a self-healing singleton Kubernetes service can be better than a stand-alone virtual machine – true high-availability is often desirable. Forary unavailability of Redis under modest load. However, under significant load, the loss of a cache may cause the application to fail.
In this post we explore how we can deploy a highly available Redis Cluster to Kubernetes.
In this post we will be using Azure Kubernetes Service (AKS) cluster which we will deploy using the Azure CLIthe Azure CLI installed before you continue.
Create a Resource Group
az group create --name my-aks \
--location eastus
Create an AKS cluster
az aks create --resource-group my-aks \
--name aks1 \
--node-count 3 \
--enable-addons monitoring \
--generate-ssh-keys
Install kubectl
if you do not have it installed already
az aks install-cli
Configure kubectl
to authenticate to your cluster
az aks get-credentials --resource-group my-aks \
--name aks1
Helm is the easiest way to deploy complex applications to Kubernetes. Bitnami provides helm charts for Redis as part of its Bitnami Stacks for Kubernetes which are fully supported on AKS, and actively maintained on GitHub.
Redis™ Cluster Packaged By Bitnami For Kubernetes provides two optio
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-redis bitnami/redis-cluster
u will be prompted to set a REDIS_PASSWORD
environment variable extracted from the Kubernetes secret. vironment variable (use via Windows System for Linux/macOS/Linux, etc). If you use another shell, adjust the syntax accordingly.
export REDIS_PASSWORD=$(kubectl get secret --namespace "default" my-redis-redis-cluster -o jsonpath="{.data.redis-password}" | base64 --decode)
Run a pod inside the cluster that will enable us to connect to Redis using the REDIS_PASSWORD
environment variable you set earlier.
kubectl run --namespace default my-redis-redis-cluster-client \
--rm --tty -i --restart='Never' \
--env REDIS_PASSWORD=$REDIS_PASSWORD \
--image docker.io/bitnami/redis-cluster:6.2.6-debian-10-r137 -- bash
Connect to the Redis cluster using redis-cli
. Note that the -c
switch is required when connecting to a cluster.
redis-cli -c -h my-redis-redis-cluster -a $REDIS_PASSWORD
Next run four set
commands from the CLI:
set one
set
set
set
You will see output that indicates the client is being redirected to multiple nodes.
10.244.0.6:6379> set one hello
-> Redirected to slot [9084] located at 10.244.1.5:6379
OK
10.244.1.5:6379> set two world
-> Redirected to slot [2127] located at 10.244.0.6:6379
OK
10.244.0.6:6379> set three goodbye
-> Redirected to slot [13861] located at 10.244.2.7:6379
OK
10.244.2.7:6379> set four world
-> Redirected to slot [8296] located at 10.244.1.5:6379
OK
The same will apply when you get
the values.
get one
get two
get three
get four
For a deeper dive into the functionality, Redis has an excellent Redis Cluster Tutorial which can you follow from the Playing with the cluster section. It includes sections on Writing an example app with redis-rb-cluster, Resharding the cluster, Scripting a resharding operation, A more interesting example application (with consistency-test.rb), and Testing the failover.
When testing a failover, choose a pod to delete.
kubectl get pods
You will see the pods as follows.
NAME READY STATUS RESTARTS AGE
my-redis-redis-cluster-0 1/1 Running 0 33m
my-redis-redis-cluster-1 1/1 Running 0 33m
my-redis-redis-cluster-2 1/1 Running 0 33m
my-redis-redis-cluster-3 1/1 Running 0 33m
my-redis-redis-cluster-4 1/1 Running 0 33m
my-redis-redis-cluster-5 1/1 Running 0 33m
my-redis-redis-cluster-client 1/1 Running 0 25m
Then delete a pod. It will be automatically re-created and you will see its status as ContainerCreating
.
kubectl delete pod my-redis-redis-cluster-3
Next, we will connect to our cluster with Python and Go.
The official Redis client for Python, redis/redis-py, now supports Redis Cluster. Previously Python users might have had experience with Grozken/redis-py-cluster which the official library’s support is now based on, with the original repo becoming a “laboratory for redis and cluster moving forward”.
Open a new terminal and set the REDIS_PASSWORD
environment variable.
export REDIS_PASSWORD=$(kubectl get secret --namespace "default" my-redis-redis-cluster -o jsonpath="{.data.redis-password}" | base64 --decode)
Run a python container interactively.
kubectl run --namespace default python-redis-cluster-test \
--rm --tty -i --restart='Never' \
--env REDIS_PASSWORD=$REDIS_PASSWORD \
--image python -- bash
Install the official redis/redis-py package using pip.
pip install redis
Run python in interactive mode.
python
Paste the following snippet at the prompt (>>>
).
from redis.cluster import RedisCluster as Redis
import os
REDIS_CLUSTER_NAME="my-redis-redis-cluster"
REDIS_CLUSTER_PORT=6379
REDIS_CLUSTER_PASSWORD=os.getenv('REDIS_PASSWORD')
from redis.cluster import RedisCluster as Redis
rc = Redis(host=REDIS_CLUSTER_NAME, port=REDIS_CLUSTER_PORT, password=REDIS_CLUSTER_PASSWORD)
Now you have a fully authenticated client, rc
.
Output the nodes.
rc.get_nodes()
You will see the nodes listed as follows.
>>> rc.get_nodes()
[[host=10.244.0.6,port=6379,name=10.244.0.6:6379,server_type=primary,redis_connection=Redis<ConnectionPool<Connection<host=10.244.0.6,port=6379,db=0>>>], [host=10.244.2.8,port=6379,name=10.244.2.8:6379,server_type=replica,redis_connection=Redis<ConnectionPool<Connection<host=10.244.2.8,port=6379,db=0>>>], [host=10.244.1.5,port=6379,name=10.244.1.5:6379,server_type=primary,redis_connection=Redis<ConnectionPool<Connection<host=10.244.1.5,port=6379,db=0>>>], [host=10.244.1.6,port=6379,name=10.244.1.6:6379,server_type=replica,redis_connection=Redis<ConnectionPool<Connection<host=10.244.1.6,port=6379,db=0>>>], [host=10.244.2.7,port=6379,name=10.244.2.7:6379,server_type=primary,redis_connection=Redis<ConnectionPool<Connection<host=10.244.2.7,port=6379,db=0>>>], [host=10.244.0.9,port=6379,name=10.244.0.9:6379,server_type=replica,redis_connection=Redis<ConnectionPool<Connection<host=10.244.0.9,port=6379,db=0>>>]]
Set and get some values.
rc.set("one", "goodbye")
rc.get("two")
Finally, create a loop that will lpush the current time to now
every two seconds.
import datetime, time
while True:
now = datetime.datetime.now().strftime("%H:%M:%S")
rc.lpush('now', now)
time.sleep(2)
You will see the length of the list increase.
>>> import datetime, time
>>> while True:
... now = datetime.datetime.now().strftime("%H:%M:%S")
... rc.lpush('now', now)
... time.sleep(2)
...
1
2
3
Switch to the terminal where you ran redis-cli
in the my-redis-redis-cluster-client
pod, and run rpop now
repeatedly.
10.244.0.6:6379> rpop now
"18:33:14"
10.244.0.6:6379> rpop now
"18:33:16"
10.244.0.6:6379> rpop now
"18:33:18"
10.244.0.6:6379>
Finally, we will deploy the pre-built container application that will log our new items. This provides an example of how to use the go-redis/redis package. This example uses the universal client.
We provided a pre-built container at ghcr.io/asw101/go-redis-sample:v1 and you can view the full source code of the sample at github.com/asw101/go-redis-sample. The repo includes a Dockerfile that uses distroless base image and helper scripts that allow you to run the same container to consume events (the default), produce events, and replicates the Python example above. You can see the code for the produce() and consume() functions in main.go, alongside a simple Set/Get example() from the original go-redis/redis quickstart.
Make sure you have the previous Python snippet above running in a terminal.
In another terminal window, we will set the REDIS_PASSWORD
environment variable as before, and also get the internal endpoints for the my-redis-redis-cluster
service which we will provide to the REDIS_ADDR
.
Set theREDIS_PASSWORD
andREDIS_ADDR
variables and then deploy the ghcr.io/asw101/go-redis-sample:v1
container.
export REDIS_PASSWORD=$(kubectl get secret --namespace "default" my-redis-redis-cluster -o jsonpath="{.data.redis-password}" | base64 --decode)
export REDIS_ADDR=$(kubectl get endpoints my-redis-redis-cluster -o=jsonpath='{.subsets[0].addresses[*].ip}')
kubectl run --namespace default go-redis-sample \
--rm --tty -i --restart='Never' \
--env REDIS_PASSWORD="$REDIS_PASSWORD" \
--env REDIS_ADDR="$REDIS_ADDR" \
--image ghcr.io/asw101/go-redis-sample:v1
To run the above as a producer, replacing the Python snippet above, you can use the do/kubectl-run-produce.sh script in the repo. do/docker-buildx.sh is an example of the Docker build command for the image.
The go-redis Universal Client lets us connect using single-node Client, a ClusterClient, or a FailoverClient, depending on whether a single endpoint, multiple endpoints, or a MasterName
are provided. However, unlike the Python client, this snippet does not automatically discover the endpoints, and we are using the internal endpoints of the service for testing purposes only – do not do this in production. Finally, you may wonder why we cannot use kubectl port-forward
to the my-redis-redis-cluster
service and use that for local testing. This is because the client will attempt to connect to each shard directly, which will not be possible as your machine will not be able to reach those endpoints via the port-forward.
Finally, if you want to continue to hack on this inside the cluster, without any other dependencies, you could run asw101/nvgo container insider your cluster. It uses neovim, gopls language server in the official golang Docker image.
In this post you have deployed Redis Cluster to your AKS cluster with helm, explored it interactively with the Redis CLI and Python, as well as a pre-built Go container.
At this stage, you may wish to build another container in the language of your choice to future explore Redis within your Kubernetes environment. Or perhaps even try it with Dapr or KEDA.
Once you have finished exploring, you should delete the my-aks
resource group for your AKS cluster to avoid any further charges.
az group delete -n my-aks
by Aaron Wislang (@as_w)
Redis is a successful open source in-memory data structure store first released in 2009. It is most commonly used as a database, cache, and message broker. Developers enjoy Redis for its versatility and simplicity. Low cognitive load makes development fast and efficient.
Language support for Redis is very mature with clients across every major language one might expect, with multiple community-supported implementations per language (including Go, Java, Python and Rust), as well as support for advanced functionality such as Redis Cluster and Redis Sentinel, which enable sharding and master-replica topologies respectively.
Many other open source projects integrate with Redis. Just two examples in the cloud native ecosystem that we contribute to in the CNCF are Dapr (Distributed Application Runtime) and KEDA (Kubernetes event-driven autoscaling). Dapr supports Redis for both its state store and pub/sub broker components, and Redis is used by default for local development. KEDA has scalers for Redis that support Redis Lists and Redis Streams, including support for Redis Cluster and Redis Sentinel.
Why run Redis containerized, or on Kubernetes?
Redis is fully supported by managed cloud services such as Azure Cache for Redis which are often the first choice for customers deploying Redis open-source and Redis Enterprise in the cloud.
However, there are many valid reasons developers choose to deploy their own instances of Redis. Linux is the recommended deployment platform, and Docker is a seamless, cross-platform, option to run Redis locally for development purposes. Redis is especially popular on container-oriented cloud native platforms such as Kubernetes, including managed platforms such as Azure Kubernetes Service. Common reasons to deploy Redis using Docker or Kubernetes include:
Resilient and highly-available Redis
While Kubernetes can help make Redis more resilient – even a self-healing singleton Kubernetes service can be better than a stand-alone virtual machine – true high-availability is often desirable. For example, if you are using Redis as a cache, your application may be able to survive the temporary unavailability of Redis under modest load. However, under significant load, the loss of a cache may cause the application to fail.
In this post we explore how we can deploy a highly available Redis Cluster to Kubernetes.
Deploy AKS
In this post we will be using Azure Kubernetes Service (AKS) cluster which we will deploy using the Azure CLI. Make sure you have the Azure CLI installed before you continue.
Create a Resource Group
Create an AKS cluster
Install
kubectl
if you do not have it installed alreadyConfigure
kubectl
to authenticate to your clusterDeploy Redis Cluster using the Bitnami Helm chart repository
Helm is the easiest way to deploy complex applications to Kubernetes. Bitnami provides helm charts for Redis as part of its Bitnami Stacks for Kubernetes which are fully supported on AKS, and actively maintained on GitHub.
Redis™ Cluster Packaged By Bitnami For Kubernetes provides two options for deploying a production-grade Redis deployment on Kubernetes. Redis Cluster enables sharing and disaster recovery, but supports a single database, and Redis Sentinel uses a master-replica, with a single master, and supports multiple databases. You can read more about replication in the Redis documentation. Let’s install it.
Add the bitnami helm chart repository.
Install the redis-cluster helm chart.
You will be prompted to set a
REDIS_PASSWORD
environment variable extracted from the Kubernetes secret. Note this is a bash environment variable (use via Windows System for Linux/macOS/Linux, etc). If you use another shell, adjust the syntax accordingly.export REDIS_PASSWORD=$(kubectl get secret --namespace "default" my-redis-redis-cluster -o jsonpath="{.data.redis-password}" | base64 --decode)
Explore with redis-cli
Run a pod inside the cluster that will enable us to connect to Redis using the
REDIS_PASSWORD
environment variable you set earlier.kubectl run --namespace default my-redis-redis-cluster-client \ --rm --tty -i --restart='Never' \ --env REDIS_PASSWORD=$REDIS_PASSWORD \ --image docker.io/bitnami/redis-cluster:6.2.6-debian-10-r137 -- bash
Connect to the Redis cluster using
redis-cli
. Note that the-c
switch is required when connecting to a cluster.redis-cli -c -h my-redis-redis-cluster -a $REDIS_PASSWORD
Next run four
set
commands from the CLI:set one hello set two world set three goodbye set four world
You will see output that indicates the client is being redirected to multiple nodes.
10.244.0.6:6379> set one hello -> Redirected to slot [9084] located at 10.244.1.5:6379 OK 10.244.1.5:6379> set two world -> Redirected to slot [2127] located at 10.244.0.6:6379 OK 10.244.0.6:6379> set three goodbye -> Redirected to slot [13861] located at 10.244.2.7:6379 OK 10.244.2.7:6379> set four world -> Redirected to slot [8296] located at 10.244.1.5:6379 OK
The same will apply when you
get
the values.For a deeper dive into the functionality, Redis has an excellent Redis Cluster Tutorial which can you follow from the Playing with the cluster section. It includes sections on Writing an example app with redis-rb-cluster, Resharding the cluster, Scripting a resharding operation, A more interesting example application (with consistency-test.rb), and Testing the failover.
When testing a failover, choose a pod to delete.
You will see the pods as follows.
NAME READY STATUS RESTARTS AGE my-redis-redis-cluster-0 1/1 Running 0 33m my-redis-redis-cluster-1 1/1 Running 0 33m my-redis-redis-cluster-2 1/1 Running 0 33m my-redis-redis-cluster-3 1/1 Running 0 33m my-redis-redis-cluster-4 1/1 Running 0 33m my-redis-redis-cluster-5 1/1 Running 0 33m my-redis-redis-cluster-client 1/1 Running 0 25m
Then delete a pod. It will be automatically re-created and you will see its status as
ContainerCreating
.Next, we will connect to our cluster with Python and Go.
Explore with Python
The official Redis client for Python, redis/redis-py, now supports Redis Cluster. Previously Python users might have had experience with Grozken/redis-py-cluster which the official library’s support is now based on, with the original repo becoming a “laboratory for redis and cluster moving forward”.
Open a new terminal and set the
REDIS_PASSWORD
environment variable.export REDIS_PASSWORD=$(kubectl get secret --namespace "default" my-redis-redis-cluster -o jsonpath="{.data.redis-password}" | base64 --decode)
Run a python container interactively.
kubectl run --namespace default python-redis-cluster-test \ --rm --tty -i --restart='Never' \ --env REDIS_PASSWORD=$REDIS_PASSWORD \ --image python -- bash
Install the official redis/redis-py package using pip.
Run python in interactive mode.
Paste the following snippet at the prompt (
>>>
).from redis.cluster import RedisCluster as Redis import os REDIS_CLUSTER_NAME="my-redis-redis-cluster" REDIS_CLUSTER_PORT=6379 REDIS_CLUSTER_PASSWORD=os.getenv('REDIS_PASSWORD') from redis.cluster import RedisCluster as Redis rc = Redis(host=REDIS_CLUSTER_NAME, port=REDIS_CLUSTER_PORT, password=REDIS_CLUSTER_PASSWORD)
Now you have a fully authenticated client,
rc
.Output the nodes.
You will see the nodes listed as follows.
>>> rc.get_nodes() [[host=10.244.0.6,port=6379,name=10.244.0.6:6379,server_type=primary,redis_connection=Redis<ConnectionPool<Connection<host=10.244.0.6,port=6379,db=0>>>], [host=10.244.2.8,port=6379,name=10.244.2.8:6379,server_type=replica,redis_connection=Redis<ConnectionPool<Connection<host=10.244.2.8,port=6379,db=0>>>], [host=10.244.1.5,port=6379,name=10.244.1.5:6379,server_type=primary,redis_connection=Redis<ConnectionPool<Connection<host=10.244.1.5,port=6379,db=0>>>], [host=10.244.1.6,port=6379,name=10.244.1.6:6379,server_type=replica,redis_connection=Redis<ConnectionPool<Connection<host=10.244.1.6,port=6379,db=0>>>], [host=10.244.2.7,port=6379,name=10.244.2.7:6379,server_type=primary,redis_connection=Redis<ConnectionPool<Connection<host=10.244.2.7,port=6379,db=0>>>], [host=10.244.0.9,port=6379,name=10.244.0.9:6379,server_type=replica,redis_connection=Redis<ConnectionPool<Connection<host=10.244.0.9,port=6379,db=0>>>]]
Set and get some values.
rc.set("one", "goodbye") rc.get("two")
Finally, create a loop that will lpush the current time to
now
every two seconds.import datetime, time while True: now = datetime.datetime.now().strftime("%H:%M:%S") rc.lpush('now', now) time.sleep(2)
You will see the length of the list increase.
>>> import datetime, time >>> while True: ... now = datetime.datetime.now().strftime("%H:%M:%S") ... rc.lpush('now', now) ... time.sleep(2) ... 1 2 3
Switch to the terminal where you ran
redis-cli
in themy-redis-redis-cluster-client
pod, and runrpop now
repeatedly.10.244.0.6:6379> rpop now "18:33:14" 10.244.0.6:6379> rpop now "18:33:16" 10.244.0.6:6379> rpop now "18:33:18" 10.244.0.6:6379>
Explore with Go
Finally, we will deploy the pre-built container application that will log our new items. This provides an example of how to use the go-redis/redis package. This example uses the universal client.
We provided a pre-built container at ghcr.io/asw101/go-redis-sample:v1 and you can view the full source code of the sample at github.com/asw101/go-redis-sample. The repo includes a Dockerfile that uses distroless base image and helper scripts that allow you to run the same container to consume events (the default), produce events, and replicates the Python example above. You can see the code for the produce() and consume() functions in main.go, alongside a simple Set/Get example() from the original go-redis/redis quickstart.
Make sure you have the previous Python snippet above running in a terminal.
In another terminal window, we will set the
REDIS_PASSWORD
environment variable as before, and also get the internal endpoints for themy-redis-redis-cluster
service which we will provide to theREDIS_ADDR
.REDIS_PASSWORD
andREDIS_ADDR
variables and then deploy theghcr.io/asw101/go-redis-sample:v1
container.export REDIS_PASSWORD=$(kubectl get secret --namespace "default" my-redis-redis-cluster -o jsonpath="{.data.redis-password}" | base64 --decode) export REDIS_ADDR=$(kubectl get endpoints my-redis-redis-cluster -o=jsonpath='{.subsets[0].addresses[*].ip}') kubectl run --namespace default go-redis-sample \ --rm --tty -i --restart='Never' \ --env REDIS_PASSWORD="$REDIS_PASSWORD" \ --env REDIS_ADDR="$REDIS_ADDR" \ --image ghcr.io/asw101/go-redis-sample:v1
To run the above as a producer, replacing the Python snippet above, you can use the do/kubectl-run-produce.sh script in the repo. do/docker-buildx.sh is an example of the Docker build command for the image.
The go-redis Universal Client lets us connect using single-node Client, a ClusterClient, or a FailoverClient, depending on whether a single endpoint, multiple endpoints, or a
MasterName
are provided. However, unlike the Python client, this snippet does not automatically discover the endpoints, and we are using the internal endpoints of the service for testing purposes only – do not do this in production. Finally, you may wonder why we cannot usekubectl port-forward
to themy-redis-redis-cluster
service and use that for local testing. This is because the client will attempt to connect to each shard directly, which will not be possible as your machine will not be able to reach those endpoints via the port-forward.Finally, if you want to continue to hack on this inside the cluster, without any other dependencies, you could run asw101/nvgo container insider your cluster. It uses neovim, gopls language server in the official golang Docker image.
Summary
In this post you have deployed Redis Cluster to your AKS cluster with helm, explored it interactively with the Redis CLI and Python, as well as a pre-built Go container.
At this stage, you may wish to build another container in the language of your choice to future explore Redis within your Kubernetes environment. Or perhaps even try it with Dapr or KEDA.
Once you have finished exploring, you should delete the
my-aks
resource group for your AKS cluster to avoid any further charges.by Aaron Wislang (@as_w)