Tip: running a cheap AKS lab

[stephaneeyskens]

I am currently running three AKS labs in one of my subscriptions and I constantly try to minimize costs so I'm quite regularly spinning a "cluster" with only one worker node (Standard DS2 V2) which I stop whenever I don't use it. Lately, when working with KEDA, I enabled the Virtual Nodes (see below) when creating the cluster:

which makes use of Azure Container Instances (ACI) behind the scenes. Then I realized that, I could leverage Virtual Nodes more often in order to reduce costs. Therefore, I thought of spinning up the cluster with the smallest possible Virtual Machine as the main worker node.

While  virtual machines represent a fixed cost, the pricing model of ACIs is based on actual consumption as they are only charged per second of execution which may be just enough for test scenarios/demos/lab-kind of activities. By default, if you do not specify any resource request in your container, AKS will provision ACIs with 1 CPU and 1.5 GB of memory. When specifying resources, for instance like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: serverless
    app.kubernetes.io/part-of: serverless
    app.kubernetes.io/version: v9
  name: serverless
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: serverless
  template:
    metadata:
      labels:
        app: serverless     
    spec:
      containers:
      - name: serverless       
        image: akspremiumregistry.azurecr.io/serverlesscontainer:1
        resources:
         requests:
          memory: "4000Mi"
          cpu: "4"       
      imagePullSecrets:
       - name: regsecret
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: azure
        effect: NoSchedule

, AKS applies the requested resources to the ACI config as illustrated here:

Benefits of such a setup

• Cost-friendly, providing you do not let ACIs hanging around
• Allows for resource-greedy scenarios 

Limitations of such a setup

Although interesting from a financial perspective, it comes with a few limitations:

• Suitable for lab-only activities
• Suitable for short-running containers (again, perfect in a demo scenario, etc.). When used like this, you might end up with only a few euros/dollars to pay at the end of the month instead of much more for a full blown worker node.
• Container startup is a little delayed because of the ACI provisioning.
• By default, only 100 concurrent ACIs are allowed per subscription, meaning that you can't have more than 100 concurrent pods. This can be changed by reaching out to support.
• Make sure to clean your stuff after use, else the ACIs will keep running forever which then will become very costly. The easiest way is to run a kubectl delete namespace .... but the bare minimum is to run a kubectl scale deploy --all --replicas=0 so as to let AKS destroy the corresponding ACIs. 

To wrap it up, to get a cheap AKS lab, just proceed the following steps:

• Create the cluster & enable Virtual Nodes
• Choose the smallest possible VM as your main worker node