%3CLINGO-SUB%20id%3D%22lingo-sub-1410051%22%20slang%3D%22en-US%22%3ERe%3A%20Tip%3A%20running%20a%20cheap%20AKS%20lab%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1410051%22%20slang%3D%22en-US%22%3E%3CP%3EThanks.%20Definitely%20worth%20sharing%3B%20I%20always%20appreciate%20ways%20to%20stretch%20my%20monthly%20credit%20as%20far%20as%20possible.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1407997%22%20slang%3D%22en-US%22%3ETip%3A%20running%20a%20cheap%20AKS%20lab%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1407997%22%20slang%3D%22en-US%22%3E%3CP%3EI%20am%20currently%20running%20three%20AKS%20labs%20in%20one%20of%20my%20subscriptions%20and%20I%20constantly%20try%20to%20minimize%20costs%20so%20I'm%20quite%20regularly%20spinning%20a%20%22cluster%22%20with%20only%20one%20worker%20node%20(Standard%20DS2%20V2)%20which%20I%20stop%20whenever%20I%20don't%20use%20it.%20Lately%2C%20when%20working%20with%20KEDA%2C%20I%20enabled%20the%20Virtual%20Nodes%20(see%20below)%20when%20creating%20the%20cluster%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22stephaneey_0-1590051010682.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F193508i6C1980F2F076C4A2%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22stephaneey_0-1590051010682.png%22%20alt%3D%22stephaneey_0-1590051010682.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3Ewhich%20makes%20use%20of%20Azure%20Container%20Instances%20(ACI)%20behind%20the%20scenes.%20Then%20I%20realized%20that%2C%20I%20could%20leverage%20Virtual%20Nodes%20more%20often%20in%20order%20to%20reduce%20costs.%20Therefore%2C%20I%20thought%20of%20spinning%20up%20the%20cluster%20with%20the%20smallest%20possible%20Virtual%20Machine%20as%20the%20main%20worker%20node.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWhile%26nbsp%3B%20virtual%20machines%20represent%20a%20fixed%20cost%2C%20the%20pricing%20model%20of%20ACIs%20is%20based%20on%20actual%20consumption%20as%20they%20are%20only%20charged%20per%20second%20of%20execution%20which%20may%20be%20just%20enough%20for%20test%20scenarios%2Fdemos%2Flab-kind%20of%20activities.%20By%20default%2C%20if%20you%20do%20not%20specify%20any%20resource%20request%20in%20your%20container%2C%20AKS%20will%20provision%20ACIs%20with%201%20CPU%20and%201.5%20GB%20of%20memory.%20When%20specifying%20resources%2C%20for%20instance%20like%20this%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-yaml%22%3E%3CCODE%3EapiVersion%3A%20apps%2Fv1%0Akind%3A%20Deployment%0Ametadata%3A%0A%20%20labels%3A%0A%20%20%20%20app.kubernetes.io%2Fname%3A%20serverless%0A%20%20%20%20app.kubernetes.io%2Fpart-of%3A%20serverless%0A%20%20%20%20app.kubernetes.io%2Fversion%3A%20v9%0A%20%20name%3A%20serverless%0A%20%20namespace%3A%20default%0Aspec%3A%0A%20%20replicas%3A%201%0A%20%20selector%3A%0A%20%20%20%20matchLabels%3A%0A%20%20%20%20%20%20app%3A%20serverless%0A%20%20template%3A%0A%20%20%20%20metadata%3A%0A%20%20%20%20%20%20labels%3A%0A%20%20%20%20%20%20%20%20app%3A%20serverless%20%20%20%20%20%0A%20%20%20%20spec%3A%0A%20%20%20%20%20%20containers%3A%0A%20%20%20%20%20%20-%20name%3A%20serverless%20%20%20%20%20%20%20%0A%20%20%20%20%20%20%20%20image%3A%20akspremiumregistry.azurecr.io%2Fserverlesscontainer%3A1%0A%20%20%20%20%20%20%20%20resources%3A%0A%20%20%20%20%20%20%20%20%20requests%3A%0A%20%20%20%20%20%20%20%20%20%20memory%3A%20%224000Mi%22%0A%20%20%20%20%20%20%20%20%20%20cpu%3A%20%224%22%20%20%20%20%20%20%20%0A%20%20%20%20%20%20imagePullSecrets%3A%0A%20%20%20%20%20%20%20-%20name%3A%20regsecret%0A%20%20%20%20%20%20tolerations%3A%0A%20%20%20%20%20%20-%20key%3A%20virtual-kubelet.io%2Fprovider%0A%20%20%20%20%20%20%20%20operator%3A%20Equal%0A%20%20%20%20%20%20%20%20value%3A%20azure%0A%20%20%20%20%20%20%20%20effect%3A%20NoSchedule%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%2C%20AKS%20applies%20the%20requested%20resources%20to%20the%20ACI%20config%20as%20illustrated%20here%3A%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22stephaneey_0-1590058095304.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F193527iACB45907A61E1D2A%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22stephaneey_0-1590058095304.png%22%20alt%3D%22stephaneey_0-1590058095304.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CFONT%20size%3D%226%22%3EBenefits%20of%20such%20a%20setup%3C%2FFONT%3E%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3ECost-friendly%2C%20providing%20you%20do%20not%20let%20ACIs%20hanging%20around%3C%2FLI%3E%0A%3CLI%3EAllows%20for%20resource-greedy%20scenarios%26nbsp%3B%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CFONT%20size%3D%226%22%3ELimitations%20of%20such%20a%20setup%3C%2FFONT%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAlthough%20interesting%20from%20a%20financial%20perspective%2C%20it%20comes%20with%20a%20few%20limitations%3A%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3ESuitable%20for%20lab-only%20activities%3C%2FLI%3E%0A%3CLI%3ESuitable%20for%20short-running%20containers%20(again%2C%20perfect%20in%20a%20demo%20scenario%2C%20etc.).%20When%20used%20like%20this%2C%20you%20might%20end%20up%20with%20only%20a%20few%20euros%2Fdollars%20to%20pay%20at%20the%20end%20of%20the%20month%20instead%20of%20much%20more%20for%20a%20full%20blown%20worker%20node.%3C%2FLI%3E%0A%3CLI%3EContainer%20startup%20is%20a%20little%20delayed%20because%20of%20the%20ACI%20provisioning.%3C%2FLI%3E%0A%3CLI%3EBy%20default%2C%20only%20100%20concurrent%20ACIs%20are%20allowed%20per%20subscription%2C%20meaning%20that%20you%20can't%20have%20more%20than%20100%20concurrent%20pods.%20This%20can%20be%20changed%20by%20reaching%20out%20to%20support.%3C%2FLI%3E%0A%3CLI%3EMake%20sure%20to%20clean%20your%20stuff%20after%20use%2C%20else%20the%20ACIs%20will%20keep%20running%20forever%20which%20then%20will%20become%20very%20costly.%20The%20easiest%20way%20is%20to%20run%20a%20%3CEM%3Ekubectl%20delete%20namespace%20....%3C%2FEM%3E%20but%20the%20bare%20minimum%20is%20to%20run%20a%20%3CEM%3Ekubectl%20scale%20deploy%20--all%20--replicas%3D0%3C%2FEM%3E%20so%20as%20to%20let%20AKS%20destroy%20the%20corresponding%20ACIs.%26nbsp%3B%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3ETo%20wrap%20it%20up%2C%20to%20get%20a%20cheap%20AKS%20lab%2C%20just%20proceed%20the%20following%20steps%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3ECreate%20the%20cluster%20%26amp%3B%20enable%20Virtual%20Nodes%3C%2FLI%3E%0A%3CLI%3EChoose%20the%20smallest%20possible%20VM%20as%20your%20main%20worker%20node%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1407997%22%20slang%3D%22en-US%22%3E%3CP%3EWant%20to%20play%20with%20AKS%20but%20don't%20want%20to%20spend%20too%20much%20money%3F%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1407997%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAKS%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E

I am currently running three AKS labs in one of my subscriptions and I constantly try to minimize costs so I'm quite regularly spinning a "cluster" with only one worker node (Standard DS2 V2) which I stop whenever I don't use it. Lately, when working with KEDA, I enabled the Virtual Nodes (see below) when creating the cluster:

 

stephaneey_0-1590051010682.png

which makes use of Azure Container Instances (ACI) behind the scenes. Then I realized that, I could leverage Virtual Nodes more often in order to reduce costs. Therefore, I thought of spinning up the cluster with the smallest possible Virtual Machine as the main worker node.

 

While  virtual machines represent a fixed cost, the pricing model of ACIs is based on actual consumption as they are only charged per second of execution which may be just enough for test scenarios/demos/lab-kind of activities. By default, if you do not specify any resource request in your container, AKS will provision ACIs with 1 CPU and 1.5 GB of memory. When specifying resources, for instance like this:

 

 

 

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: serverless
    app.kubernetes.io/part-of: serverless
    app.kubernetes.io/version: v9
  name: serverless
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: serverless
  template:
    metadata:
      labels:
        app: serverless     
    spec:
      containers:
      - name: serverless       
        image: akspremiumregistry.azurecr.io/serverlesscontainer:1
        resources:
         requests:
          memory: "4000Mi"
          cpu: "4"       
      imagePullSecrets:
       - name: regsecret
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: azure
        effect: NoSchedule

 

 

 

, AKS applies the requested resources to the ACI config as illustrated here:

stephaneey_0-1590058095304.png

 

Benefits of such a setup

  • Cost-friendly, providing you do not let ACIs hanging around
  • Allows for resource-greedy scenarios 

 

Limitations of such a setup

 

Although interesting from a financial perspective, it comes with a few limitations:

  • Suitable for lab-only activities
  • Suitable for short-running containers (again, perfect in a demo scenario, etc.). When used like this, you might end up with only a few euros/dollars to pay at the end of the month instead of much more for a full blown worker node.
  • Container startup is a little delayed because of the ACI provisioning.
  • By default, only 100 concurrent ACIs are allowed per subscription, meaning that you can't have more than 100 concurrent pods. This can be changed by reaching out to support.
  • Make sure to clean your stuff after use, else the ACIs will keep running forever which then will become very costly. The easiest way is to run a kubectl delete namespace .... but the bare minimum is to run a kubectl scale deploy --all --replicas=0 so as to let AKS destroy the corresponding ACIs. 

To wrap it up, to get a cheap AKS lab, just proceed the following steps:

  • Create the cluster & enable Virtual Nodes
  • Choose the smallest possible VM as your main worker node

 

1 Comment
Occasional Contributor

Thanks. Definitely worth sharing; I always appreciate ways to stretch my monthly credit as far as possible.