How do AKS and AKS on Azure Stack HCI compare?

[scooley]

Hey! I’m Abhilasha, a PM on the AKS-HCI team. In this blog, I wanted to share some of the similarities and differences between AKS-HCI and AKS.

AKS on Azure and has been generally available for years. AKS on Azure Stack HCI is an on-premises implementation of AKS, it runs in customer environments on customer managed hardware. We basically want to achieve world peace and support running AKS everywhere outside of Azure.

Feature Set	AKS-HCI	AKS
Kubernetes Management Cluster (lifecycle operations)
Who manages the management cluster?	Split management – mostly managed by the service with some admin config (networking, required certs, etc).	Managed by AKS; hidden from your view
Where is the management cluster located?	In your datacenter. Currently, we support: Azure Stack HCI 21H2 Windows Server 2019 Datacenter Windows Server 2022 Datacenter	Azure cloud; hidden from your view
Well, who manages the hardware where this is running?	All physical hardware things – storage, network, servers, etc are managed by you	Storage, network, etc are created and managed for you when you create an AKS cluster; visible to you
How do you update the management cluster?	PS commands initiated by you (Update-Akshci) Windows Admin Center GUI initiated by you Note: Workload cluster update is covered in the below section.	Azure managed updates; hidden from customer view
Kubernetes Target Cluster (lifecycle operations)
Who manages the cluster?	Managed by you	Managed by you
Where is the cluster located?	In your datacenter alongside your AKS-HCI management cluster	Azure cloud
K8s cluster lifecycle management (create, scale, update and delete clusters)	PowerShell (PS) Windows Admin Center (WAC)	Az CLI Az PowerShell Azure Portal ARM templates Terraform Pulumi and many more!
Can you use kubectl and other open-source Kubernetes tools?	Yes! AKS-HCI is a Cloud Native Computing Foundation (CNCF) certified Kubernetes distribution	Yes! AKS is a Cloud Native Computing Foundation (CNCF) certified Kubernetes distribution
Workload cluster updates	K8s version upgrade through PowerShell or WAC. Initiated by you. node OS image update initiated by you; Updates in a target cluster happen at the cluster level – control plane nodes + node pools updated.	Azure CLI, Azure PS, Portal, ARM templates, GitHub Actions; OS image patch upgrade; Automatic upgrades; Planned maintenance windows;
Kubernetes versions	~same as AKS: 1.19.11, 1.19.13, 1.20.7, 1.20.9 (default), 1.21.1, 1.21.2 (as of 11/5/2021)	1.19.11, 1.19.13, 1.20.7, 1.20.9 (default), 1.21.1, 1.21.2, 1.22.1, 1.22.2 (as of 11/5/2021)
Can you start/stop K8s clusters to save costs?	Yes, by stopping the underlying failover cluster	Yes, by using Az CLI commands or Azure Portal
Node Pools
Do you support running Linux and Windows node pools in the same cluster?	Yes! Linux nodes: CBL-Mariner Windows nodes: Windows Server 2019 Datacenter	Yes! Linux nodes: Ubuntu 18.04 Windows nodes: Windows Server 2019 Datacenter
What’s your container runtime?	Linux nodes: containerd Windows nodes: Docker	Linux nodes: containerd Windows nodes: Docker (default), containerd (preview)
Can you scale node pools?	Yes! No auto-scaler yet	Yes! You can auto scale node pools in AKS Scale down a cluster without deleting the K8s nodes
What about virtual nodes?	No, we do not have support for virtual nodes yet	Yes! Virtual nodes are only supported with Linux pods and nodes. This is based on the open-source project Virtual Kubelet.
Can you upgrade a node pool?	We do not support upgrading individual node pools. All upgrades happen at the K8s cluster level.	You can perform node pool specific upgrades in an AKS cluster.
Networking
Who creates and manages the networks?	All networks (for both the management cluster and target K8s clusters) are created and managed by you	By default, Azure creates the virtual network and subnet for you. You can also choose an existing virtual network to create your AKS clusters
What type of network options are supported?	Support logical networks: DHCP networks with/without VLAN ID Static IP networks with/without VLAN ID We do not have SDN support yet	Support Azure virtual networks in 2 network models: Kubenet (default) – Networks created alongside AKS Azure-CNI – use existing virtual networks and configurations
Load balancer	HAProxy – runs in a separate VM in the target K8s cluster kubeVIP – runs as a K8s service in the control plane K8s node Load balancers are always given Static IP addresses to ensure application and K8s cluster availability	Azure load balancer – Basic SKU or Standard SKU Can also use internal load balancer By default, load balancer IP address is tied to load balancer ARM resource. You can also assign a static public IP address directly to your Kubernetes service
CNI/Network plugin	Flannel Calico (default) Note: Network policies are covered in the Security and Authentication section.	Azure CNI Calico Note: Network policies are covered in the Security and Authentication section.
Load balancers	Default load balancer for services is HA Proxy, however, you can also use your own cluster load balancer like MetalLB
Ingress controllers	You can add ingress controllers like Nginx, Traefik etc. using common Kubernetes guidance.	Default ingress control in Azure is the Azure load balancer. You can deploy additional ingress controls like Nginx.
Egress controls	Egress is controlled by Network policies, by default all outbound traffic from pods is blocked. You can deploy additional egress controls and policies.	Default setting is to not allow traffic from pod-to-pod. You can use Azure Policy and NSGs to control network flow or use Calico policies. You can also use Azure FW and Azure Security Groups.
Egress types	Load Balancer is the only built-in egress type in AKS-HCI	Azure load balancer, managed NAT gateway and user defined routes are the supported egress types
Customize CoreDNS	Allowed	Allowed
Storage
Where is the storage provisioned?	On-premises	Azure Storage. You can also deploy any custom storage class.
What types of persistent volumes are supported?	Read Write Once Read Write Many	Read Write Once Read Write Many
Do the storage drivers support Container Storage Interface (CSI)?	Yes	Yes
Is dynamic provisioning supported?	Yes	Yes
Is volume resizing supported?	Yes	Yes
Are volume snapshots supported?	No	Yes
Security and Authentication
How do you access your Kubernetes cluster?	Certificate based kubeconfig (default) AD based kubeconfig	Certificate based kubeconfig (default) AAD based kubeconfig
Azure Policy	Yes! Through Azure Arc enabled K8s (free for AKS-HCI customers)	Yes!
Azure Security	Yes! Through Azure Arc enabled K8s Azure Security Center Azure Defender	Yes! Azure Security Center Azure Defender
Network Policies	Yes! We support Calico network policies	Yes! We support Calico and Azure CNI network policies
Limit access to API server	Yes! By using VIP pools.	Yes! By using the “-api-server-authorized-ip-ranges” parameter and private clusters.
Cluster RBAC	Yes, with added setup (RBAC manager)	Yes! Azure RBAC, AAD
Certificate rotation and secrets encryption	Yes!	Yes!
Support for private cluster	Not supported yet	Yes! You can create private AKS clusters
Secrets store CSI driver	Yes!	Yes!
Support for disk encryption	Yes via bitlocker	Disks are encrypted on the storage side with platform managed keys and with support for customer provided keys. Hosts and locally attached disks can also be encrypted with encryption at host.
Support for node-to-node traffic encryption	Can be enabled via Calico Wireguard	Can be enabled via Calico Wireguard
Pod identity support	Not yet Windows containers - gMSA	Pod identity is in public preview Windows containers - gMSA
Monitoring and Logging
Support for Azure Monitor	Yes! Through Azure Arc enabled K8s	Yes!
Collect logs	Yes! Through PS (Get-AksHciLogs) and WAC. All logs – management cluster, control plane nodes, target K8s clusters are collected.	Yes! Through Azure Portal, Az CLI, etc
SSH into K8s nodes to troubleshoot	Yes! You can SSH into Linux and Windows nodes	Yes! You can SSH into Linux and Windows nodes. However, we recommend using cloud native logging tools like AKS periscope or inspector gadget.
On-premises logging and monitoring tools	Yes! You can connect to Prometheus & Grafana using AKS-HCI PS commands (Install-AksHciMonitoring) You can also setup EFK on your AKS-HCI clusters.	Not applicable. AKS is entirely based in Azure!
Develop and run applications
Azure App service	Yes! Through Azure Arc enabled K8s	Yes!
Azure Functions	Yes! Through Azure Arc enabled K8s	Yes!
Azure Logic Apps	Yes! Through Azure Arc enabled K8s	Yes!
Install applications using Helm	Yes!	Yes!
OpenFaas	Yes!	Yes!
Bridge to Kubernetes	Yes!	Yes!
Open Liberty/WebSphere Liberty	Yes!	Yes!
Run Spark jobs	Pending validation	Yes!
Use GPUs	In preview	Yes!
PostgreSQL	Pending validation	Yes!
MySQL	Pending validation	Yes!
Service Mesh	Yes! We support Open Service Mesh (OSM) through Azure Arc enabled Kubernetes.	Open Service Mesh Istio LinkerD Consul Connect
DevOps	Azure DevOps through Azure Arc enabled K8s. GitOps through Azure Arc enabled Kubernetes is free for AKS-HCI customers.	Ansible Jenkins Azure DevOps GitHub Actions Terraform
Product Pricing
Product pricing	Pricing based on number of workload cluster vCPUs. Management cluster, control plane nodes, load balancers are free.	Unlimited free clusters, pay for on-demand compute of the worker nodes.
Azure Support	AKS-HCI is supported out of the Windows Server support organization aligned with Arc for Kubernetes and Azure Stack HCI. You can open support requests through the Azure portal and other support channels like Premier Support.	AKS in Azure is supported through enterprise class support in the Azure team. You can open support requests in the Azure portal.
SLA	We do not offer SLAs since AKS-HCI runs in your environment.	Paid uptime SLA clusters for production with fixed cost on the API + worker node compute cost.