Azure Arc Blog

6 MIN READ

How do AKS and AKS on Azure Stack HCI compare?

Microsoft

Mar 13, 2023

This blog is an update to the original blog published comparing AKS in Azure and on Azure Stack HCI, a year ago. Since then, we’ve released multiple features and fixes aimed at improving AKS consistency between Azure and on-premises that warranted a fresh blog 😊

Features in preview are marked by (*)

Feature Set	AKS on Azure Stack HCI & AKS on Windows Server	AKS
Kubernetes Management Cluster/AKS host	AKS on Azure Stack HCI and Windows Server is a Cluster API based hosted Kubernetes offering. A management Kubernetes cluster is used to manage Kubernetes workload clusters. The management Kubernetes cluster runs in customer datacenters and is managed by the infrastructure administrator.	AKS is a managed Kubernetes offering. AKS control plane is hosted and managed by Microsoft. AKS worker nodes are created in customer subscriptions.

Kubernetes Target Cluster (lifecycle operations)
Cloud Native Computing Foundation (CNCF) certification	Yes	Yes
Who manages the cluster?	Managed by you	Managed by you
Where is the cluster located?	In your datacenter alongside your AKS hybrid management cluster. Azure Stack HCI 21H2 Windows Server 2019 Datacenter Windows Server 2022 Datacenter Windows 10/11 IoT Enterprise* Windows 10/11 Enterprise* Windows 10/11 Pro*	Azure cloud
K8s cluster lifecycle management tools (create, scale, update and delete clusters)	PowerShell (PS) Windows Admin Center (WAC) Az CLI* Azure Portal* ARM templates*	Az CLI Az PowerShell Azure Portal Bicep ARM templates
Can you use kubectl and other open-source Kubernetes tools?	Yes	Yes
Workload cluster updates	K8s version upgrade through PowerShell or WAC. Initiated by you. Node OS image update initiated by you; Updates in a target cluster happen at the cluster level – control plane nodes + node pools updated.	Azure CLI, Azure PS, Portal, ARM templates, GitHub Actions; OS image patch upgrade; Automatic upgrades; Planned maintenance windows;
Kubernetes versions	Continuous updates to supported Kubernetes versions. For latest version support, visit AKS hybrid releases on GitHub.	Continuous updates to supported Kubernetes versions. For latest version support, run az aks get-versions.
Can you start/stop K8s clusters to save costs?	Yes, by stopping the underlying failover cluster	Yes
Azure Fleet Manager integration	Not yet.	Yes*
Terraform support	Not yet.	Yes

Node Pools
Do you support running Linux and Windows node pools in the same cluster?	Yes! Linux nodes: CBL-Mariner Windows nodes: Windows Server 2019 Datacenter, Windows Server 2022 Datacenter	Yes. Linux nodes: Ubuntu 18.04, CBL-Mariner Windows nodes: Windows Server 2019 Datacenter Windows Server 2022 Datacenter
What’s your container runtime?	Linux nodes: containerd Windows nodes: containerd	Linux nodes: containerd Windows nodes: containerd
Can you scale node pools?	Manually Cluster autoscaler Vertical pod autoscalar	Manually Cluster autoscaler Vertical pod autoscalar
Horizontal pod autoscalar	Yes	Yes
What about virtual nodes?
Azure container instance	No	Yes
Can you upgrade a node pool?	We do not support upgrading individual node pools. All upgrades happen at the K8s cluster level.	You can perform node pool specific upgrades in an AKS cluster.
GPU enabled node pools	Yes*	Yes
Azure Container Registry	Yes	Yes
KEDA support	Not yet	Yes*

Networking
Who creates and manages the networks?	All networks (for both the management cluster and target K8s clusters) are created and managed by you	By default, Azure creates the virtual network and subnet for you. You can also choose an existing virtual network to create your AKS clusters
What type of network options are supported?	DHCP networks with/without VLAN ID Static IP networks with/without VLAN ID SDN support for AKS on Azure Stack HCI	Bring your own Azure virtual network for AKS clusters.
Load balancers	HAProxy (default) runs in a separate VM in the target K8s cluster kubeVIP – runs as a K8s service in the control plane K8s node Bring your own load balancer Load balancers are always given sIP addresses from a customer vip pool to ensure application and K8s cluster availability. You can create multiple instances of a LB (active-passive) for high availability	Azure load balancer – Basic SKU or Standard SKU Can also use internal load balancer By default, load balancer IP address is tied to load balancer ARM resource. You can also assign a static public IP address directly to your Kubernetes service
CNI/Network plugin	Calico (default) Note: Network policies are covered in the Security and Authentication section.	Azure CNI Calico Azure CNI Overlay Bring your own CNI Note: Network policies are covered in the Security and Authentication section.
Ingress controllers	No but you can use 3^rd party addons – Nginx. 3^rd party addons are not supported by Microsoft’s support policy.	Support for Nginx with web app routing addon.
Egress controls	Egress is controlled by Network policies, by default all outbound traffic from pods is blocked. You can deploy additional egress controls and policies.	You can use Azure Policy and NSGs to control network flow or use Calico policies. You can also use Azure FW and Azure Security Groups.
Egress types	Egress types and options depend on your network architecture.	Azure load balancer, managed NAT gateway and user defined routes are the supported egress types.
Customize CoreDNS	Allowed	Allowed
Service Mesh	Yes, Open Service Mesh (OSM) through Azure Arc enabled Kubernetes. 3^rd party addons – Istio, etc. 3^rd party addons are not supported by Microsoft’s support policy.	Open Service Mesh Marketplace offering available for Istio
Storage
Where is the storage provisioned?	On-premises	Azure Storage. Azure Files and Azure Disk premium CSI drivers deployed by default. You can also deploy any custom storage class.
What types of persistent volumes are supported?	Read Write Once Read Write Many	Read Write Once Read Write Many
Do the storage drivers support Container Storage Interface (CSI)?	Yes	Yes
Is dynamic provisioning supported?	Yes	Yes
Is volume resizing supported?	Yes	Yes
Are volume snapshots supported?	No	Yes

Security and Authentication
How do you access your Kubernetes cluster?	Certificate based kubeconfig (default) AD based kubeconfig Azure AD and Kubernetes RBAC Azure AD and Azure RBAC*	Certificate based kubeconfig (default) Azure AD and Kubernetes RBAC Azure AD and Azure RBAC
Network Policies	Yes, we support Calico network policies	Yes, we support Calico and Azure CNI network policies
Limit source networks that can access API server	Yes, by using VIP pools.	Yes, by using the “-api-server-authorized-ip-ranges” parameter and private clusters.
Certificate rotation and secrets encryption	Yes	Yes
Support for private cluster	Not supported yet	Yes! You can create private AKS clusters
Secrets store CSI driver	Yes	Yes
Support for disk encryption	Yes, via bitlocker	Disks are encrypted on the storage side with platform managed keys and with support for customer provided keys. Hosts and locally attached disks can also be encrypted with encryption at host.
gMSA v2 support for Windows containers	Yes	Yes
Azure Policy	Yes, through Azure Arc enabled K8s	Yes
Azure Defender	Yes, through Azure Arc enabled K8s*	Yes

Monitoring and Logging
Collect logs	Yes, through PS and WAC. All logs – management cluster, control plane nodes, target K8s clusters are collected.	Yes, through Azure Portal, Az CLI, etc
Support for Azure Monitor	Yes, through Azure Arc enabled K8s.	Yes
3^rd party addons for monitoring and logging		AKS works with Azure managed Prometheus* and Azure managed Grafana*
Subscribe to Azure Event Grid Events	Yes, via Azure Arc enabled Kubernetes*	Yes

Develop and run applications
Azure App service	Yes, through Azure Arc enabled K8s*	Yes
Azure Functions	Yes, through Azure Arc enabled K8s*	Yes
Azure Logic Apps	Yes, through Azure Arc enabled K8s*	You can directly create App Service, Functions, Logic Apps on Azure instead of creating on AKS
Develop applications using Helm	Yes	Yes
Develop applications using Dapr	Yes, through Azure Arc enabled K8s*	Yes
DevOps	Azure DevOps via Azure Arc enabled K8s. GitHub Actions via Azure Arc enabled K8s. GitOps Flux v2 via Azure Arc enabled K8s. 3^rd party addon: ArgoCD. 3^rd party addons are not supported by Microsoft’s support policy. GitOps Flux v2 through Azure Arc enabled Kubernetes is free for AKS-HCI customers.	Azure DevOps GitHub Actions GitOps Flux v2

Product Pricing
Product pricing	If you have Azure Hybrid Benefit, you can use AKS-HCI at no additional cost. If you do not have Azure Hybrid Benefit pricing based on number of workload cluster vCPUs. Management cluster, control plane nodes, load balancers are free.	Unlimited free clusters, pay for on-demand compute of the worker nodes. Paid tier available with uptime SLA, support for 5k nodes.
Azure Support	AKS-HCI is supported out of the Windows Server support organization aligned with Arc for Kubernetes and Azure Stack HCI. You can open support requests through the Azure portal and other support channels like Premier Support.	AKS in Azure is supported through enterprise class support in the Azure team. You can open support requests in the Azure portal.
SLA	We do not offer SLAs since AKS-HCI runs in your environment.	Paid uptime SLA clusters for production with fixed cost on the API + worker node compute, storage and networking costs.

Updated Mar 13, 2023

Version 1.0

azure arc

cloud adoption framework

hybrid

kubernetes

AbhilashaAgarwala

Microsoft

Joined February 23, 2021

View Profile

Azure Arc Blog

Follow this blog board to get notified when there's new activity

\n Feature Set \n	\n AKS on Azure Stack HCI & AKS on Windows Server \n	\n AKS \n
\n Kubernetes Management Cluster/AKS host \n	\n AKS on Azure Stack HCI and Windows Server is a Cluster API based hosted Kubernetes offering. A management Kubernetes cluster is used to manage Kubernetes workload clusters. The management Kubernetes cluster runs in customer datacenters and is managed by the infrastructure administrator. \n	\n AKS is a managed Kubernetes offering. AKS control plane is hosted and managed by Microsoft. AKS worker nodes are created in customer subscriptions. \n
\n \n	\n \n	\n \n
\n Kubernetes Target Cluster \n (lifecycle operations) \n	\n \n	\n \n
\n Cloud Native Computing Foundation (CNCF) certification \n	\n Yes \n	\n Yes \n
\n Who manages the cluster? \n	\n Managed by you \n	\n Managed by you \n
\n Where is the cluster located? \n	\n In your datacenter alongside your AKS hybrid management cluster. \n Azure Stack HCI 21H2 \n Windows Server 2019 Datacenter \n Windows Server 2022 Datacenter \n Windows 10/11 IoT Enterprise* Windows 10/11 Enterprise* Windows 10/11 Pro* \n	\n Azure cloud \n
\n K8s cluster lifecycle management tools (create, scale, update and delete clusters) \n	\n PowerShell (PS) \n Windows Admin Center (WAC) \n Az CLI* \n Azure Portal* \n ARM templates* \n	\n Az CLI \n Az PowerShell \n Azure Portal \n Bicep \n ARM templates \n
\n Can you use kubectl and other open-source Kubernetes tools? \n	\n Yes \n	\n Yes \n
\n Workload cluster updates \n	\n K8s version upgrade through PowerShell or WAC. Initiated by you. \n Node OS image update initiated by you; \n Updates in a target cluster happen at the cluster level – control plane nodes + node pools updated. \n	\n Azure CLI, Azure PS, Portal, ARM templates, GitHub Actions; \n OS image patch upgrade; \n Automatic upgrades; \n Planned maintenance windows; \n \n
\n Kubernetes versions \n	\n Continuous updates to supported Kubernetes versions. For latest version support, visit AKS hybrid releases on GitHub. \n	\n Continuous updates to supported Kubernetes versions. For latest version support, run az aks get-versions. \n
\n Can you start/stop K8s clusters to save costs? \n	\n Yes, by stopping the underlying failover cluster \n	\n Yes \n
\n Azure Fleet Manager integration \n	\n Not yet. \n	\n Yes* \n
\n Terraform support \n	\n Not yet. \n	\n Yes \n
\n \n	\n \n	\n \n
\n Node Pools \n	\n \n	\n \n
\n Do you support running Linux and Windows node pools in the same cluster? \n	\n Yes! \n Linux nodes: CBL-Mariner \n Windows nodes: \n Windows Server 2019 Datacenter, Windows Server 2022 Datacenter \n	\n Yes. \n Linux nodes: Ubuntu 18.04, CBL-Mariner \n Windows nodes: \n Windows Server 2019 Datacenter \n Windows Server 2022 Datacenter \n
\n What’s your container runtime? \n	\n Linux nodes: containerd \n Windows nodes: containerd \n	\n Linux nodes: containerd \n Windows nodes: containerd \n
\n Can you scale node pools? \n	\n Manually \n Cluster autoscaler \n Vertical pod autoscalar \n	\n Manually \n Cluster autoscaler \n Vertical pod autoscalar \n
\n Horizontal pod autoscalar \n	\n Yes \n	\n Yes \n
\n What about virtual nodes? \n	\n \n	\n \n \n \n
\n Azure container instance \n	\n No \n	\n Yes \n
\n Can you upgrade a node pool? \n	\n We do not support upgrading individual node pools. All upgrades happen at the K8s cluster level. \n	\n You can perform node pool specific upgrades in an AKS cluster. \n
\n GPU enabled node pools \n	\n Yes* \n	\n Yes \n
\n Azure Container Registry \n	\n Yes \n	\n Yes \n
\n KEDA support \n	\n Not yet \n	\n Yes* \n
\n \n	\n \n	\n \n
\n Networking \n	\n \n	\n \n
\n Who creates and manages the networks? \n	\n All networks (for both the management cluster and target K8s clusters) are created and managed by you \n	\n By default, Azure creates the virtual network and subnet for you. You can also choose an existing virtual network to create your AKS clusters \n
\n What type of network options are supported? \n	\n DHCP networks with/without VLAN ID \n Static IP networks with/without VLAN ID \n SDN support for AKS on Azure Stack HCI \n	\n Bring your own Azure virtual network for AKS clusters. \n
\n Load balancers \n	\n HAProxy (default) runs in a separate VM in the target K8s cluster \n kubeVIP – runs as a K8s service in the control plane K8s node \n Bring your own load balancer \n Load balancers are always given sIP addresses from a customer vip pool to ensure application and K8s cluster availability. \n You can create multiple instances of a LB (active-passive) for high availability \n	\n Azure load balancer – Basic SKU or Standard SKU \n Can also use internal load balancer \n By default, load balancer IP address is tied to load balancer ARM resource. You can also assign a static public IP address directly to your Kubernetes service \n
\n CNI/Network plugin \n	\n Calico (default) \n Note: Network policies are covered in the Security and Authentication section. \n	\n Azure CNI \n Calico \n Azure CNI Overlay \n Bring your own CNI \n Note: Network policies are covered in the Security and Authentication section. \n
\n Ingress controllers \n	\n No but you can use 3^rd party addons – Nginx. 3^rd party addons are not supported by Microsoft’s support policy. \n	\n \n Support for Nginx with web app routing addon. \n
\n Egress controls \n	\n Egress is controlled by Network policies, by default all outbound traffic from pods is blocked. You can deploy additional egress controls and policies. \n	\n You can use Azure Policy and NSGs to control network flow or use Calico policies. You can also use Azure FW and Azure Security Groups. \n
\n Egress types \n	\n Egress types and options depend on your network architecture. \n	\n Azure load balancer, managed NAT gateway and user defined routes are the supported egress types. \n
\n Customize CoreDNS \n	\n Allowed \n	\n Allowed \n
\n Service Mesh \n	\n Yes, Open Service Mesh (OSM) through Azure Arc enabled Kubernetes. \n 3^rd party addons – Istio, etc. 3^rd party addons are not supported by Microsoft’s support policy. \n	\n Open Service Mesh \n Marketplace offering available for Istio \n
\n Storage \n	\n \n	\n \n
\n Where is the storage provisioned? \n	\n On-premises \n	\n Azure Storage. \n Azure Files and Azure Disk premium CSI drivers deployed by default. You can also deploy any custom storage class. \n
\n What types of persistent volumes are supported? \n	\n Read Write Once \n Read Write Many \n	\n Read Write Once \n Read Write Many \n
\n Do the storage drivers support Container Storage Interface (CSI)? \n	\n Yes \n	\n Yes \n
\n Is dynamic provisioning supported? \n	\n Yes \n	\n Yes \n
\n Is volume resizing supported? \n	\n Yes \n	\n Yes \n
\n Are volume snapshots supported? \n	\n No \n	\n Yes \n
\n \n	\n \n	\n \n
\n Security and Authentication \n	\n \n	\n \n
\n How do you access your Kubernetes cluster? \n	\n Certificate based kubeconfig (default) \n AD based kubeconfig \n Azure AD and Kubernetes RBAC \n Azure AD and Azure RBAC* \n	\n Certificate based kubeconfig (default) \n Azure AD and Kubernetes RBAC \n Azure AD and Azure RBAC \n
\n Network Policies \n	\n Yes, we support Calico network policies \n	\n Yes, we support Calico and Azure CNI network policies \n
\n Limit source networks that can access API server \n	\n Yes, by using VIP pools. \n	\n Yes, by using the “-api-server-authorized-ip-ranges” parameter and private clusters. \n
\n Certificate rotation and secrets encryption \n	\n Yes \n	\n Yes \n
\n Support for private cluster \n	\n Not supported yet \n	\n Yes! You can create private AKS clusters \n
\n Secrets store CSI driver \n	\n Yes \n	\n Yes \n
\n Support for disk encryption \n	\n Yes, via bitlocker \n	\n Disks are encrypted on the storage side with platform managed keys and with support for customer provided keys. \n Hosts and locally attached disks can also be encrypted with encryption at host. \n
\n gMSA v2 support for Windows containers \n	\n Yes \n	\n Yes \n
\n Azure Policy \n	\n Yes, through Azure Arc enabled K8s \n	\n Yes \n
\n Azure Defender \n	\n Yes, through Azure Arc enabled K8s* \n	\n Yes \n
\n \n	\n \n	\n \n
\n Monitoring and Logging \n	\n \n	\n \n
\n Collect logs \n	\n Yes, through PS and WAC. All logs – management cluster, control plane nodes, target K8s clusters are collected. \n	\n Yes, through Azure Portal, Az CLI, etc \n
\n Support for Azure Monitor \n	\n Yes, through Azure Arc enabled K8s. \n	\n Yes \n
\n 3^rd party addons for monitoring and logging \n	\n \n \n \n	\n AKS works with Azure managed Prometheus* and Azure managed Grafana* \n
\n Subscribe to Azure Event Grid Events \n	\n Yes, via Azure Arc enabled Kubernetes* \n	\n Yes \n
\n \n	\n \n	\n \n
\n Develop and run applications \n	\n \n	\n \n
\n Azure App service \n	\n Yes, through Azure Arc enabled K8s* \n	\n Yes \n
\n Azure Functions \n	\n Yes, through Azure Arc enabled K8s* \n	\n Yes \n
\n Azure Logic Apps \n	\n Yes, through Azure Arc enabled K8s* \n	\n You can directly create App Service, Functions, Logic Apps on Azure instead of creating on AKS \n
\n Develop applications using Helm \n	\n Yes \n	\n Yes \n
\n Develop applications using Dapr \n	\n Yes, through Azure Arc enabled K8s* \n	\n Yes \n
\n DevOps \n	\n Azure DevOps via Azure Arc enabled K8s. \n GitHub Actions via Azure Arc enabled K8s. \n GitOps Flux v2 via Azure Arc enabled K8s. \n \n 3^rd party addon: ArgoCD. 3^rd party addons are not supported by Microsoft’s support policy. \n \n GitOps Flux v2 through Azure Arc enabled Kubernetes is free for AKS-HCI customers. \n	\n Azure DevOps \n GitHub Actions \n GitOps Flux v2 \n
\n \n	\n \n	\n \n
\n Product Pricing \n	\n \n	\n \n
\n Product pricing \n	\n If you have Azure Hybrid Benefit, you can use AKS-HCI at no additional cost. \n \n If you do not have Azure Hybrid Benefit pricing based on number of workload cluster vCPUs. Management cluster, control plane nodes, load balancers are free. \n	\n Unlimited free clusters, pay for on-demand compute of the worker nodes. \n Paid tier available with uptime SLA, support for 5k nodes. \n \n
\n Azure Support \n	\n AKS-HCI is supported out of the Windows Server support organization aligned with Arc for Kubernetes and Azure Stack HCI. You can open support requests through the Azure portal and other support channels like Premier Support. \n	\n AKS in Azure is supported through enterprise class support in the Azure team. You can open support requests in the Azure portal. \n
\n SLA \n	\n We do not offer SLAs since AKS-HCI runs in your environment. \n	\n Paid uptime SLA clusters for production with fixed cost on the API + worker node compute, storage and networking costs. \n \n

Feature Set

AKS on Azure Stack HCI & AKS on Windows Server

AKS

Kubernetes Management Cluster/AKS host

AKS on Azure Stack HCI and Windows Server is a Cluster API based hosted Kubernetes offering. A management Kubernetes cluster is used to manage Kubernetes workload clusters. The management Kubernetes cluster runs in customer datacenters and is managed by the infrastructure administrator.

AKS is a managed Kubernetes offering. AKS control plane is hosted and managed by Microsoft. AKS worker nodes are created in customer subscriptions.

Kubernetes Target Cluster

(lifecycle operations)

Cloud Native Computing Foundation (CNCF) certification

Yes

Who manages the cluster?

Managed by you

Where is the cluster located?

In your datacenter alongside your AKS hybrid management cluster.

Azure Stack HCI 21H2

Windows Server 2019 Datacenter

Windows Server 2022 Datacenter

Windows 10/11 IoT Enterprise*
Windows 10/11 Enterprise*
Windows 10/11 Pro*

Azure cloud

K8s cluster lifecycle management tools (create, scale, update and delete clusters)

PowerShell (PS)

Windows Admin Center (WAC)

Az CLI*

Azure Portal*

ARM templates*

Az CLI

Az PowerShell

Azure Portal

Bicep

ARM templates

Can you use kubectl and other open-source Kubernetes tools?

Yes

Workload cluster updates

K8s version upgrade through PowerShell or WAC. Initiated by you.

Node OS image update initiated by you;

Updates in a target cluster happen at the cluster level – control plane nodes + node pools updated.

Azure CLI, Azure PS, Portal, ARM templates, GitHub Actions;

OS image patch upgrade;

Automatic upgrades;

Planned maintenance windows;

Kubernetes versions

Continuous updates to supported Kubernetes versions. For latest version support, visit AKS hybrid releases on GitHub.

Continuous updates to supported Kubernetes versions. For latest version support, run az aks get-versions.

Can you start/stop K8s clusters to save costs?

Yes, by stopping the underlying failover cluster

Yes

Azure Fleet Manager integration

Not yet.

Yes*

Terraform support

Not yet.

Yes

Node Pools

Do you support running Linux and Windows node pools in the same cluster?

Yes!

Linux nodes: CBL-Mariner

Windows nodes:

Windows Server 2019 Datacenter, Windows Server 2022 Datacenter

Yes.

Linux nodes: Ubuntu 18.04, CBL-Mariner

Windows nodes:

Windows Server 2019 Datacenter

Windows Server 2022 Datacenter

What’s your container runtime?

Linux nodes: containerd

Windows nodes: containerd

Linux nodes: containerd

Windows nodes: containerd

Can you scale node pools?

Manually

Cluster autoscaler

Vertical pod autoscalar

Manually

Cluster autoscaler

Vertical pod autoscalar

Horizontal pod autoscalar

Yes

What about virtual nodes?

Azure container instance

Yes

Can you upgrade a node pool?

We do not support upgrading individual node pools. All upgrades happen at the K8s cluster level.

You can perform node pool specific upgrades in an AKS cluster.

GPU enabled node pools

Yes*

Yes

Azure Container Registry

Yes

KEDA support

Not yet

Yes*

Networking

Who creates and manages the networks?

All networks (for both the management cluster and target K8s clusters) are created and managed by you

By default, Azure creates the virtual network and subnet for you. You can also choose an existing virtual network to create your AKS clusters

What type of network options are supported?

DHCP networks with/without VLAN ID

Static IP networks with/without VLAN ID

SDN support for AKS on Azure Stack HCI

Bring your own Azure virtual network for AKS clusters.

Load balancers

HAProxy (default) runs in a separate VM in the target K8s cluster

kubeVIP – runs as a K8s service in the control plane K8s node

Bring your own load balancer

Load balancers are always given sIP addresses from a customer vip pool to ensure application and K8s cluster availability.

You can create multiple instances of a LB (active-passive) for high availability

Azure load balancer – Basic SKU or Standard SKU

Can also use internal load balancer

By default, load balancer IP address is tied to load balancer ARM resource. You can also assign a static public IP address directly to your Kubernetes service

CNI/Network plugin

Calico (default)

Note: Network policies are covered in the Security and Authentication section.

Azure CNI

Calico

Azure CNI Overlay

Bring your own CNI

Note: Network policies are covered in the Security and Authentication section.

Ingress controllers

No but you can use 3^rd party addons – Nginx. 3^rd party addons are not supported by Microsoft’s support policy.

Support for Nginx with web app routing addon.

Egress controls

Egress is controlled by Network policies, by default all outbound traffic from pods is blocked. You can deploy additional egress controls and policies.

You can use Azure Policy and NSGs to control network flow or use Calico policies. You can also use Azure FW and Azure Security Groups.

Egress types

Egress types and options depend on your network architecture.

Azure load balancer, managed NAT gateway and user defined routes are the supported egress types.

Customize CoreDNS

Allowed

Service Mesh

Yes, Open Service Mesh (OSM) through Azure Arc enabled Kubernetes.

3^rd party addons – Istio, etc. 3^rd party addons are not supported by Microsoft’s support policy.

Open Service Mesh

Marketplace offering available for Istio

Storage

Where is the storage provisioned?

On-premises

Azure Storage.

Azure Files and Azure Disk premium CSI drivers deployed by default. You can also deploy any custom storage class.

What types of persistent volumes are supported?

Read Write Once

Read Write Many

Read Write Once

Read Write Many

Do the storage drivers support Container Storage Interface (CSI)?

Yes

Is dynamic provisioning supported?

Yes

Is volume resizing supported?

Yes

Are volume snapshots supported?

Yes

Security and Authentication

How do you access your Kubernetes cluster?

Certificate based kubeconfig (default)

AD based kubeconfig

Azure AD and Kubernetes RBAC

Azure AD and Azure RBAC*

Certificate based kubeconfig (default)

Azure AD and Kubernetes RBAC

Azure AD and Azure RBAC

Network Policies

Yes, we support Calico network policies

Yes, we support Calico and Azure CNI network policies

Limit source networks that can access API server

Yes, by using VIP pools.

Yes, by using the “-api-server-authorized-ip-ranges” parameter and private clusters.

Certificate rotation and secrets encryption

Yes

Support for private cluster

Not supported yet

Yes! You can create private AKS clusters

Secrets store CSI driver

Yes

Support for disk encryption

Yes, via bitlocker

Disks are encrypted on the storage side with platform managed keys and with support for customer provided keys.

Hosts and locally attached disks can also be encrypted with encryption at host.

gMSA v2 support for Windows containers

Yes

Azure Policy

Yes, through Azure Arc enabled K8s

Yes

Azure Defender

Yes, through Azure Arc enabled K8s*

Yes

Monitoring and Logging

Collect logs

Yes, through PS and WAC. All logs – management cluster, control plane nodes, target K8s clusters are collected.

Yes, through Azure Portal, Az CLI, etc

Support for Azure Monitor

Yes, through Azure Arc enabled K8s.

Yes

3^rd party addons for monitoring and logging

AKS works with Azure managed Prometheus* and Azure managed Grafana*

Subscribe to Azure Event Grid Events

Yes, via Azure Arc enabled Kubernetes*

Yes

Develop and run applications

Azure App service

Yes, through Azure Arc enabled K8s*

Yes

Azure Functions

Yes, through Azure Arc enabled K8s*

Yes

Azure Logic Apps

Yes, through Azure Arc enabled K8s*

You can directly create App Service, Functions, Logic Apps on Azure instead of creating on AKS

Develop applications using Helm

Yes

Develop applications using Dapr

Yes, through Azure Arc enabled K8s*

Yes

DevOps

Azure DevOps via Azure Arc enabled K8s.

GitHub Actions via Azure Arc enabled K8s.

GitOps Flux v2 via Azure Arc enabled K8s.

3^rd party addon: ArgoCD. 3^rd party addons are not supported by Microsoft’s support policy.

GitOps Flux v2 through Azure Arc enabled Kubernetes is free for AKS-HCI customers.

Azure DevOps

GitHub Actions

GitOps Flux v2

Product Pricing

Product pricing

If you have Azure Hybrid Benefit, you can use AKS-HCI at no additional cost.

If you do not have Azure Hybrid Benefit pricing based on number of workload cluster vCPUs. Management cluster, control plane nodes, load balancers are free.

Unlimited free clusters, pay for on-demand compute of the worker nodes.

Paid tier available with uptime SLA, support for 5k nodes.

Azure Support

AKS-HCI is supported out of the Windows Server support organization aligned with Arc for Kubernetes and Azure Stack HCI. You can open support requests through the Azure portal and other support channels like Premier Support.

AKS in Azure is supported through enterprise class support in the Azure team. You can open support requests in the Azure portal.

SLA

We do not offer SLAs since AKS-HCI runs in your environment.

Paid uptime SLA clusters for production with fixed cost on the API + worker node compute, storage and networking costs.

\n Feature Set \n	\n AKS on Azure Stack HCI & AKS on Windows Server \n	\n AKS \n
\n Kubernetes Management Cluster/AKS host \n	\n AKS on Azure Stack HCI and Windows Server is a Cluster API based hosted Kubernetes offering. A management Kubernetes cluster is used to manage Kubernetes workload clusters. The management Kubernetes cluster runs in customer datacenters and is managed by the infrastructure administrator. \n	\n AKS is a managed Kubernetes offering. AKS control plane is hosted and managed by Microsoft. AKS worker nodes are created in customer subscriptions. \n
\n \n	\n \n	\n \n
\n Kubernetes Target Cluster \n (lifecycle operations) \n	\n \n	\n \n
\n Cloud Native Computing Foundation (CNCF) certification \n	\n Yes \n	\n Yes \n
\n Who manages the cluster? \n	\n Managed by you \n	\n Managed by you \n
\n Where is the cluster located? \n	\n In your datacenter alongside your AKS hybrid management cluster. \n Azure Stack HCI 21H2 \n Windows Server 2019 Datacenter \n Windows Server 2022 Datacenter \n Windows 10/11 IoT Enterprise* Windows 10/11 Enterprise* Windows 10/11 Pro* \n	\n Azure cloud \n
\n K8s cluster lifecycle management tools (create, scale, update and delete clusters) \n	\n PowerShell (PS) \n Windows Admin Center (WAC) \n Az CLI* \n Azure Portal* \n ARM templates* \n	\n Az CLI \n Az PowerShell \n Azure Portal \n Bicep \n ARM templates \n
\n Can you use kubectl and other open-source Kubernetes tools? \n	\n Yes \n	\n Yes \n
\n Workload cluster updates \n	\n K8s version upgrade through PowerShell or WAC. Initiated by you. \n Node OS image update initiated by you; \n Updates in a target cluster happen at the cluster level – control plane nodes + node pools updated. \n	\n Azure CLI, Azure PS, Portal, ARM templates, GitHub Actions; \n OS image patch upgrade; \n Automatic upgrades; \n Planned maintenance windows; \n \n
\n Kubernetes versions \n	\n Continuous updates to supported Kubernetes versions. For latest version support, visit AKS hybrid releases on GitHub. \n	\n Continuous updates to supported Kubernetes versions. For latest version support, run az aks get-versions. \n
\n Can you start/stop K8s clusters to save costs? \n	\n Yes, by stopping the underlying failover cluster \n	\n Yes \n
\n Azure Fleet Manager integration \n	\n Not yet. \n	\n Yes* \n
\n Terraform support \n	\n Not yet. \n	\n Yes \n
\n \n	\n \n	\n \n
\n Node Pools \n	\n \n	\n \n
\n Do you support running Linux and Windows node pools in the same cluster? \n	\n Yes! \n Linux nodes: CBL-Mariner \n Windows nodes: \n Windows Server 2019 Datacenter, Windows Server 2022 Datacenter \n	\n Yes. \n Linux nodes: Ubuntu 18.04, CBL-Mariner \n Windows nodes: \n Windows Server 2019 Datacenter \n Windows Server 2022 Datacenter \n
\n What’s your container runtime? \n	\n Linux nodes: containerd \n Windows nodes: containerd \n	\n Linux nodes: containerd \n Windows nodes: containerd \n
\n Can you scale node pools? \n	\n Manually \n Cluster autoscaler \n Vertical pod autoscalar \n	\n Manually \n Cluster autoscaler \n Vertical pod autoscalar \n
\n Horizontal pod autoscalar \n	\n Yes \n	\n Yes \n
\n What about virtual nodes? \n	\n \n	\n \n \n \n
\n Azure container instance \n	\n No \n	\n Yes \n
\n Can you upgrade a node pool? \n	\n We do not support upgrading individual node pools. All upgrades happen at the K8s cluster level. \n	\n You can perform node pool specific upgrades in an AKS cluster. \n
\n GPU enabled node pools \n	\n Yes* \n	\n Yes \n
\n Azure Container Registry \n	\n Yes \n	\n Yes \n
\n KEDA support \n	\n Not yet \n	\n Yes* \n
\n \n	\n \n	\n \n
\n Networking \n	\n \n	\n \n
\n Who creates and manages the networks? \n	\n All networks (for both the management cluster and target K8s clusters) are created and managed by you \n	\n By default, Azure creates the virtual network and subnet for you. You can also choose an existing virtual network to create your AKS clusters \n
\n What type of network options are supported? \n	\n DHCP networks with/without VLAN ID \n Static IP networks with/without VLAN ID \n SDN support for AKS on Azure Stack HCI \n	\n Bring your own Azure virtual network for AKS clusters. \n
\n Load balancers \n	\n HAProxy (default) runs in a separate VM in the target K8s cluster \n kubeVIP – runs as a K8s service in the control plane K8s node \n Bring your own load balancer \n Load balancers are always given sIP addresses from a customer vip pool to ensure application and K8s cluster availability. \n You can create multiple instances of a LB (active-passive) for high availability \n	\n Azure load balancer – Basic SKU or Standard SKU \n Can also use internal load balancer \n By default, load balancer IP address is tied to load balancer ARM resource. You can also assign a static public IP address directly to your Kubernetes service \n
\n CNI/Network plugin \n	\n Calico (default) \n Note: Network policies are covered in the Security and Authentication section. \n	\n Azure CNI \n Calico \n Azure CNI Overlay \n Bring your own CNI \n Note: Network policies are covered in the Security and Authentication section. \n
\n Ingress controllers \n	\n No but you can use 3^rd party addons – Nginx. 3^rd party addons are not supported by Microsoft’s support policy. \n	\n \n Support for Nginx with web app routing addon. \n
\n Egress controls \n	\n Egress is controlled by Network policies, by default all outbound traffic from pods is blocked. You can deploy additional egress controls and policies. \n	\n You can use Azure Policy and NSGs to control network flow or use Calico policies. You can also use Azure FW and Azure Security Groups. \n
\n Egress types \n	\n Egress types and options depend on your network architecture. \n	\n Azure load balancer, managed NAT gateway and user defined routes are the supported egress types. \n
\n Customize CoreDNS \n	\n Allowed \n	\n Allowed \n
\n Service Mesh \n	\n Yes, Open Service Mesh (OSM) through Azure Arc enabled Kubernetes. \n 3^rd party addons – Istio, etc. 3^rd party addons are not supported by Microsoft’s support policy. \n	\n Open Service Mesh \n Marketplace offering available for Istio \n
\n Storage \n	\n \n	\n \n
\n Where is the storage provisioned? \n	\n On-premises \n	\n Azure Storage. \n Azure Files and Azure Disk premium CSI drivers deployed by default. You can also deploy any custom storage class. \n
\n What types of persistent volumes are supported? \n	\n Read Write Once \n Read Write Many \n	\n Read Write Once \n Read Write Many \n
\n Do the storage drivers support Container Storage Interface (CSI)? \n	\n Yes \n	\n Yes \n
\n Is dynamic provisioning supported? \n	\n Yes \n	\n Yes \n
\n Is volume resizing supported? \n	\n Yes \n	\n Yes \n
\n Are volume snapshots supported? \n	\n No \n	\n Yes \n
\n \n	\n \n	\n \n
\n Security and Authentication \n	\n \n	\n \n
\n How do you access your Kubernetes cluster? \n	\n Certificate based kubeconfig (default) \n AD based kubeconfig \n Azure AD and Kubernetes RBAC \n Azure AD and Azure RBAC* \n	\n Certificate based kubeconfig (default) \n Azure AD and Kubernetes RBAC \n Azure AD and Azure RBAC \n
\n Network Policies \n	\n Yes, we support Calico network policies \n	\n Yes, we support Calico and Azure CNI network policies \n
\n Limit source networks that can access API server \n	\n Yes, by using VIP pools. \n	\n Yes, by using the “-api-server-authorized-ip-ranges” parameter and private clusters. \n
\n Certificate rotation and secrets encryption \n	\n Yes \n	\n Yes \n
\n Support for private cluster \n	\n Not supported yet \n	\n Yes! You can create private AKS clusters \n
\n Secrets store CSI driver \n	\n Yes \n	\n Yes \n
\n Support for disk encryption \n	\n Yes, via bitlocker \n	\n Disks are encrypted on the storage side with platform managed keys and with support for customer provided keys. \n Hosts and locally attached disks can also be encrypted with encryption at host. \n
\n gMSA v2 support for Windows containers \n	\n Yes \n	\n Yes \n
\n Azure Policy \n	\n Yes, through Azure Arc enabled K8s \n	\n Yes \n
\n Azure Defender \n	\n Yes, through Azure Arc enabled K8s* \n	\n Yes \n
\n \n	\n \n	\n \n
\n Monitoring and Logging \n	\n \n	\n \n
\n Collect logs \n	\n Yes, through PS and WAC. All logs – management cluster, control plane nodes, target K8s clusters are collected. \n	\n Yes, through Azure Portal, Az CLI, etc \n
\n Support for Azure Monitor \n	\n Yes, through Azure Arc enabled K8s. \n	\n Yes \n
\n 3^rd party addons for monitoring and logging \n	\n \n \n \n	\n AKS works with Azure managed Prometheus* and Azure managed Grafana* \n
\n Subscribe to Azure Event Grid Events \n	\n Yes, via Azure Arc enabled Kubernetes* \n	\n Yes \n
\n \n	\n \n	\n \n
\n Develop and run applications \n	\n \n	\n \n
\n Azure App service \n	\n Yes, through Azure Arc enabled K8s* \n	\n Yes \n
\n Azure Functions \n	\n Yes, through Azure Arc enabled K8s* \n	\n Yes \n
\n Azure Logic Apps \n	\n Yes, through Azure Arc enabled K8s* \n	\n You can directly create App Service, Functions, Logic Apps on Azure instead of creating on AKS \n
\n Develop applications using Helm \n	\n Yes \n	\n Yes \n
\n Develop applications using Dapr \n	\n Yes, through Azure Arc enabled K8s* \n	\n Yes \n
\n DevOps \n	\n Azure DevOps via Azure Arc enabled K8s. \n GitHub Actions via Azure Arc enabled K8s. \n GitOps Flux v2 via Azure Arc enabled K8s. \n \n 3^rd party addon: ArgoCD. 3^rd party addons are not supported by Microsoft’s support policy. \n \n GitOps Flux v2 through Azure Arc enabled Kubernetes is free for AKS-HCI customers. \n	\n Azure DevOps \n GitHub Actions \n GitOps Flux v2 \n
\n \n	\n \n	\n \n
\n Product Pricing \n	\n \n	\n \n
\n Product pricing \n	\n If you have Azure Hybrid Benefit, you can use AKS-HCI at no additional cost. \n \n If you do not have Azure Hybrid Benefit pricing based on number of workload cluster vCPUs. Management cluster, control plane nodes, load balancers are free. \n	\n Unlimited free clusters, pay for on-demand compute of the worker nodes. \n Paid tier available with uptime SLA, support for 5k nodes. \n \n
\n Azure Support \n	\n AKS-HCI is supported out of the Windows Server support organization aligned with Arc for Kubernetes and Azure Stack HCI. You can open support requests through the Azure portal and other support channels like Premier Support. \n	\n AKS in Azure is supported through enterprise class support in the Azure team. You can open support requests in the Azure portal. \n
\n SLA \n	\n We do not offer SLAs since AKS-HCI runs in your environment. \n	\n Paid uptime SLA clusters for production with fixed cost on the API + worker node compute, storage and networking costs. \n \n

Feature Set

AKS on Azure Stack HCI & AKS on Windows Server

AKS

Kubernetes Management Cluster/AKS host

AKS is a managed Kubernetes offering. AKS control plane is hosted and managed by Microsoft. AKS worker nodes are created in customer subscriptions.

Kubernetes Target Cluster

(lifecycle operations)

Cloud Native Computing Foundation (CNCF) certification

Yes

Who manages the cluster?

Managed by you

Where is the cluster located?

In your datacenter alongside your AKS hybrid management cluster.

Azure Stack HCI 21H2

Windows Server 2019 Datacenter

Windows Server 2022 Datacenter

Windows 10/11 IoT Enterprise*
Windows 10/11 Enterprise*
Windows 10/11 Pro*

Azure cloud

K8s cluster lifecycle management tools (create, scale, update and delete clusters)

PowerShell (PS)

Windows Admin Center (WAC)

Az CLI*

Azure Portal*

ARM templates*

Az CLI

Az PowerShell

Azure Portal

Bicep

ARM templates

Can you use kubectl and other open-source Kubernetes tools?

Yes

Workload cluster updates

K8s version upgrade through PowerShell or WAC. Initiated by you.

Node OS image update initiated by you;

Updates in a target cluster happen at the cluster level – control plane nodes + node pools updated.

Azure CLI, Azure PS, Portal, ARM templates, GitHub Actions;

OS image patch upgrade;

Automatic upgrades;

Planned maintenance windows;

Kubernetes versions

Continuous updates to supported Kubernetes versions. For latest version support, visit AKS hybrid releases on GitHub.

Continuous updates to supported Kubernetes versions. For latest version support, run az aks get-versions.

Can you start/stop K8s clusters to save costs?

Yes, by stopping the underlying failover cluster

Yes

Azure Fleet Manager integration

Not yet.

Yes*

Terraform support

Not yet.

Yes

Node Pools

Do you support running Linux and Windows node pools in the same cluster?

Yes!

Linux nodes: CBL-Mariner

Windows nodes:

Windows Server 2019 Datacenter, Windows Server 2022 Datacenter

Yes.

Linux nodes: Ubuntu 18.04, CBL-Mariner

Windows nodes:

Windows Server 2019 Datacenter

Windows Server 2022 Datacenter

What’s your container runtime?

Linux nodes: containerd

Windows nodes: containerd

Linux nodes: containerd

Windows nodes: containerd

Can you scale node pools?

Manually

Cluster autoscaler

Vertical pod autoscalar

Manually

Cluster autoscaler

Vertical pod autoscalar

Horizontal pod autoscalar

Yes

What about virtual nodes?

Azure container instance

Yes

Can you upgrade a node pool?

We do not support upgrading individual node pools. All upgrades happen at the K8s cluster level.

You can perform node pool specific upgrades in an AKS cluster.

GPU enabled node pools

Yes*

Yes

Azure Container Registry

Yes

KEDA support

Not yet

Yes*

Networking

Who creates and manages the networks?

All networks (for both the management cluster and target K8s clusters) are created and managed by you

By default, Azure creates the virtual network and subnet for you. You can also choose an existing virtual network to create your AKS clusters

What type of network options are supported?

DHCP networks with/without VLAN ID

Static IP networks with/without VLAN ID

SDN support for AKS on Azure Stack HCI

Bring your own Azure virtual network for AKS clusters.

Load balancers

HAProxy (default) runs in a separate VM in the target K8s cluster

kubeVIP – runs as a K8s service in the control plane K8s node

Bring your own load balancer

Load balancers are always given sIP addresses from a customer vip pool to ensure application and K8s cluster availability.

You can create multiple instances of a LB (active-passive) for high availability

Azure load balancer – Basic SKU or Standard SKU

Can also use internal load balancer

By default, load balancer IP address is tied to load balancer ARM resource. You can also assign a static public IP address directly to your Kubernetes service

CNI/Network plugin

Calico (default)

Note: Network policies are covered in the Security and Authentication section.

Azure CNI

Calico

Azure CNI Overlay

Bring your own CNI

Note: Network policies are covered in the Security and Authentication section.

Ingress controllers

No but you can use 3^rd party addons – Nginx. 3^rd party addons are not supported by Microsoft’s support policy.

Support for Nginx with web app routing addon.

Egress controls

Egress is controlled by Network policies, by default all outbound traffic from pods is blocked. You can deploy additional egress controls and policies.

You can use Azure Policy and NSGs to control network flow or use Calico policies. You can also use Azure FW and Azure Security Groups.

Egress types

Egress types and options depend on your network architecture.

Azure load balancer, managed NAT gateway and user defined routes are the supported egress types.

Customize CoreDNS

Allowed

Service Mesh

Yes, Open Service Mesh (OSM) through Azure Arc enabled Kubernetes.

3^rd party addons – Istio, etc. 3^rd party addons are not supported by Microsoft’s support policy.

Open Service Mesh

Marketplace offering available for Istio

Storage

Where is the storage provisioned?

On-premises

Azure Storage.

Azure Files and Azure Disk premium CSI drivers deployed by default. You can also deploy any custom storage class.

What types of persistent volumes are supported?

Read Write Once

Read Write Many

Read Write Once

Read Write Many

Do the storage drivers support Container Storage Interface (CSI)?

Yes

Is dynamic provisioning supported?

Yes

Is volume resizing supported?

Yes

Are volume snapshots supported?

Yes

Security and Authentication

How do you access your Kubernetes cluster?

Certificate based kubeconfig (default)

AD based kubeconfig

Azure AD and Kubernetes RBAC

Azure AD and Azure RBAC*

Certificate based kubeconfig (default)

Azure AD and Kubernetes RBAC

Azure AD and Azure RBAC

Network Policies

Yes, we support Calico network policies

Yes, we support Calico and Azure CNI network policies

Limit source networks that can access API server

Yes, by using VIP pools.

Yes, by using the “-api-server-authorized-ip-ranges” parameter and private clusters.

Certificate rotation and secrets encryption

Yes

Support for private cluster

Not supported yet

Yes! You can create private AKS clusters

Secrets store CSI driver

Yes

Support for disk encryption

Yes, via bitlocker

Disks are encrypted on the storage side with platform managed keys and with support for customer provided keys.

Hosts and locally attached disks can also be encrypted with encryption at host.

gMSA v2 support for Windows containers

Yes

Azure Policy

Yes, through Azure Arc enabled K8s

Yes

Azure Defender

Yes, through Azure Arc enabled K8s*

Yes

Monitoring and Logging

Collect logs

Yes, through PS and WAC. All logs – management cluster, control plane nodes, target K8s clusters are collected.

Yes, through Azure Portal, Az CLI, etc

Support for Azure Monitor

Yes, through Azure Arc enabled K8s.

Yes

3^rd party addons for monitoring and logging

AKS works with Azure managed Prometheus* and Azure managed Grafana*

Subscribe to Azure Event Grid Events

Yes, via Azure Arc enabled Kubernetes*

Yes

Develop and run applications

Azure App service

Yes, through Azure Arc enabled K8s*

Yes

Azure Functions

Yes, through Azure Arc enabled K8s*

Yes

Azure Logic Apps

Yes, through Azure Arc enabled K8s*

You can directly create App Service, Functions, Logic Apps on Azure instead of creating on AKS

Develop applications using Helm

Yes

Develop applications using Dapr

Yes, through Azure Arc enabled K8s*

Yes

DevOps

Azure DevOps via Azure Arc enabled K8s.

GitHub Actions via Azure Arc enabled K8s.

GitOps Flux v2 via Azure Arc enabled K8s.

3^rd party addon: ArgoCD. 3^rd party addons are not supported by Microsoft’s support policy.

GitOps Flux v2 through Azure Arc enabled Kubernetes is free for AKS-HCI customers.

Azure DevOps

GitHub Actions

GitOps Flux v2

Product Pricing

Product pricing

If you have Azure Hybrid Benefit, you can use AKS-HCI at no additional cost.

If you do not have Azure Hybrid Benefit pricing based on number of workload cluster vCPUs. Management cluster, control plane nodes, load balancers are free.

Unlimited free clusters, pay for on-demand compute of the worker nodes.

Paid tier available with uptime SLA, support for 5k nodes.

Azure Support

AKS in Azure is supported through enterprise class support in the Azure team. You can open support requests in the Azure portal.

SLA

We do not offer SLAs since AKS-HCI runs in your environment.

Paid uptime SLA clusters for production with fixed cost on the API + worker node compute, storage and networking costs.

Blog Post

How do AKS and AKS on Azure Stack HCI compare?

3 Comments