Private AKS Deployment with Application Gateway: Leveraging Terraform and Azure Devops

Introduction

This repository provides a comprehensive guide and toolkit for creating a private Azure Kubernetes Service (AKS) cluster using Terraform. It showcases a detailed process for deploying a private AKS cluster with robust integrations including Azure Container Registry, Azure Storage Account, Azure Key Vault, and more, using Terraform as the infrastructure as code (IaC) tool.

Repository

For complete details and Terraform scripts, visit my GitHub repository at PrivateAKSCluster-Terraform.

This project mirrors the architecture suggested by Microsoft, providing a practical implementation of their recommended private AKS cluster setup. For further reference on the Microsoft architecture, visit their guide here: Azure Architecture - AKS with Application Gateway Ingress Controller.

Description

This sample shows how to create a private AKS clusters using:

• Terraform as infrastructure as code (IaC) tool to build, change, and version the infrastructure on Azure in a safe, repeatable, and efficient way.
• Azure DevOps Pipelines to automate the deployment and undeployment of the entire infrastructure on multiple environments on the Azure platform.

In a private AKS cluster, the API server endpoint is not exposed via a public IP address. Hence, to manage the API server, you will need to use a virtual machine that has access to the AKS cluster's Azure Virtual Network (VNet). This sample deploys a jumpbox virtual machine in the hub virtual network peered with the virtual network that hosts the private AKS cluster. There are several options for establishing network connectivity to the private cluster.

• Create a virtual machine in the same Azure Virtual Network (VNet) as the AKS cluster.
• Use a virtual machine in a separate network and set up Virtual network peering. See the section below for more information on this option.
• Use an Express Route or VPN connection.

Creating a virtual machine in the same virtual network as the AKS cluster or in a peered virtual network is the easiest option. Express Route and VPNs add costs and require additional networking complexity. Virtual network peering requires you to plan your network CIDR ranges to ensure there are no overlapping ranges. For more information, see Create a private Azure Kubernetes Service cluster. For more information on Azure Private Links, see What is Azure Private Link?

In addition, the sample creates a private endpoint to access all the managed services deployed by the Terraform modules via a private IP address:

• Azure Container Registry
• Azure Storage Account
• Azure Key Vault

NOTE
If you want to deploy a private AKS cluster using a public DNS zone to simplify the DNS resolution of the API Server to the private IP address of the private endpoint, you can use this project under my GitHub account or on Azure Quickstart Templates.

Architecture

The following picture shows the high-level architecture created by the Terraform modules included in this sample:

The following picture provides a more detailed view of the infrastructure on Azure.

The architecture is composed of the following elements:

• A hub virtual network with three subnets:
  • AzureBastionSubnet used by Azure Bastion
  • AzureFirewallSubnet used by Azure Firewall
• A new virtual network with three subnets:
  • SystemSubnet used by the AKS system node pool
  • UserSubnet used by the AKS user node pool
  • VmSubnet used by the jumpbox virtual machine and private endpoints
• The private AKS cluster uses a user-defined managed identity to create additional resources like load balancers and managed disks in Azure.
• The private AKS cluster is composed of a:
  • System node pool hosting only critical system pods and services. The worker nodes have node taint which prevents application pods from beings scheduled on this node pool.
  • User node pool hosting user workloads and artifacts.
• An Azure Firewall used to control the egress traffic from the private AKS cluster. For more information on how to lock down your private AKS cluster and filter outbound traffic, see:
  • Control egress traffic for cluster nodes in Azure Kubernetes Service (AKS)
  • Use Azure Firewall to protect Azure Kubernetes Service (AKS) Deployments
• An AKS cluster with a private endpoint to the API server hosted by an AKS-managed Azure subscription. The cluster can communicate with the API server exposed via a Private Link Service using a private endpoint.
• An Azure Bastion resource that provides secure and seamless SSH connectivity to the Vm virtual machine directly in the Azure portal over SSL
• An Azure Container Registry (ACR) to build, store, and manage container images and artifacts in a private registry for all types of container deployments.
• When the ACR SKU is equal to Premium, a Private Endpoint is created to allow the private AKS cluster to access ACR via a private IP address. For more information, see Connect privately to an Azure container registry using Azure Private Link.
• A jumpbox virtual machine used to manage the Azure Kubernetes Service cluster
• A Private DNS Zone for the name resolution of each private endpoint.
• A Virtual Network Link between each Private DNS Zone and both the hub and spoke virtual networks
• A Log Analytics workspace to collect the diagnostics logs and metrics of both the AKS cluster and Vm virtual machine.