Private AKS and ACR Using Private Endpoint – Part 1/2
Published Jan 30 2022 04:33 AM 11.2K Views
Microsoft

Introduction 

Many companies use AKS to deploy their containerized workloads. To secure their infrastructure, they make it private. They also make the Azure Container Registry private. As a result, no external access is allowed outside of the company network boundary. Access to this private environment will be done through the resource VNET, peered VNET, VPN or Express Route.  

This tutorial will provide a guidance to setup a private environment for AKS and ACR with only access from an Azure VM. We will leverage Azure Private Link with Private Endpoint to get access to these resources. 

 

This tutorial will be in two parts. First part will deal with connection between VM and AKS. It will be done with the following steps: 

  1. Create a private AKS cluster within its own VNET 
  2. Create an Azure VM within its own VNET 
  3. Setup connection between the VM and AKS 

Then the second part will deal with connection between VM, AKS and ACR, covering these steps: 

  1. Configure access to ACR using Private Endpoint 
  2. Setup connection between the VM and ACR 
  3. Setup connection between the AKS and ACR 

At the end of this first part, we should have the following architecture implemented for AKS and VM. 

 

HoussemDellai_12-1643544533677.jpeg

 

This tutorial is also available as a video.

 

1. Create a private AKS cluster within its own VNET 

From the Azure portal, create a new AKS cluster and make sure to enable Private cluster. The choice between kubenet and Azure CNI won't impact our demo. Note here that a new VNET and Subnet will be created for this cluster. 

 

HoussemDellai_13-1643544533669.png

 

In "Integrations" section, add and attach an ACR. Even if AKS is private here, ACR remains public, but we’ll make it private later. Click "Review and Create" to create the resources. 

 

HoussemDellai_14-1643544533671.png

 

Check the created resources (AKS, ACR and VNET) inside the AKS Resource Group: 

 

HoussemDellai_15-1643544533672.png

 

Check also the created Private Endpoint, Network Interface and Private DNS zone inside the AKS node Resource Group. 

 

HoussemDellai_16-1643544533679.png

 

Check how the private DNS zone is configured. Note the private IP address (10.240.0.4) within the AKS VNET. 

 

HoussemDellai_17-1643544533674.png

HoussemDellai_0-1643545592514.png

 

Also note the link to the AKS VNET. This means any resource in the AKS VNET will be able to resolve the private IP of the Private Endpoint for communication to the API server. If we have a VM in this VNET, it will be able to connect to AKS with no additional steps.

 

2. Options to connect to a private AKS 

The API server endpoint has only a private IP and no public IP address. To access the API server, we have the following options: 

 

  1. Use the AKS command invoke feature: this is the easiest option as there is no additional steps required. 
  2. Use a VM in the same AKS VNET: requires creating an Azure VM. 
  3. Use a VM in a peered VNET: requires creating an Azure VM and configure the peering. 
  4. Use an Express Route or VPN connection: should be the recommended option for hybrid environments. 

 

3. Setup connection from an Azure VM to a private AKS 

To access the API server, the easiest and recommended option would be to use the AKS command invoke feature. But in this tutorial, and for learning purposes, we’ll use a JumpBox/DevBox VM. This VM will be hosted in its own VNET

We follow these steps to create the VM: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/quick-create-portal. 

To add even more security to the environment, we can leverage Azure Bastion to securely connect to the VM. This is optional for this tutorial. Here are the steps https://docs.microsoft.com/en-us/azure/bastion/tutorial-create-host-portal 

 

In this tutorial, to connect to AKS from our VM, we will use the VNET peering option that that requires 2 steps:  

  1. Create a VNET Peering between VM VNET and AKS VNET. 
  2. Add link to the DevBox VM VNET in the AKS private DNS zone. 

Create a VNET Peering between VM VNET and AKS VNET: go to one of the two VNETs in the portal, then go to Peering and add a new peering. Choose names for both pairs of peering and select the other VNET. 

HoussemDellai_1-1643545624060.png

 

Check the created peering in both VNETs. 

 

HoussemDellai_2-1643545657394.png

HoussemDellai_3-1643545657396.png

 

Add a link to the VM VNET in the AKS Private DNS zone: go to the Private DNS zone of AKS, choose Virtual Network links, then add new link. Choose a name and select the VM VNET. 

  

HoussemDellai_4-1643545716492.png

 

As a result, we should see 2 links in the AKS Private DNS zone: one for the AKS VNET and a second one for the VM VNET. 

HoussemDellai_5-1643545724674.png

 

Now we are all set up and we can connect to the private AKS from the DevBox VM. Let’s connect to the VM, connect to Azure, connect to AKS. And then try to get list of nodes and deploy a Pod into the private cluster. 

 

 $ az login 
 $ az aks get-credentials -g rg-private-aks -n private-aks 
Merged "private-aks" as current context in C:\Users\houssem\.kube\config 
 $ 
 $ kubectl get nodes 
NAME                                STATUS   ROLES   AGE   VERSION 
aks-agentpool-12335431-vmss000000   Ready    agent   15h   v1.22.4 
aks-agentpool-12335431-vmss000001   Ready    agent   15h   v1.22.4 
aks-agentpool-12335431-vmss000002   Ready    agent   15h   v1.22.4 
 $ 
 $ kubectl run nginx --image=nginx:1.21.4 
pod/nginx created 
 $ 
 $ kubectl get pods -w 
NAME    READY   STATUS    RESTARTS   AGE 
nginx   1/1     Running   0          8s 

 

 

 

This a validation that our VM can connect securely through Private Endpoint to the cluster. 

In the next part of this tutorial, we’ll cover the remaining steps: 

  

  1. Configure access to ACR using Private Endpoint. 
  2. Setup connection between the VM and ACR. 
  3. Setup connection between the AKS and ACR. 

Disclaimer

The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

2 Comments
Co-Authors
Version history
Last update:
‎Feb 03 2022 11:12 PM
Updated by: