Introduction
The objective of this tutorial is to create a private environment for AKS, ACR and a JumpBox VM. AKS and ACR will be accessible only through Private Endpoint.
As of part 1, our VM did connect securely through Private Endpoint to the private AKS cluster. In this next part, we’ll cover the remaining steps:
At the end of the tutorial, we should have the following architecture implemented for AKS, ACR and VM. Storage Account and Key Vault are just for illustration as we still can use Private Endpoint to access them privately.
Part 1 is available here: Private AKS and ACR Using Private Endpoint – Part 1/2 (microsoft.com)
This tutorial is also available as a video.
1. Configure access to ACR using Private Endpoint
In the previous tutorial, we had already created an ACR attached to AKS. By default, ACR is public. Here want to disable its public FQDN and make it accessible only using Private Endpoint (aka PE). Private Endpoint will create a private IP address in a Subnet/VNET. Documentation about private ACR is here.
Private Endpoint will disable NSGs on its host Subnet. We don’t want this to happen to AKS Subnet, so we can create another Subnet in AKS VNET dedicated for Private Endpoints. Let’s do that from the Azure portal.
Note: It is also possible to create a dedicated VNET for PEs in scenarios where we use Hub and Spoke model.
We can have a Spoke for PEs peered only with the Spokes that needs access to the PEs resources.
Now let’s got to create the Private Endpoint for ACR. Private Endpoint is a feature of the Premium plan of ACR.
Go to Overview then Update and update the plan to Premium.
Go to Networking, you can see the more options now with the premium plan. Set Public network access to Disabled to disable public access to the ACR through the internet.
Our goal here is to connect to the ACR (from the DevBox VM and AKS) using the private endpoint.
Using the private endpoint will allow only resources from our network to access ACR.
Then select Private access and create private endpoint connection:
Let’s follow the 5 steps to create the private endpoint for ACR. First, we choose the resource group where to create the Private Endpoint and its Network Interface Card (NIC), here I choose the AKS node resource group.
In the second step, we keep the default configuration.
In configuration section, we choose the VNET and Subnet where to attach the Network Interface for the PE. Here I choose the AKS VNET, but I choose a dedicated subnet (the one created earlier). Note the information about disabling NSGs on this subnet.
Then we enable Integrate with private DNS zone. This will create a Private DNS Zone and add a link to the VNET. This will allow resources deployed in that VNET to be able to resolve the private endpoint link.
I choose to deploy the Private DNS Zone into the AKS node resource group as I've done with the PE.
Finally, we create the PE.
We go into the AKS nodes resource group, and we will see the Private Endpoint, NIC card and the Private DNS Zone.
Check the config of the Private DNS Zone. Note the private IP addresses (10.241.0.4 for the Storage Account behind ACR and 10.241.0.5 for ACR).
Also note the private link: acr1privateaks.privatelink.azurecr.io
Note the link added to the AKS VNET. This means each resource in this VNET can connect to the ACR through the PE. Which also means our AKS cluster can connect to it to pull the images.
Let’s also check the Private Endpoint configuration. Note how it relies on the Private DNS Zone to resolve the ACR link.
Note: The NIC card is in read-only mode, and it follows the same life cycle as the PE.
2. Setup connection between the VM and ACR
Our objective now is to connect to private ACR from the VM and push an image. We will use a DevBox/JumpBox VM to push the image. If this VM is in the same AKS VNET, it will be able to connect directly to ACR. But in this demo, we will use a VM hosted in its own VNET.
In real life this is the case for architectures where a JumpBox or a DevOps build agent in one VNET are used to deploy to resources in other VNETs.
For the DevBox VM to connect to the private ACR through the Private Endpoint sitting in another VNET, we need to do 2 steps:
Let’s go to one of the two VNETs and create a VNET Peering between VM VNET and ACR Private Endpoint VNET. But this was already done in the previous tutorial! Just in case, check the created peering:
Let’s add link to the JumpBox VM's VNET in the ACR Private DNS Zone. In the ACR Private DNS Zone, add a link to the JumpBox VM's VNET.
Now we are ready to connect from the VM to private ACR. Take a moment to install Azure CLI, Docker and kubectl in the VM.
We go to the ACR to import an image from MCR using the DevBox VM.
$ $acrName="acr1privateaks"
$ az acr login -n $acrName
Login Succeeded
$ az acr import -n $acrName --source mcr.microsoft.com/oss/nginx/nginx:1.21.4 --image nginx:1.21.4
Alternatively, we can also pull an image from MCR, tag it, and push it to ACR.
$ docker pull mcr.microsoft.com/hello-world
$ docker tag mcr.microsoft.com/hello-world "$acrName.azurecr.io/hello-world:v1.0"
$ docker push "$acrName.azurecr.io/hello-world:v1.0"
Great! Now the DevBox VM connects to private ACR and pushes the images.
3. Setup connection between AKS and private ACR
To enable the connection between AKS and private ACR:
For AKS to connect to Private ACR: just add the AKS VNET as a Link to the Private DNS Zone of the private ACR.
From the private DNS Zone, we can see the two links.
Now, we can validate that private AKS connects successfully to private ACR. Let’s deploy a container from ACR and watch how that deploys successfully.
$ kubectl run nginx --image="$acrName.azurecr.io/nginx:1.21.4"
$ kubectl get pods -w
Conclusion and final notes
We had learned in this tutorial how to provision a secure AKS environment using Private Endpoint. The only allowed connections will be through Bastion using the DevBox VM. AKS and ACR both are private and accessible only from the private VNET network using Private Endpoint.
Typically, with AKS, we would also use Storage Account, Key Vault, Azure SQL. We can make these resources private and accessible only through Private Endpoint.
It is possible to disable the public FQDN of AKS API Server. Details here:
https://docs.microsoft.com/en-us/azure/aks/private-clusters#disable-public-fqdn
To resume, as it is mentioned briefly here, virtual network peering is one way to access your private cluster. To use virtual network peering, you need to set up a link between virtual network and the private DNS zone.
An important note from this source: Multiple private endpoints can be created using the same private link resource. For a single network using a common DNS server configuration, the recommended practice is to use a single private endpoint for a given private link resource. Use this practice to avoid duplicate entries or conflicts in DNS resolution.
Another important note from this source, if you plan on using private endpoints to access resources from an on-premises system, deploy the endpoints in your hub. With this arrangement, you can take advantage of some of the benefits:
If you're planning on accessing resources from applications that you've deployed in Azure, the situation's different:
The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.