Introduction
After deploying an application and its services into a Kubernetes cluster, a question rises on the surface, how to access it with a custom domain name ? A simple solution would be to create an A record that points the domain name into the service IP address. This could be done manually, so it will be too hard to scale as you add many services. And this could be fully automated by using External DNS! This tutorial describes how to manage custom domain names in Azure DNS using External DNS in AKS.
External DNS is a Kubernetes controller that watches for new Ingresses and Services with specific annotations, then creates corresponding DNS records in Azure DNS. It is available as an opensource project in Github: https://github.com/kubernetes-sigs/external-dns. It supports more than 30 DNS providers including Azure DNS and Private DNS Zone.
External DNS pods authenticates to Azure DNS using one of three methods:
- Service principal.
- Kubelet Managed Identity.
- User assigned Managed Identity controlled by AAD Pod Identity.
Note: Pod Identity is deprecated and will be replaced by Workload Identity. However, ExternalDNS dos not support yet Workload Identity.
Note: If you want to use Kubelet Managed Identity, giving it the Contributor role on the DNS zone is not secure by default. That is because any pod in the cluster can access it. To mitigate this issue, you need to implement a Network Policy that restricts access to the IMDS endpoint to only the ExternalDNS pods.
In this tutorial, you will work with Service Principal.
This article is available as a video in this link: External DNS for Kubernetes.
1. Create an AKS cluster with an ingress controller
Create an AKS cluster.
$AKS_RG="rg-aks-cluster"
$AKS_NAME="aks-cluster"
az group create -n $AKS_RG -l Yousteurope
az aks create -g $AKS_RG -n $AKS_NAME `
--kubernetes-version "1.25.5" `
--node-count 3 `
--network-plugin azure
az aks get-credentials -n $AKS_NAME -g $AKS_RG --overwrite-existing
Install nginx ingress controller to use it later.
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx `
--create-namespace `
--namespace ingress-nginx `
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz
2. Create Azure DNS Zone, or use an existing one
You can create a new Azure DNS Zone with or without delegated domain name. Without delegated domain name means it will not be able to publicly resolve the domain name. But you will still see the created DNS records.
In this lab, I use a delegated domain name: houssem.cloud. Replace it with your own.
$DNS_ZONE_NAME="houssem.cloud"
$DNS_ZONE_RG="rg-azure-dns"
az group create -n $DNS_ZONE_RG -l Yousteurope
az network dns zone create -g $DNS_ZONE_RG -n $DNS_ZONE_NAME
3. Create a service principal for ExternalDNS
ExternalDNS will connect to Azure DNS to change its configuration. So, it needs to be authenticated. As mentioned before, You will be using a Service Principal.
$EXTERNALDNS_SPN_NAME="spn-external-dns-aks"
# Create the service principal
$DNS_SPN=$(az ad sp create-for-rbac --name $EXTERNALDNS_SPN_NAME)
$EXTERNALDNS_SPN_APP_ID=$(echo $DNS_SPN | jq -r '.appId')
$EXTERNALDNS_SPN_PASSWORD=$(echo $DNS_SPN | jq -r '.password')
4. Assign the RBAC for the service principal
Grant access to Azure DNS zone for the service principal.
# fetch DNS id and RG used to grant access to the service principal
$DNS_ZONE_ID=$(az network dns zone show -n $DNS_ZONE_NAME -g $DNS_ZONE_RG --query "id" -o tsv)
$DNS_ZONE_RG_ID=$(az group show -g $DNS_ZONE_RG --query "id" -o tsv)
# assign reader to the resource group
az role assignment create --role "Reader" --assignee $EXTERNALDNS_SPN_APP_ID --scope $DNS_ZONE_RG_ID
# assign contributor to DNS Zone itself
az role assignment create --role "DNS Zone Contributor" --assignee $EXTERNALDNS_SPN_APP_ID --scope $DNS_ZONE_ID
Verify role assignments.
az role assignment list --all --assignee $EXTERNALDNS_SPN_APP_ID -o table
# Principal Role Scope
# ------------------------------------ -------------------- ----------------------------------------------------------------------------------------------------------------------------------
# 9cc6c0d1-99a3-4d86-9df4-a84df55b8232 Reader /subscriptions/82f6d75e-85f4-434a-ab74-5dddd9fa8910/resourceGroups/rg-azure-dns
# 9cc6c0d1-99a3-4d86-9df4-a84df55b8232 DNS Zone Contributor /subscriptions/82f6d75e-85f4-434a-ab74-5dddd9fa8910/resourceGroups/rg-azure-dns/providers/Microsoft.Network/dnszones/houssem.cloud
5. Create a Kubernetes secret for the service principal
ExternalDNS expects to find the Service Principal credentials in a JSON file called azure.json saved as a Kubernetes secret. Let's create the file.
@"
{
"tenantId": "$(az account show --query tenantId -o tsv)",
"subscriptionId": "$(az account show --query id -o tsv)",
"resourceGroup": "$DNS_ZONE_RG",
"aadClientId": "$EXTERNALDNS_SPN_APP_ID",
"aadClientSecret": "$EXTERNALDNS_SPN_PASSWORD"
}
"@ > azure.json
cat azure.json
# {
# "tenantId": "16b3c013-d300-468d-ac64-7eda0820b6d3",
# "subscriptionId": "82f6d75e-85f4-434a-ab74-5dddd9fa8910",
# "resourceGroup": "rg-dns-zone-houssem-cloud",
# "aadClientId": "9cc6c0d1-99a3-4d86-9df4-a84df55b8232",
# "aadClientSecret": "LJS8Q~ZeuAPJfE7Hjzy6bYZ8NQ4O5YrlJfATxbL6"
# }
Deploy the credentials as a Kubernetes secret.
kubectl create namespace external-dns
# namespace/external-dns created
kubectl create secret generic azure-config-file -n external-dns --from-file azure.json
# secret/azure-config-file created
Verify secret created
kubectl describe secret azure-config-file -n external-dns
# Name: azure-config-file
# Namespace: external-dns
# Labels: <none>
# Annotations: <none>
#
# Type: Opaque
#
# Data
# ====
# azure.json: 552 bytes
6. Deploy External DNS
ExternalDNS could be deployed through raw YAML manifest, Helm chart or as an operator. For simplicity, you will be using official YAML deployment available here: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/azure.md#manifest-for-clusters-with-rbac-enabled-cluster-access. Refer to this link to check any possible future change in YAML.
Before deploying the YAML, change the namespace name in ClusterRoleBinding in external-dns.yaml file.
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: external-dns
rules:
- apiGroups: [""]
resources: ["services","endpoints","pods", "nodes"]
verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get","watch","list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: external-dns-viewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: external-dns
subjects:
- kind: ServiceAccount
name: external-dns
namespace: external-dns # default
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-dns
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: external-dns
template:
metadata:
labels:
app: external-dns
spec:
serviceAccountName: external-dns
containers:
- name: external-dns
image: registry.k8s.io/external-dns/external-dns:v0.13.2
args:
- --source=service
- --source=ingress
- --provider=azure
- --txt-prefix=externaldns-
volumeMounts:
- name: azure-config-file
mountPath: /etc/kubernetes
readOnly: true
volumes:
- name: azure-config-file
secret:
secretName: azure-config-file
kubectl apply -f external-dns.yaml -n external-dns
# serviceaccount/external-dns created
# clusterrole.rbac.authorization.k8s.io/external-dns created
# clusterrolebinding.rbac.authorization.k8s.io/external-dns-vieYour created
# deployment.apps/external-dns created
Note: To deploy ExternalDNS using Helm charts, checkout these resources: https://artifacthub.io/packages/helm/bitnami/external-dns https://github.com/bitnami/charts/tree/main/bitnami/external-dns/#installing-the-chart
Verify deployment.
kubectl get pods,sa -n external-dns
NAME READY STATUS RESTARTS AGE
pod/external-dns-5fd5797df-xklxn 1/1 Running 0 96s
NAME SECRETS AGE
serviceaccount/default 0 96m
serviceaccount/external-d
7. Using External DNS with Kubernetes services
You will create a public service of type LoadBalancer. This will create a new public IP address to access the service. Then add an annotation external-dns.alpha.kubernetes.io/hostname with value the custom domain name. This annotation will be red by External DNS to add the IP address to the DNS Zone (in this case app01.houssem.cloud).
You will use this YAML template.
# app-lb.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app01
spec:
selector:
matchLabels:
app: app01
template:
metadata:
labels:
app: app01
spec:
containers:
- image: mcr.microsoft.com/dotnet/samples:aspnetapp
name: aspnetapp
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: app01-svc
annotations:
external-dns.alpha.kubernetes.io/hostname: app01.houssem.cloud # external-dns configuration
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: app01
type: LoadBalancer
kubectl apply -f app-lb.yaml
# deployment.apps/nginx created
# service/nginx-svc created
kubectl get pods,svc
# NAME READY STATUS RESTARTS AGE
# pod/app01-67745dc95d-bwzgf 1/1 Running 0 100s
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# service/app01-svc LoadBalancer 10.0.95.113 20.86.202.21 80:31067/TCP 100s
# service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 2m30s
Check what is happening in the external DNS pod. Note how External DNS detected the annotation and is creating an A record to the public IP address of the service (20.103.4.205).
kubectl logs external-dns-5fd5797df-xklxn -n external-dns
# time="2023-03-06T09:01:15Z" level=info msg="Updating A record named 'app01' to '20.103.4.205' for Azure DNS zone 'houssem.cloud'."
# time="2023-03-06T09:01:16Z" level=info msg="Updating TXT record named 'externaldns-app01' to '\"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/default/app01-svc\"' for Azure DNS zone 'houssem.cloud'."
# time="2023-03-06T09:01:16Z" level=info msg="Updating TXT record named 'externaldns-a-app01' to '\"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/default/app01-svc\"' for Azure DNS zone 'houssem.cloud'."
Check the DNS record is created by external DNS.
az network dns record-set a list -g $DNS_ZONE_RG --zone-name $DNS_ZONE_NAME
# [{
# "aRecords": [
# {
# "ipv4Address": "20.103.57.97"
# }
# ],
# "aaaaRecords": null,
# "caaRecords": null,
# "cnameRecord": null,
# "etag": "99b46f74-8388-44d1-80e9-2aafe1f4802d",
# "fqdn": "myapp.houssem.cloud.",
# "id": "/subscriptions/82f6d75e-85f4-434a-ab74-5dddd9fa8910/resourceGroups/rg-dns-zone-houssem-cloud/providers/Microsoft.Network/dnszones/houssem.cloud/A/myapp",
# "metadata": null,
# "mxRecords": null,
# "name": "myapp",
# "nsRecords": null,
# "provisioningState": "Succeeded",
# "ptrRecords": null,
# "resourceGroup": "rg-dns-zone-houssem-cloud",
# "soaRecord": null,
# "srvRecords": null,
# "targetResource": {
# "id": null
# },
# "ttl": 300,
# "txtRecords": null,
# "type": "Microsoft.Network/dnszones/A"
# }]
8. Create a sample app exposed through ingress
You will expose and application through an ingress controller. In the ingress resource you will add a configuration that will be used by External DNS to manage domain names in Azure DNS. That configuration is native to ingress resources which is the host.
# app-ingress.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app02
spec:
selector:
matchLabels:
app: app02
template:
metadata:
labels:
app: app02
spec:
containers:
- image: mcr.microsoft.com/dotnet/samples:aspnetapp
name: aspnetapp
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: app02-svc
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: app02
type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app02-ingress
spec:
ingressClassName: nginx
rules:
- host: app02.houssemd.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: app02-svc
port:
number: 80
kubectl apply -f app-ingress.yaml
# deployment.apps/app02 created
# service/app02-svc created
# ingress.networking.k8s.io/app02-ingress created
kubectl get pods,svc,ingress
# NAME READY STATUS RESTARTS AGE
# pod/app02-9bdd6845f-vh422 1/1 Running 0 92s
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# service/app02-svc ClusterIP 10.0.74.196 <none> 80/TCP 92s
# service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 2m30s
# NAME CLASS HOSTS ADDRESS PORTS AGE
# ingress.networking.k8s.io/app02-ingress nginx app02.houssem.cloud 20.73.123.67 80 92s
Check the DNS record is created by external DNS
az network dns record-set a list -g $DNS_ZONE_RG --zone-name $DNS_ZONE_NAME
# [
# {
# "aRecords": [
# {
# "ipv4Address": "20.73.123.67"
# }
# ],
# "aaaaRecords": null,
# "caaRecords": null,
# "cnameRecord": null,
# "etag": "f1038e1a-85d3-440e-bd91-fc6f8252e3f1",
# "fqdn": "app02.houssem.cloud.",
# "id": "/subscriptions/82f6d75e-85f4-434a-ab74-5dddd9fa8910/resourceGroups/rg-dns-zone-houssem-cloud/providers/Microsoft.Network/dnszones/houssem.cloud/A/app02",
# "metadata": null,
# "mxRecords": null,
# "name": "app02",
# "nsRecords": null,
# "provisioningState": "Succeeded",
# "ptrRecords": null,
# "resourceGroup": "rg-dns-zone-houssem-cloud",
# "soaRecord": null,
# "srvRecords": null,
# "targetResource": {
# "id": null
# },
# "ttl": 300,
# "txtRecords": null,
# "type": "Microsoft.Network/dnszones/A"
# }
# ]
Let's check the app and DNS resolution. Just open the URL on the browser.
Let us check the Azure DNS zone configuration. Note the A records was added with public IP for service and ingress controller.
Conclusion
You learned in this tutorial how to configure custom domain names in Azure DNS for external services using External DNS. More details about the project are available here: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/azure.md.
For exposing custom domain names inside the kubernetes cluster, you can use Core DNS (previously Kube DNS).
Disclaimer
The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.
Updated May 01, 2023
Version 1.0
HoussemDellai
Microsoft
Microsoft