Core Infrastructure and Security Blog

10 MIN READ

Kubernetes External DNS for Azure DNS & AKS

Microsoft

May 01, 2023

Introduction

After deploying an application and its services into a Kubernetes cluster, a question rises on the surface, how to access it with a custom domain name ? A simple solution would be to create an A record that points the domain name into the service IP address. This could be done manually, so it will be too hard to scale as you add many services. And this could be fully automated by using External DNS! This tutorial describes how to manage custom domain names in Azure DNS using External DNS in AKS.

External DNS is a Kubernetes controller that watches for new Ingresses and Services with specific annotations, then creates corresponding DNS records in Azure DNS. It is available as an opensource project in Github: https://github.com/kubernetes-sigs/external-dns. It supports more than 30 DNS providers including Azure DNS and Private DNS Zone.

External DNS pods authenticates to Azure DNS using one of three methods:

Service principal.
Kubelet Managed Identity.
User assigned Managed Identity controlled by AAD Pod Identity.

Note: Pod Identity is deprecated and will be replaced by Workload Identity. However, ExternalDNS dos not support yet Workload Identity.

Note: If you want to use Kubelet Managed Identity, giving it the Contributor role on the DNS zone is not secure by default. That is because any pod in the cluster can access it. To mitigate this issue, you need to implement a Network Policy that restricts access to the IMDS endpoint to only the ExternalDNS pods.

In this tutorial, you will work with Service Principal.

This article is available as a video in this link: External DNS for Kubernetes.

1. Create an AKS cluster with an ingress controller

Create an AKS cluster.

$AKS_RG="rg-aks-cluster"
$AKS_NAME="aks-cluster"

az group create -n $AKS_RG -l Yousteurope

az aks create -g $AKS_RG -n $AKS_NAME `
              --kubernetes-version "1.25.5" `
              --node-count 3 `
              --network-plugin azure

az aks get-credentials -n $AKS_NAME -g $AKS_RG --overwrite-existing

Install nginx ingress controller to use it later.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm repo update

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx `
     --create-namespace `
     --namespace ingress-nginx `
     --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz

2. Create Azure DNS Zone, or use an existing one

You can create a new Azure DNS Zone with or without delegated domain name. Without delegated domain name means it will not be able to publicly resolve the domain name. But you will still see the created DNS records.

In this lab, I use a delegated domain name: houssem.cloud. Replace it with your own.

$DNS_ZONE_NAME="houssem.cloud"
$DNS_ZONE_RG="rg-azure-dns"

az group create -n $DNS_ZONE_RG -l Yousteurope

az network dns zone create -g $DNS_ZONE_RG -n $DNS_ZONE_NAME

3. Create a service principal for ExternalDNS

ExternalDNS will connect to Azure DNS to change its configuration. So, it needs to be authenticated. As mentioned before, You will be using a Service Principal.

$EXTERNALDNS_SPN_NAME="spn-external-dns-aks"

# Create the service principal
$DNS_SPN=$(az ad sp create-for-rbac --name $EXTERNALDNS_SPN_NAME)
$EXTERNALDNS_SPN_APP_ID=$(echo $DNS_SPN | jq -r '.appId')
$EXTERNALDNS_SPN_PASSWORD=$(echo $DNS_SPN | jq -r '.password')

4. Assign the RBAC for the service principal

Grant access to Azure DNS zone for the service principal.

# fetch DNS id and RG used to grant access to the service principal
$DNS_ZONE_ID=$(az network dns zone show -n $DNS_ZONE_NAME -g $DNS_ZONE_RG --query "id" -o tsv)
$DNS_ZONE_RG_ID=$(az group show -g $DNS_ZONE_RG --query "id" -o tsv)

# assign reader to the resource group
az role assignment create --role "Reader" --assignee $EXTERNALDNS_SPN_APP_ID --scope $DNS_ZONE_RG_ID

# assign contributor to DNS Zone itself
az role assignment create --role "DNS Zone Contributor" --assignee $EXTERNALDNS_SPN_APP_ID --scope $DNS_ZONE_ID

Verify role assignments.

az role assignment list --all --assignee $EXTERNALDNS_SPN_APP_ID -o table
# Principal                                         Role                        Scope
# ------------------------------------  --------------------  ----------------------------------------------------------------------------------------------------------------------------------
# 9cc6c0d1-99a3-4d86-9df4-a84df55b8232  Reader                /subscriptions/82f6d75e-85f4-434a-ab74-5dddd9fa8910/resourceGroups/rg-azure-dns
# 9cc6c0d1-99a3-4d86-9df4-a84df55b8232  DNS Zone Contributor  /subscriptions/82f6d75e-85f4-434a-ab74-5dddd9fa8910/resourceGroups/rg-azure-dns/providers/Microsoft.Network/dnszones/houssem.cloud

5. Create a Kubernetes secret for the service principal

ExternalDNS expects to find the Service Principal credentials in a JSON file called azure.json saved as a Kubernetes secret. Let's create the file.

@"
{
  "tenantId": "$(az account show --query tenantId -o tsv)",
  "subscriptionId": "$(az account show --query id -o tsv)",
  "resourceGroup": "$DNS_ZONE_RG",
  "aadClientId": "$EXTERNALDNS_SPN_APP_ID",
  "aadClientSecret": "$EXTERNALDNS_SPN_PASSWORD"
}
"@ > azure.json

cat azure.json
# {
#   "tenantId": "16b3c013-d300-468d-ac64-7eda0820b6d3",
#   "subscriptionId": "82f6d75e-85f4-434a-ab74-5dddd9fa8910",
#   "resourceGroup": "rg-dns-zone-houssem-cloud",
#   "aadClientId": "9cc6c0d1-99a3-4d86-9df4-a84df55b8232",
#   "aadClientSecret": "LJS8Q~ZeuAPJfE7Hjzy6bYZ8NQ4O5YrlJfATxbL6"
# }

Deploy the credentials as a Kubernetes secret.

kubectl create namespace external-dns
# namespace/external-dns created

kubectl create secret generic azure-config-file -n external-dns --from-file azure.json
# secret/azure-config-file created

Verify secret created

kubectl describe secret azure-config-file -n external-dns
# Name:         azure-config-file
# Namespace:    external-dns
# Labels:       <none>
# Annotations:  <none>
# 
# Type:  Opaque
# 
# Data
# ====
# azure.json:  552 bytes

6. Deploy External DNS

ExternalDNS could be deployed through raw YAML manifest, Helm chart or as an operator. For simplicity, you will be using official YAML deployment available here: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/azure.md#manifest-for-clusters-with-rbac-enabled-cluster-access. Refer to this link to check any possible future change in YAML.

Before deploying the YAML, change the namespace name in ClusterRoleBinding in external-dns.yaml file.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
  - apiGroups: [""]
    resources: ["services","endpoints","pods", "nodes"]
    verbs: ["get","watch","list"]
  - apiGroups: ["extensions","networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get","watch","list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
  - kind: ServiceAccount
    name: external-dns
    namespace: external-dns # default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
        - name: external-dns
          image: registry.k8s.io/external-dns/external-dns:v0.13.2
          args:
            - --source=service
            - --source=ingress
            - --provider=azure
            - --txt-prefix=externaldns-
          volumeMounts:
            - name: azure-config-file
              mountPath: /etc/kubernetes
              readOnly: true
      volumes:
        - name: azure-config-file
          secret:
            secretName: azure-config-file

kubectl apply -f external-dns.yaml -n external-dns
# serviceaccount/external-dns created
# clusterrole.rbac.authorization.k8s.io/external-dns created
# clusterrolebinding.rbac.authorization.k8s.io/external-dns-vieYour created
# deployment.apps/external-dns created

Note: To deploy ExternalDNS using Helm charts, checkout these resources: https://artifacthub.io/packages/helm/bitnami/external-dns https://github.com/bitnami/charts/tree/main/bitnami/external-dns/#installing-the-chart

Verify deployment.

kubectl get pods,sa -n external-dns
NAME                               READY   STATUS    RESTARTS   AGE
pod/external-dns-5fd5797df-xklxn   1/1     Running   0          96s
NAME                          SECRETS   AGE
serviceaccount/default        0         96m
serviceaccount/external-d

7. Using External DNS with Kubernetes services

You will create a public service of type LoadBalancer. This will create a new public IP address to access the service. Then add an annotation external-dns.alpha.kubernetes.io/hostname with value the custom domain name. This annotation will be red by External DNS to add the IP address to the DNS Zone (in this case app01.houssem.cloud).

You will use this YAML template.

# app-lb.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app01
spec:
  selector:
    matchLabels:
      app: app01
  template:
    metadata:
      labels:
        app: app01
    spec:
      containers:
        - image: mcr.microsoft.com/dotnet/samples:aspnetapp
          name: aspnetapp
          ports:
          - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: app01-svc
  annotations:
    external-dns.alpha.kubernetes.io/hostname: app01.houssem.cloud # external-dns configuration
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: app01
  type: LoadBalancer

kubectl apply -f app-lb.yaml 
# deployment.apps/nginx created
# service/nginx-svc created

kubectl get pods,svc
# NAME                                              READY   STATUS    RESTARTS   AGE
# pod/app01-67745dc95d-bwzgf      1/1        Running   0                 100s

# NAME                       TYPE                 CLUSTER-IP    EXTERNAL-IP    PORT(S)            AGE
# service/app01-svc    LoadBalancer   10.0.95.113     20.86.202.21     80:31067/TCP   100s
# service/kubernetes   ClusterIP          10.0.0.1           <none>            443/TCP            2m30s

Check what is happening in the external DNS pod. Note how External DNS detected the annotation and is creating an A record to the public IP address of the service (20.103.4.205).

kubectl logs external-dns-5fd5797df-xklxn -n external-dns
# time="2023-03-06T09:01:15Z" level=info msg="Updating A record named 'app01' to '20.103.4.205' for Azure DNS zone 'houssem.cloud'."
# time="2023-03-06T09:01:16Z" level=info msg="Updating TXT record named 'externaldns-app01' to '\"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/default/app01-svc\"' for Azure DNS zone 'houssem.cloud'."
# time="2023-03-06T09:01:16Z" level=info msg="Updating TXT record named 'externaldns-a-app01' to '\"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/default/app01-svc\"' for Azure DNS zone 'houssem.cloud'."

Check the DNS record is created by external DNS.

az network dns record-set a list -g $DNS_ZONE_RG --zone-name $DNS_ZONE_NAME
# [{
#     "aRecords": [
#       {
#         "ipv4Address": "20.103.57.97"
#       }
#     ],
#     "aaaaRecords": null,
#     "caaRecords": null,
#     "cnameRecord": null,
#     "etag": "99b46f74-8388-44d1-80e9-2aafe1f4802d",
#     "fqdn": "myapp.houssem.cloud.",
#     "id": "/subscriptions/82f6d75e-85f4-434a-ab74-5dddd9fa8910/resourceGroups/rg-dns-zone-houssem-cloud/providers/Microsoft.Network/dnszones/houssem.cloud/A/myapp",
#     "metadata": null,
#     "mxRecords": null,
#     "name": "myapp",
#     "nsRecords": null,
#     "provisioningState": "Succeeded",
#     "ptrRecords": null,
#     "resourceGroup": "rg-dns-zone-houssem-cloud",
#     "soaRecord": null,
#     "srvRecords": null,
#     "targetResource": {
#       "id": null
#     },
#     "ttl": 300,
#     "txtRecords": null,
#     "type": "Microsoft.Network/dnszones/A"
#   }]

8. Create a sample app exposed through ingress

You will expose and application through an ingress controller. In the ingress resource you will add a configuration that will be used by External DNS to manage domain names in Azure DNS. That configuration is native to ingress resources which is the host.

# app-ingress.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app02
spec:
  selector:
    matchLabels:
      app: app02
  template:
    metadata:
      labels:
        app: app02
    spec:
      containers:
        - image: mcr.microsoft.com/dotnet/samples:aspnetapp
          name: aspnetapp
          ports:
          - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: app02-svc
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: app02
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app02-ingress
spec:
  ingressClassName: nginx
  rules:
    - host: app02.houssemd.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app02-svc
                port:
                  number: 80

kubectl apply -f app-ingress.yaml
# deployment.apps/app02 created
# service/app02-svc created
# ingress.networking.k8s.io/app02-ingress created

kubectl get pods,svc,ingress
# NAME                                             READY   STATUS     RESTARTS   AGE
# pod/app02-9bdd6845f-vh422       1/1          Running   0                 92s

# NAME                       TYPE            CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
# service/app02-svc    ClusterIP      10.0.74.196     <none>            80/TCP          92s
# service/kubernetes   ClusterIP      10.0.0.1           <none>            443/TCP        2m30s

# NAME                                                         CLASS   HOSTS                          ADDRESS        PORTS   AGE
# ingress.networking.k8s.io/app02-ingress   nginx    app02.houssem.cloud   20.73.123.67   80         92s

Check the DNS record is created by external DNS

az network dns record-set a list -g $DNS_ZONE_RG --zone-name $DNS_ZONE_NAME
# [
# {
#     "aRecords": [
#       {
#         "ipv4Address": "20.73.123.67"
#       }
#     ],
#     "aaaaRecords": null,
#     "caaRecords": null,
#     "cnameRecord": null,
#     "etag": "f1038e1a-85d3-440e-bd91-fc6f8252e3f1",
#     "fqdn": "app02.houssem.cloud.",
#     "id": "/subscriptions/82f6d75e-85f4-434a-ab74-5dddd9fa8910/resourceGroups/rg-dns-zone-houssem-cloud/providers/Microsoft.Network/dnszones/houssem.cloud/A/app02",
#     "metadata": null,
#     "mxRecords": null,
#     "name": "app02",
#     "nsRecords": null,
#     "provisioningState": "Succeeded",
#     "ptrRecords": null,
#     "resourceGroup": "rg-dns-zone-houssem-cloud",
#     "soaRecord": null,
#     "srvRecords": null,
#     "targetResource": {
#       "id": null
#     },
#     "ttl": 300,
#     "txtRecords": null,
#     "type": "Microsoft.Network/dnszones/A"
#   }
# ]

Let's check the app and DNS resolution. Just open the URL on the browser.

Let us check the Azure DNS zone configuration. Note the A records was added with public IP for service and ingress controller.

Conclusion

You learned in this tutorial how to configure custom domain names in Azure DNS for external services using External DNS. More details about the project are available here: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/azure.md.

For exposing custom domain names inside the kubernetes cluster, you can use Core DNS (previously Kube DNS).

Disclaimer
The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

Updated May 01, 2023

Version 1.0

HoussemDellai