Deploying multi-tier application on Azure Kubernetes Service
Published Mar 05 2021 10:28 AM 12.1K Views


This blog demonstrates a multi-tier application deployment on to Azure Kubernetes Service along with several other Azure managed services such as Azure Database for MySQL, Azure Functions, etc.

Note: There may be few features that are used in this blog such as Azure Active Directory Pod Identity are still in preview, these features are not recommended for production deployment. 






We will create and setup the infrastructure including the following services:

  1. Azure Container Registry for storing images
  2. AAD Enabled, Managed AKS Cluster with the below addons and components
    1. Application Gateway Ingress Controller Addon
    2. Monitoring Addon
    3. LetsEncrypt for Certificate authority
    4. KEDA runtime for Azure Functions on Kubernetes clusters
  3. Azure Database for MySQL Service
  4. Azure Storage Queues
  5. DNS Zone for custom domain
  6. SendGrid Account for email service


Cluster Creation

Clone repository

git clone
cd multitiered-app-on-azure
# You could use script.azcli as your working file. Don't run the script as is!


Initialize variables

# Add variables (sample values below change as required)
# Identity name must be lower case
# Storage name must be lower case and globally unique


Login to Azure

az login
az account set -s $subscriptionId


Register to AKS preview features

# Follow
az feature register --name EnablePodIdentityPreview --namespace Microsoft.ContainerService
az provider register -n Microsoft.ContainerService
az extension add --name aks-preview
az extension update --name aks-preview


Create Resource Group

az group create --name $resourcegroupName --location $location


Create ACR

az acr create --resource-group $resourcegroupName --name $acrName --sku Standard


Get Object ID of the AAD Group (create AAD Group and add the members, in this case: expenses-ad-azure)

This group is needed as admin group for the cluster to grant cluster admin permissions. You can use an existing Azure AD group, or create a new one. Record the object ID of your Azure AD group.

az ad group create --display-name expenses-ad-azure --mail-nickname expenses-ad-azure --description 'Group for Managing AAD based AKS cluster'
objectId=$(az ad group list --filter "displayname eq 'expenses-ad-azure'" --query '[].objectId' -o tsv)


Create an AKS-managed Azure AD cluster with AGIC add-on and AAD Pod Identity

az aks create \
    -n $clusterName \
    -g $resourcegroupName \
    --network-plugin azure \
    --enable-managed-identity \
    -a ingress-appgw --appgw-name $appGtwyName \
    --appgw-subnet-cidr "" \
    --enable-aad \
    --enable-pod-identity \
    --aad-admin-group-object-ids $objectId \
    --generate-ssh-keys \
    --attach-acr $acrName

# Enable monitoring on the cluster
az aks enable-addons -a monitoring -n $clusterName -g $resourcegroupName


Add Public IP to custom domain

# Get Node Resource Group
nodeRG=$(az aks show --resource-group $resourcegroupName --name $clusterName --query nodeResourceGroup -o tsv)

# Get Public IP created by App Gtwy in AKS created cluster
appIP=$(az network public-ip show -g $nodeRG -n $appGtwyName-appgwpip --query ipAddress -o tsv)

# Create DNS zone, if not created
az network dns zone create -g $dnsRG -n $domainName

# Once created, add Nameservers in the domain provider (eg go daddy, may take sometime to update the name servers)
az network dns record-set a add-record --resource-group $resourcegroupName --zone-name $domainName --record-set-name $subDomain --ipv4-address $appIP

For more details see the tutorial on registering custom domain: 


Connect to the Cluster

Merge Kubeconfig

az aks get-credentials --resource-group $resourcegroupName --name $clusterName --admin


Install Cert Manager

# Install the CustomResourceDefinition resources separately
# Note: --validate=false is required per
kubectl apply -f --validate=false

kubectl create namespace cert-manager
kubectl label namespace cert-manager
helm repo add jetstack
helm repo update
helm install cert-manager --namespace cert-manager --version v0.13.0 jetstack/cert-manager
kubectl apply -f yml/clusterissuer.yaml

# Test a sample application. The below command will deploy a Pod, Service and Ingress resource. Application Gateway will be configured with the associated rules.
sed -i "s//$domainName/g" yml/Test-App-Ingress.yaml
sed -i "s//$subDomain/g" yml/Test-App-Ingress.yaml
kubectl apply -f yml/Test-App-Ingress.yaml

# Clean up after successfully verifying AGIC
kubectl delete -f yml/Test-App-Ingress.yaml


Install KEDA runtime

helm repo add kedacore
helm repo update
kubectl create namespace keda
helm install keda kedacore/keda --namespace keda


Install CSI Provider for Azure KeyVault

helm repo add csi-secrets-store-provider-azure
helm repo update
kubectl create namespace csi
helm install csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --namespace csi


Assign managed identity

clientId=$(az aks show -n $clusterName -g $resourcegroupName --query identityProfile.kubeletidentity.clientId -o tsv)
scope=$(az group show -g $nodeRG --query id -o tsv)
az role assignment create --role "Managed Identity Operator" --assignee $clientId --scope $scope


Create Azure KeyVault for saving secrets and assign identity

az keyvault create --location $location --name $keyvaultName --resource-group $resourcegroupName
kvscope=$(az keyvault show -g $resourcegroupName -n $keyvaultName --query id -o tsv)
az identity create -g $resourcegroupName -n $identityName
idClientid=$(az identity show -n $identityName -g $resourcegroupName --query clientId -o tsv)
idPrincipalid=$(az identity show -n $identityName -g $resourcegroupName --query principalId -o tsv)
identityId=$(az identity show -n $identityName -g $resourcegroupName --query id -o tsv)
az role assignment create --role "Reader" --assignee $idPrincipalid --scope $kvscope

# Set permissions az keyvault set-policy -n $keyvaultName --secret-permissions get --spn $idClientid
# Add Pod Identity az aks pod-identity add --resource-group $resourcegroupName --cluster-name $clusterName --namespace default --name $identityName --identity-resource-id $identityId


Create MySQL managed service (basic sku) and add Kubernetes Load Balancer's public ip in it firewall rules

aksPublicIpName=$(az network lb show -n kubernetes -g $nodeRG --query "frontendIpConfigurations[0].name" -o tsv)
aksPublicIpAddress=$(az network public-ip show -n $aksPublicIpName -g $nodeRG --query ipAddress -o tsv)
az mysql server create --resource-group $resourcegroupName --name $mysqlSvr --location $location --admin-user $adminUser --admin-password $mysqlPwd --sku-name B_Gen5_2
az mysql server firewall-rule create --name allowip --resource-group $resourcegroupName --server-name $mysqlSvr --start-ip-address $aksPublicIpAddress --end-ip-address $aksPublicIpAddress

# Replace  with your Local Machine IP. You can use:
az mysql server firewall-rule create --name devbox --resource-group $resourcegroupName --server-name $mysqlSvr --start-ip-address <Dev station ip> --end-ip-address <Dev station ip>


Login to MySQL (you may need to add you ip to firewall rules as well). # Login to MySQL Client your your local dev box: sudo apt install mysql-client-core-8.0

mysql -h $ -u $adminUser@$mysqlSvr -p
show databases;


USE conexpapi;

   CostCenterId int(11)  NOT NULL,
   SubmitterEmail text NOT NULL,
   ApproverEmail text NOT NULL,
   CostCenterName text NOT NULL,
   PRIMARY KEY ( CostCenterId )

# Insert example records
INSERT INTO CostCenters (CostCenterId, SubmitterEmail,ApproverEmail,CostCenterName)  values (1, '', '','123E42');
INSERT INTO CostCenters (CostCenterId, SubmitterEmail,ApproverEmail,CostCenterName)  values (2, '', '','456C14');
INSERT INTO CostCenters (CostCenterId, SubmitterEmail,ApproverEmail,CostCenterName)  values (3, '', '','456C14');

# Verify Records
SELECT * FROM CostCenters;


Create Storage queue

az storage account create -n $storageAcc -g $resourcegroupName -l $location --sku Standard_LRS
# Do not change queue name of contosoexpenses. KedaFunction queue trigger relies on this queue name.
az storage queue create -n contosoexpenses --account-name $storageAcc


Add corresponding secrets to the create KeyVault

  1. MySQL Connection strings (choose ADO.NET) - both for API and Web
    1. mysqlconnapi
    2. mysqlconnweb
  2. Storage Connection strings
    1. storageconn
  3. Sendgrid Key
    1. sendgridapi
az keyvault secret set --vault-name $KeyVault --name mysqlconnapi --value '<replace>Connection strings for MySQL API connection</replace>'
az keyvault secret set --vault-name $keyvaultName --name mysqlconnweb --value '<replace>Connection strings for MySQL Web connection</replace>'
az keyvault secret set --vault-name $keyvaultName --name storageconn --value '<replace>Connection strings for Storage account</replace>'
# Make sure to register for free SendGrid Account and verify identity. Visit
az keyvault secret set --vault-name $keyvaultName --name sendgridapi --value '<replace>Sendgrid Key</replace>'
az keyvault secret set --vault-name $keyvaultName --name funcruntime --value 'dotnet'


Application Deployment

registryHost=$(az acr show -n $acrName --query loginServer -o tsv)

az acr login -n $acrName

cd source/Contoso.Expenses.API
docker build -t $registryHost/conexp/api:latest .
docker push $registryHost/conexp/api:latest

cd ..
docker build -t $registryHost/conexp/web:latest -f Contoso.Expenses.Web/Dockerfile .
docker push $registryHost/conexp/web:latest

docker build -t $registryHost/conexp/emaildispatcher:latest -f Contoso.Expenses.KedaFunctions/Dockerfile .
docker push $registryHost/conexp/emaildispatcher:latest

cd ..

# Update yamls files and change identity name, keyvault name, queue name and image used refer values between <> in all files
# Create CSI Provider Class
# Use gsed for MacOS
sed -i "s/<Tenant ID>/$tenantId/g" yml/csi-sync.yaml
sed -i "s/<Cluster RG Name>/$resourcegroupName/g" yml/csi-sync.yaml
sed -i "s/<Subscription ID>/$subscriptionId/g" yml/csi-sync.yaml
sed -i "s/<Keyvault Name>/$keyvaultName/g" yml/csi-sync.yaml
kubectl apply -f yml/csi-sync.yaml

# Create API
sed -i "s/<identity name created>/$identityName/g" yml/backend.yaml
sed -i "s/<Backend image built>/$registryHost\/conexp\/api:latest/g" yml/backend.yaml
sed -i "s/<Keyvault Name>/$keyvaultName/g" yml/backend.yaml
kubectl apply -f yml/backend.yaml

# Create frontend
sed -i "s/<identity name created>/$identityName/g" yml/frontend.yaml
sed -i "s/<frontend image built>/$registryHost\/conexp\/web:latest/g" yml/frontend.yaml
sed -i "s/<Keyvault Name>/$keyvaultName/g" yml/frontend.yaml
kubectl apply -f yml/frontend.yaml

# Create ingress resource
sed -i "s/<custom domain name>/$domainName/g" yml/ingress.yaml
sed -i "s//$subDomain/g" yml/ingress.yaml
kubectl apply -f yml/ingress.yaml

# Create KEDA function
sed -i "s/<identity name created>/$identityName/g" yml/function.yaml
sed -i "s/<function image built>/$registryHost\/conexp\/emaildispatcher:latest/g" yml/function.yaml
kubectl apply -f yml/function.yaml

Once the ingress controller updates with new frontend deployed it may take a min for Application gateway to update.

Browse the application URL:


Next Steps

  • Implement Service Mesh (like OSM) for securing service to service communications
  • Enabling managed identity to access MySQL and Storage services, thus removing Key Vault references
  • Enabling Github Actions for CI/CD pipelines


Version history
Last update:
‎Mar 05 2021 10:28 AM
Updated by: