I have been working with a lot of customers recently who are using containers (specially AKS) to host their .NET based web apps. I wanted to take this opportunity to get the steps and troubleshooting techniques all at once place for all of us who want to use AKS to run .NET based web apps.
Let's start with deploying the application in AKS using Linux Container.
Step 1- Login using CLI:
az login
Step 2- Create a resource group where you will host the container:
az group create --name AKSTest --location eastus
Step 3- Create an AKS cluster:
az aks create --resource-group AKSTest --name explore-docker-aks98 --node-vm-size Standard_B2s --node-count 1 --generate-ssh-keys --location eastus
When you create an AKS cluster, an additional resource group gets created which has all the services like load-balancer, Virtual network, virtual machine scale sets etc which are required to manage the cluster.
Now, we will be hosting our Asp.net core application into this AKS cluster.
I have used the Github Code sample: https://github.com/dotnet-architecture/explore-docker
Here, we are using Docker to store the images and containers. Docker provides an ability to package and run the application in an isolated environment. Due to this isolation, we can run multiple containers in a given host.
Before running the application, you can go to your CMD and run the command docker images and see there will not be any image present in your docker at this moment.
Let’s run the application in debug mode.
You will get the web page like below:
In the output window, you would be able to see the container and images getting created as per the docker-compose.yml file.
Now, if you do docker images again, you will see two images for web app and web api gets created with dev tag.
Step 4- Create an ACR(Azure Container Registry):
az acr create --name exploredocker98 --resource-group AKSTest --sku basic --admin-enabled
ACR is a private registry to store all your containers and images. Since it is very close to the infrastructure, downloading the images from container registry becomes faster.
Run the application in release mode so that it’s ready for the deployment:
If you run docker images now, you will see two more images got created with the latest tag.
Step 5 - Tag the latest images with login server name:
docker tag webapp:latest exploredocker98.azurecr.io/webapp:v1
docker tag webapi:latest exploredocker98.azurecr.io/webapi:v1
You can get the login server name from the ACR you created in step 4.
If you run docker images now, you will see the new image getting created with the v1 tag which you gave in the previous command.
Step 6-Login to your container registry:
az acr login --name exploredocker98
Step 7- Push both image with v1 tag into the Container registry:
docker push exploredocker98.azurecr.io/webapp:v1
docker push exploredocker98.azurecr.io/webapi:v1
Step 8- Create a Pull Permission in the container registry from Azure Portal.
Now, your images are pushed to the container registry, in the next steps, you would need to create a pull permission in ACR.
As at the end, the objective is to deploy the application in AKS. After pushing the images in ACR, we have to add permission to ACR so that the images can be pulled from here into the AKS.
Navigate to: container registry->Identity->User assigned->select your AKS resource Group->Add
Once the identity is Added, create a ACR pull Permission.
Select identity->Azure Role Assignments->Add role assignment->AcrPull
Step 9- Now, get the credentials from AKS so that you can get into the AKS and communicate with the Kubernetes Cluster:
az aks get-credentials --resource-group AKSTest --name explore-docker-aks98
Step 10- Deploy the YAML File into AKS. Refer article: Build ASP.NET Core applications deployed as Linux containers into AKS/Kubernetes clusters - .NET | M... to get the YAML file, save it as deploy-webapp.yml and deploy-webapi.yml in your local machine.
Navigate to the location where you have saved your Yaml file and run the below commands to deploy the YAML into AKS.
kubectl apply -f deploy-webapp.yml
kubectl apply -f deploy-webapi.yml
You have now created a deployment with the yml file. Based on replicas, it is going to create pods within the AKS Cluster.
Step 11 - To get the cluster Information, run:
kubectl get all
Here you get the external IP address of the webApp/ load-balancer. It takes the traffic from the outside world and pass it down to the containers/Pods within AKS.
Your application is now Successfully deployed to the AKS cluster.
Now, Let’s Get into some troubleshooting:
If you try accessing your application with the external Ip address of the Web-App, you will see the below error.
Let's understand why this error is coming up. To get more information about the error message, we can utilize Live log feature of AKS in Azure Portal.
Navigate to your AKS->Deployment->Click on webapp->select your pod and refresh the error page.
You will see, it’s making a get request to the below url and encounters a system.net exception.
Let's get back to the application and have a look at the appsettings.json file.
In the app setting, you will see that your web app is making an outbound call to the web Api with the backend URL as http://host.docker.internal and once we host it in Kubernetes, it is unable to recognize the URL.
To fix this, Let’s replace the backend URL with private Ip address of the web Api as we know that our web App is making an outbound call to the Web Api.
You can get the Private Ip of Web Api by running the command (kubectl get all) after publishing the application in AKS.
Now, deploy your web app in the ACR again through visual studio which is going to push the latest images with the changes we have made in our web application.
Once you have published the application in AKS, you would need to Deploy the YAML file of webapp into the AKS again.
kubectl apply -f deploy-webapp.yml
Restart your webapp and webapi pods.
kubectl rollout restart deployment.apps/webapp
kubectl rollout restart deployment.apps/webapi
Now, if you run the application with the public Ip address of the webapp, you will get the desired page.
Now let's try to collect some troubleshooting data like Dotnet Dump which would be helpful in examining performance issues with the application which is now hosted on AKS.
Collecting Dotnet-Dumps on AKS Cluster.
I have the same Application deployed in an AKS cluster with some additional code changes and now the app takes around 8 seconds to load.
We will now collect the dumps for this application to troubleshoot Slowness issue.
Open another command prompt instance.
Step 1 - Get the AKS Credential:
az aks get-credentials --resource-group NewGroup98 --name explore-docker-aks98
Step 2- Get the details of the POD and check for the running Pods:
Kubectl get all
You can also get this information through azure portal by navigating to AKS cluster->Deployment->webapp->pod
Step 3- Get inside the Bash with the command:
kubectl exec -it pod/webapp-7d9b577bf6-hncpz -- /bin/sh
Step 4- List all the packages:
apt-get update
Step 5-Install the package:
apt-get install wget
Step 6-Install the dotnet dump:
wget -o dotnet-dump https://aka.ms/dotnet-dump/linux-x64
Step 7-Provide permission to the dotnet dump:
chmod 777 ./dotnet-dump
Step 8- Get the PID of the running dotnet process:
./dotnet-dump ps
The output looks like this.
Now, Browse the site which will reproduce the slowness issue.
Step 9- collect the dump on the above Pid which is 1 in our case:
./dotnet-dump collect -p 1
This will collect a full user dump.
Step 10 - Zip the collected dumps:
gzip core_20230922_142506
Step 11- Exit from the bash with Exit 1.
Step 12- Save the zipped file into the local machine:
kubectl cp webapp-7d9b577bf6-hncpz:/app/core_20230922_142506.gz ./dotnetdump/mydump1.gz
Dumps will be present in the above-mentioned folder path.
The same way, you can also collect dotnet trace if required.
References:
dotnet-dump diagnostic tool - .NET CLI - .NET | Microsoft Learn
dotnet-trace diagnostic tool - .NET CLI - .NET | Microsoft Learn