Hosting ASP.Net Core application in AKS in Linux Container with Troubleshooting and Data Collection
Published Sep 29 2023 01:45 PM 4,881 Views
Microsoft

I have been working with a lot of customers recently who are using containers (specially AKS) to host their .NET based web apps. I wanted to take this opportunity to get the steps and troubleshooting techniques all at once place for all of us who want to use AKS to run .NET based web apps.

 

Let's start with deploying the application in AKS using Linux Container.

 

Step 1- Login using CLI:

 

 

 

 

az login

 

 

 

 

Step 2- Create a resource group where you will host the container: 

 

 

 

 

az group create --name AKSTest --location eastus

 

 

 

 

Step 3- Create an AKS cluster:

 

 

 

 

az aks create --resource-group  AKSTest --name explore-docker-aks98 --node-vm-size Standard_B2s --node-count 1 --generate-ssh-keys --location eastus

 

 

 

 

When you create an AKS cluster, an additional resource group gets created which has all the services like load-balancer, Virtual network, virtual machine scale sets etc which are required to manage the cluster.

jhaujala43_0-1695996541327.png

 

Now, we will be hosting our Asp.net core application into this AKS cluster.

I have used the Github Code sample: https://github.com/dotnet-architecture/explore-docker

 

Here, we are using Docker to store the images and containers. Docker provides an ability to package and run the application in an isolated environment. Due to this isolation, we can run multiple containers in a given host.

jhaujala43_1-1695996591458.png

Before running the application, you can go to your CMD and run the command docker images and see there will not be any image present in your docker at this moment.

jhaujala43_2-1695996879612.png

 

Let’s run the application in debug mode.

jhaujala43_3-1695996879662.png

 

You will get the web page like below:

jhaujala43_33-1695999208347.png

 

In the output window, you would be able to see the container and images getting created as per the docker-compose.yml file.

jhaujala43_7-1695996927526.png

 

Now, if you do docker images again, you will see two images for web app and web api gets created with dev tag.

jhaujala43_8-1695996939829.png

 

Step 4- Create an ACR(Azure Container Registry): 

 

 

 

 

az acr create --name exploredocker98 --resource-group AKSTest --sku basic --admin-enabled

 

 

 

ACR is a private registry to store all your containers and images. Since it is very close to the infrastructure, downloading the images from container registry becomes faster.

 

Run the application in release mode so that it’s ready for the deployment:

jhaujala43_9-1695996986001.png

 

If you run docker images now, you will see two more images got created with the latest tag.

jhaujala43_10-1695996996816.png

 

Step 5 - Tag the latest images with login server name: 

 

 

 

 

docker tag webapp:latest exploredocker98.azurecr.io/webapp:v1
docker tag webapi:latest exploredocker98.azurecr.io/webapi:v1

 

 

 

 

You can get the login server name from the ACR you created in step 4. 

jhaujala43_11-1695997028721.png

 

 

If you run docker images now, you will see the new image getting created with the v1 tag which you gave in the previous command.

jhaujala43_12-1695997028731.png

 

Step 6-Login to your container registry: 

 

 

 

 

az acr login --name exploredocker98

 

 

 

 

Step 7- Push both image with v1 tag into the Container registry:

 

 

 

 

docker push exploredocker98.azurecr.io/webapp:v1
docker push exploredocker98.azurecr.io/webapi:v1

 

 

 

 

jhaujala43_13-1695997028739.png

jhaujala43_14-1695997028750.png

 

Step 8- Create a Pull Permission in the container registry from Azure Portal.

Now, your images are pushed to the container registry, in the next steps, you would need to create a pull permission in ACR.

As at the end, the objective is to deploy the application in AKS. After pushing the images in ACR, we have to add permission to ACR so that the images can be pulled from here into the AKS.

Navigate to: container registry->Identity->User assigned->select your AKS resource Group->Add

jhaujala43_15-1695997028784.png

 

 

Once the identity is Added, create a ACR pull Permission.

Select identity->Azure Role Assignments->Add role assignment->AcrPull

jhaujala43_16-1695997050866.png

Step 9- Now, get the credentials from AKS so that you can get into the AKS and communicate with the Kubernetes Cluster: 

 

 

 

 

az aks get-credentials --resource-group AKSTest --name explore-docker-aks98

 

 

 

 

Step 10- Deploy the YAML File into AKS. Refer article: Build ASP.NET Core applications deployed as Linux containers into AKS/Kubernetes clusters - .NET | M... to get the YAML file, save it as deploy-webapp.yml and deploy-webapi.yml in your local machine.

Navigate to the location where you have saved your Yaml file and run the below commands to deploy the YAML into AKS.

 

 

 

 

kubectl apply -f deploy-webapp.yml
kubectl apply -f deploy-webapi.yml

 

 

 

 

jhaujala43_17-1695997066527.png

You have now created a deployment with the yml file. Based on replicas, it is going to create pods within the AKS Cluster.

Step 11 - To get the cluster Information, run: 

 

 

 

 

kubectl get all

 

 

 

 

jhaujala43_18-1695997066543.png

Here you get the external IP address of the webApp/ load-balancer. It takes the traffic from the outside world and pass it down to the containers/Pods within AKS.

 

Your application is now Successfully deployed to the AKS cluster.

 

Now, Let’s Get into some troubleshooting:

 

If you try accessing your application with the external Ip address of the Web-App, you will see the below error.

jhaujala43_19-1695997066572.png

 

Let's understand why this error is coming up. To get more information about the error message, we can utilize Live log feature of AKS in Azure Portal. 

 

Navigate to your AKS->Deployment->Click on webapp->select your pod and refresh the error page.

jhaujala43_20-1695997066591.png

 

You will see, it’s making a get request to the below url and encounters a system.net exception.

jhaujala43_21-1695997066596.png

 

jhaujala43_22-1695997066609.png

 

Let's get back to the application and have a look at the appsettings.json file. 

In the app setting, you will see that your web app is making an outbound call to the web Api with the backend URL as http://host.docker.internal and once we host it in Kubernetes, it is unable to recognize the URL. 

jhaujala43_23-1695997089930.png

 

To fix this, Let’s replace the backend URL with private Ip address of the web Api as we know that our web App is making an outbound call to the Web Api.

 

You can get the Private Ip of Web Api by running the command (kubectl get all) after publishing the application in AKS.

jhaujala43_24-1695997128287.png

 

Now, deploy your web app in the ACR again through visual studio which is going to push the latest images with the changes we have made in our web application.

jhaujala43_25-1695997128300.png

 

jhaujala43_26-1695997128306.png

 

jhaujala43_27-1695997128312.png

 

Once you have published the application in AKS, you would need to Deploy the YAML file of webapp into the AKS again.

 

 

 

 

kubectl apply -f deploy-webapp.yml

 

 

 

 

Restart your webapp and webapi pods.

 

 

 

 

kubectl rollout restart deployment.apps/webapp
kubectl rollout restart deployment.apps/webapi

 

 

 

 

Now, if you run the application with the public Ip address of the webapp, you will get the desired page.

jhaujala43_28-1695997145378.png

 

 

Now let's try to collect some troubleshooting data like Dotnet Dump which would be helpful in examining performance issues with the application which is now hosted on AKS.

 

Collecting Dotnet-Dumps on AKS Cluster.

I have the same Application deployed in an AKS cluster with some additional code changes and now the app takes around 8 seconds to load.

jhaujala43_29-1695997166153.png

 

We will now collect the dumps for this application to troubleshoot Slowness issue.

Open another command prompt instance.

Step 1 - Get the AKS Credential: 

 

 

 

 

az aks get-credentials --resource-group NewGroup98 --name explore-docker-aks98

 

 

 

 

Step 2- Get the details of the POD and check for the running Pods: 

 

 

 

 

Kubectl get all 

 

 

 

 

You can also get this information through azure portal by navigating to AKS cluster->Deployment->webapp->pod

jhaujala43_30-1695997166164.png

 

 

Step 3- Get inside the Bash with the command: 

 

 

 

 

kubectl exec -it pod/webapp-7d9b577bf6-hncpz -- /bin/sh

 

 

 

 

Step 4- List all the packages: 

 

 

 

 

apt-get update 

 

 

 

 

Step 5-Install the package: 

 

 

 

 

apt-get install wget

 

 

 

 

Step 6-Install the dotnet dump:

 

 

 

 

wget -o dotnet-dump https://aka.ms/dotnet-dump/linux-x64

 

 

 

 

Step 7-Provide permission to the dotnet dump: 

 

 

 

 

chmod 777 ./dotnet-dump

 

 

 

 

Step 8- Get the PID of the running dotnet process: 

 

 

 

 

./dotnet-dump ps

 

 

 

 

The output looks like this. 

 

jhaujala43_31-1695997166165.png

Now, Browse the site which will reproduce the slowness issue. 

Step 9- collect the dump on the above Pid which is 1 in our case: 

 

 

 

 

./dotnet-dump collect -p 1

 

 

 

 

This will collect a full user dump.

 

jhaujala43_32-1695997166166.png

 

Step 10 - Zip the collected dumps: 

 

 

 

 

gzip core_20230922_142506

 

 

 

 

Step 11- Exit from the bash with Exit 1.

Step 12- Save the zipped file into the local machine: 

 

 

 

 

kubectl cp webapp-7d9b577bf6-hncpz:/app/core_20230922_142506.gz ./dotnetdump/mydump1.gz

 

 

 

 

Dumps will be present in the above-mentioned folder path.

 

The same way, you can also collect dotnet trace if required. 

 

References:

Build ASP.NET Core applications deployed as Linux containers into AKS/Kubernetes clusters - .NET | M...

dotnet-dump diagnostic tool - .NET CLI - .NET | Microsoft Learn

dotnet-trace diagnostic tool - .NET CLI - .NET | Microsoft Learn

1 Comment
Co-Authors
Version history
Last update:
‎Sep 29 2023 01:47 PM
Updated by: