Alright, it’s time to bring together everything we learned so far in this series. Previously, we looked at the simplest way to collect logs, how to use the Log Monitor tool, and how to make things simpler with Windows Admin Center. In today’s blog post, we will look at Azure Kubernetes Service and how everything we’ve learned so far can be applied in practice for production environments.
However, I’m going to spice it up a notch, ok?
Windows containers + gMSA + Log Monitor + AKS
On a blog post prior to this series, I shared a script to create a testing environment for Group Managed Service Accounts (gMSA). The script builds an Azure vNet, an Azure Kubernetes Service (AKS) cluster, and a Domain Controller – all of these necessary to validate gMSA on AKS. The goal is to try applications that are Active Directory dependent. The module that deploys it even has a sample app for you to try it. So, here’s what I did:
- I created a new image using the sample app from the gMSA on AKS PowerShell module, but with Log Monitor attached to it.
- I published that image to Azure Container Registry and enabled the AKS cluster to access that registry.
- Then I deployed the app into the same AKS cluster and looked at the AKS logs – Voilá! The logs are now being streamed into AKS – which doesn’t happen by default with Windows containers.
The above exercise was a great learning experience for me and if you plan to troubleshoot applications on AKS at all, you might benefit from this. So, let’s take a look at the details behind this.
Setting up your test environment
With the intention of having all of this reproducible, I have shared all the artifacts needed on GitHub so you can deploy everything with minimal changes for your environment.
First, the gMSA environment can be recreated with the gMSA on AKS Trial repo. This repo has all the steps needed to deploy the environment I mentioned above. You will also need to create a new Azure Container Registry and provide access to it for your AKS cluster.
Next, you’ll need the assets on the Log Monitor gMSA repo. This is actually part of a larger repo on which I’m sharing some experimentation with the Log Monitor tool. There, you’ll find the docker file, YAML, and Log Monitor Configuration files.
Build a container image for gMSA with Log Monitor
Probably the step in this process where you’ll spend the most time. For your own application, you should have a docker file that is used to build the container image containing the application you want to deploy on AKS. Regardless of if you migrated the app from a VM to container, or if you built the application for containers directly, the container image was built using a docker file. In my case, here is the docker file I used:
FROM mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019
WORKDIR /LogMonitor
COPY LogMonitorConfig.json .
RUN powershell.exe -command wget -uri https://github.com/microsoft/windows-container-tools/releases/download/v1.1/LogMonitor.exe -outfile LogMonitor.exe
# Add required Windows features, since they are not installed by default.
RUN powershell -command Install-WindowsFeature "Web-Windows-Auth", "Web-Asp-Net45"
# Create simple ASP.Net page.
RUN powershell -command New-Item -Force -ItemType Directory -Path 'C:\inetpub\wwwroot\app'
COPY default.aspx C:\\inetpub\\wwwroot\\app
COPY Run.ps1 /LogMonitor/
RUN PowerShell -ExecutionPolicy Unrestricted -command /LogMonitor/run.ps1
# Change the startup type of the IIS service from Automatic to Manual
RUN sc config w3svc start=demand
# Enable ETW logging for Default Web Site on IIS
RUN c:\windows\system32\inetsrv\appcmd.exe set config -section:system.applicationHost/sites /"[name='Default Web Site'].logFile.logTargetW3C:"File,ETW"" /commit:apphost
EXPOSE 80
# Start "C:\LogMonitor\LogMonitor.exe C:\ServiceMonitor.exe w3svc"
ENTRYPOINT ["C:\\LogMonitor\\LogMonitor.exe", "C:\\ServiceMonitor.exe", "w3svc"]The file above builds on top of the IIS image released by Microsoft. It copies the LogMonitorConfig.json file, downloads the LogMonitor.exe binary, installs the additional IIS features (Windows Auth, and ASP. .Net), creates a simple website for IIS to use (this is my application), and then changes some IIS behavior for gMSA to properly work. Finally, the bottom portion of the file configures IIS to be started by the Log Monitor tool, and then sets the tool to be the entry point for the container.
I built this container using the following command:
Docker build -t gmsalm:v1 .Once the container is built, I then tagged it and pushed to my ACR registry:
docker tag gmsalm:v1 <my acr registry>/gmsalm:v1
docker push <my acr registry>/gmsalm:v1
Deploy app to AKS
With the application properly built and the container image stored on my ACR registry, I can now deploy it to my AKS cluster. For that, we’ll need a YAML file with the description to Kubernetes on how to deploy the app and its dependencies:
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: gmsalm-demo
name: gmsalm-demo
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: gmsalm-demo
template:
metadata:
labels:
app: gmsalm-demo
spec:
securityContext:
windowsOptions:
gmsaCredentialSpecName: credspec
containers:
- name: iis
image: <your image>
imagePullPolicy: IfNotPresent
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: agentpool
operator: In
values:
- wspool
---
apiVersion: v1
kind: Service
metadata:
labels:
app: gmsalm-demo
name: gmsalm-demo
namespace: default
spec:
ports:
- port: 80
targetPort: 80
selector:
app: gmsalm-demo
type: LoadBalancerIf you’re not familiar with a YAML file for deploying apps to Kubernetes, the above can be a bit intimidating. It’s actually very simple and easy to understand. You’ll notice there are two sections to the file, separated by “---“. In this case, we have a deployment (which deploys the containers) and a service, which is used in our case to deploy the networking load-balancer in front of our containers. There are then the required configuration for each of them. For the deployment (or container), we have some configuration such as the security context for the app – which in our case is to use the gMSA deployment. We also have the container image to be used, which is the same as you tagged on the previous step. Finally, we have the information that this should be deployed on our previously created Windows node pool called “wspool”. You might need to change that in your environment. At last, you have the configuration of the load balancer, which is set up to map the port 80 for the containers. All of these in the context of the gmsalm-demo application so it’s easier to understand when querying the AKS environment.
To apply this configuration, you can use the following command:
kubectl apply SampleApp.yamlAfter applying it, you can query the environment to check if the pods are running:
kubectl get pods -o wideWait until the pods for the application shows as “Running”. You also need to check the service:
kubectl get serviceThis will return the IP address of the service created for the application. With this IP address you can open a browser and try to access the page. You’ll notice that the application was configured to ask for credentials – which is basically Windows authentication against your Active Directory happening in the background.
This is when I try to open the application without providing the proper credentials:
When you log into the app, this is the result:
Finally, let’s see some logs!
Remember, by default, Windows containers don’t show their logs on AKS. This happens because (as explained in my very first blog) Windows don’t stream logs on STDOUT. Since AKS uses the native Kubernetes mechanism, which in turn looks for STDOUT logs, there’s nothing to show. However, with the steps above, we can clearly see the IIS logs – there are two ways to get them:
You can use kubectl:
kubectl logs <application pod name>This is the result from kubectl:
The image above shows the bottom of the output for kubectl logs for the pod on which my application is currently running.
However, you don’t necessarily need to use a PowerShell session to check logs. You can use the Azure Portal for that:
The above is a live view of events that are being collected from the pod/container. This view is much nicer as it breaks the lines into each item, which provides a better read for ETW and other events being collected by Log Monitor. It also allows you to integrate this application into Azure Monitor – but that’s a conversation for another blog post.
Conclusion
It’s the end of the road for this blog series. We looked at many ways to get insights into applications running on Windows containers, from the most basic to the most advanced with integration with the Azure Portal.
Today we proved that Windows containers provide a way to extract logs from applications in a way that you can more easily troubleshoot applications running on AKS. More than that, we looked at an application running integrated with Active Directory via gMSA.
I hope you liked today’s blog post, and more than that – that this was useful to you. You can find all the assets used in this blog post on my GitHub repo. Let us know what you think of this blog post and series in the comments below!