Collecting Debug Information from Containerized Applications

[Becky]

 

Collecting Debug Information from Containerized Applications

1. General Info

Howdy everyone! It's your favorite Debug Engineer, Will Aftring back again, this time talking about containers. This blog post will assume that you have a fundamental understanding of Windows containers. If that isn't the case, then then I highly recommend reading Get started: Run your first Windows container.

Many developers and IT Admins are in the midst of migrating long standing applications into containers to take advantage of the myriad of benefits made available with containerization.

NOTE: Not all applications are able to equally take advantage of the benefits of containerization. It is another tool for the toolbox to be used at your discretion.

But moving an existing application into a container can be a bit tricky. With this blog post I hope to help make that process a little bit easier for you.

2. Containerization steps:

2.1.1. Identify your dependencies

One of the benefits of containers is the limited deployment size that can be leveraged by Windows containers. A Windows Server Nano container can be as small as a few hundred megabytes!

However, as a part of trimming down the image size for the base container image, many things have been removed.

For example, in a Windows Server Nano container, much of the .NET framework has been removed. Meaning if your application is dependent on the .NET framework, then using a nano container image isn't viable.

Now application dependencies can be both within the Operating System (OS) or from the application itself. If you have segmented your application into executables and Dynamic Link Libraries (DLLs) then the container image needs to have those files available within the container image.

2.2.2. Find your configurations

In the Windows world, many developers manage the storage of configuration for their application via the Windows Registry. As a part of the containerization of the application, it will no longer share a registry hive with the container host. If you want your applications registry keys in place then you must add them either in the deployment of the container image or in the runtime of the application itself.

This also extends to local files and environment variables.

2.3.3. Figuring out what you need to communicate with over the network

As Ned Pyle so elegantly put it in Accelerating Your IT Career.

It's hard to find an IT system talking only to itself. Notepad, maybe (until you save a file to a network share).

It is almost guaranteed that your application will be leveraging the network at least to some degree. If it is a device on another container host then your problems are relatively simple:

• How are you going to resolve the name of the host running the relevant network endpoint?
• How is that outbound communication going to make its way back to the container?
  • If you are using a NAT or Transparent network driver, then the behavior is similar to that of a VM
  • If you are trying to use an L2Bridge then you need to make sure that the device you are communicating with is within the same LAN

If you are intending on containerizing the endpoint workload as well then things get a bit more complex. Intra-container communication is outside the scope of this blog post but if you are reading my other series on Introduction to Network Trace Analysis, then you should have some good fundamentals in place to understand this communication.

3. I've moved my application into a container and it isn't working. Help me!

First, don't panic. If application configuration were easy, it would be called baseball.

There are a few things we can do to get started with understanding the issue.

3.1.1. Is there anything logged to the console?

Anything that is written to the standard output (stdout) stream or standard error (stderr) is accessible from the container host.

Depending on your container runtime interface (CRI) you can use one of the following commands to read this output.

• With docker
  • docker logs <container id>
• With Containerd
  • ctr task attach <container id>

 

3.2.2. Are there any relevant log files being written to the disk?

Collecting log files is often an easier option. As a part of the container configuration, you are able to create a mapping between directories on the container host and directories within the container.

For the sake of being CRI independent you can use the Windows tool hcsdiag to make the location.

For example, if you are writing your log files to C:\Logs in the container, you can map that location to C:\<ServiceName>\Logs on the container host and read them in real-time.

• hcsdiag share <container id> C:\Demo\Logs C:\Logs

Then read the log files from C:\Demo\Logs on the container host.

3.3.3. Leveraging External Tools

In (most) container images, many of your typical Windows troubleshooting tools are still available. And through hcsdiag, you can invoke commands directly in the container.

• You can use wevtutil to export event logs.
• You can use klist to display kerberos tickets
• You can use Windows Error Reporting (WER) to collect process crash dumps

And through the mapping of a container directory to a container host directory, you can easily access the relevant debugging data.

Combining a mapped directory with the ability to run commands from within the container you can run all your favorite debugging tools from within the container.

For example, if you wanted to use procdump as the post-mortem debugger, you can do so:

1. Identify the container id
   1. hcsdiag list
2. Ensure that you have shared mapping between the container and the container host
   1. hcsdiag share <container id> C:\Dumps C:\Dumps
   2. Please note, this mapping will remain until a restart of the container host
3. Setup procdump as the post-mortem debugger
   1. hcsdiag exec <container id> "C:\Dumps\procdump.exe" "-accepteula" "-i" "-ma"
4. Wait for the crash to occur and it will be written to C:\Dumps on the container host

And using the basic format, substitute in your tool of choice for what you would like to do.

3.4.4. Memory Dumps

We all know our applications don't live in isolation. They need to read and write files and rely upon the Windows kernel. In some cases when debugging something like this you may want to collect a memory dump of the machine.

The good news is if you are running your containers in process isolation mode (the default) all you need to do is collect a memory dump of the container host.

Since the container host and the container share the same Windows Kernel, if you collect a complete memory dump of the container host, then the processes running within the container will be captured as well.

Within the memory dump the items in the container will present themselves as if they are running in a different session but you can proceed debugging to your hearts content.

4. Summarizing

See? Not so bad! Once you learn a few tips and tricks it isn't so different than debugging and application running anywhere else in Windows. I hope you find this post helpful in expediting your lift and shift projects.

Happy debugging!