As containers have become increasingly popular, the demand for better tooling to help developers and their organizations containerize their applications has grown. At Microsoft, we are kicking off efforts to improve the tooling experience with containers, for developers to IT Pros, on-premises and the cloud, Windows or Linux, and beyond. This blog post offers a deep-dive look into one of the first areas we decided to tackle – Logging.
Logs are one of the central methods used for debugging software. They are used to diagnose incidents, understand KPIs such as performance, reliability, and availability, as well as observe user statistics and access patterns. However, while useful, logs tend to be scattered in different locations, are unstructured, and come in wildly different formats. While locating, gathering, and parsing log content is complex even on host processes, doing the same for containers has an additional complexity due to the additional isolation and virtualization layer. Fortunately, many tools have been developed to work with containers; however, many of these tools do not have sufficient log access for Windows Containers.
Docker logs and many container ecosystem monitoring solutions rely on STDOUT as the pipeline to receive logs and information from the processes running inside the container. This fits in-line the with the Linux application model, which directs all log output to STDOUT; however, Windows applications and services do not output to STDOUT, and instead deposit information in Windows services such as ETW, Perf Counters, custom log files, etc. As a result, much of the wealth of logging information produced by the Windows systems in container is inaccessible to most container ecosystem logging solutions.
The goal of our proof-of-concept tool is to address this accessibility gap between the logs produced within the Windows container and the container ecosystem tools that pull from STDOUT. This tool is designed as a container entry point executable with the following behavior:
Monitor container services
Observe container ETW events
Observe container Perf Counters
Observe container custom application log file
Tail observed container log sources to STDOUT
The POC is authored with an observer, publisher-subscriber design pattern. Loggers subscribe to providers, and each provider watches a data source and publish events to loggers upon discovering new data. This model provides touchpoints to scale the tool to additional data sources and output destinations as desired.
The POC fits into the larger ecosystem by providing existing tooling access to Windows logs via STDOUT. One such tooling tack is the popular Fluentd, Elasticsearch, Kibana stack (EFK stack). The tool also integrated with other container monitoring panes such as Azure Monitoring.
The tool integrates with the EFK stack as below:
A deployment of an application with the logging tool on AKS with Azure Monitoring would display as below:
This POC has some known design limitations:
The tool does not collect from all critical log sources (i.e. IIS Logs, Event Logs etc.)
To redirect to STDOUT, the tool must run as the foreground entry point process. This restricts developers from wrapping this entry point within their application as the tool would lose the handle to STDOUT.
This POC log tool is the first step Microsoft is taking to improving the tooling experience with Windows Containers. We are still in the process of iterating on this tool and extending our efforts into other tooling areas. We are seeking feedback on this concept and would appreciate responses to the survey below. At the end of the survey there is a field to leave contact information if you would like further follow-up from the team.