This post is co-authored by Abhishek Saharn (Software Engineer, Microsoft)
NLog is a popular logging framework used by developers to log messages from .NET applications. One of the challenges of logging is how to store and analyze the large volume of data generated by the logs. Azure Data Explorer (ADX) is a fast and highly scalable data exploration service that can be used to store and analyze large volumes of data. In this blog post, we will show you how to use the Azure Data Explorer custom target connector with NLog to store your log data in an ADX cluster.
But first, Why logging is important?
Logging is a crucial component in software development, providing a record of events that occur during the operation of an application. This record can be used for a variety of purposes, including debugging, auditing, performance monitoring, security, troubleshooting, analytics, historical analysis, and continuous improvement.
When developers encounter errors or bugs in their code, logging can help identify the root cause by providing a detailed history of the events leading up to the issue. For compliance and auditing purposes, logging can track user activity and system events, providing an essential record of all transactions and changes. Additionally, provides developers with the opportunity for continuous improvement by identifying areas for optimization and refinement.
How Azure Data Explorer helps here?
One of the biggest challenges with logging is the sheer volume of data that is generated. For example, a single server in a typical enterprise environment can generate more than 10 GB of log data per day. Multiply that by the number of servers and applications in an organization, and you're looking at a truly staggering amount of data to manage and analyze.
To address this challenge, Azure Data Explorer(ADX) comes to help. ADX is a fast and highly scalable data analytics platform that is specifically designed for ingesting, analyzing, and visualizing large volumes of data, including log data.
By leveraging Azure Data Explorer's powerful query capabilities and built-in data visualization tools, organizations can quickly and easily analyze log data to gain insights into application performance, troubleshoot issues, and identify trends. It is feature-rich in time series analytics, log analytics, full-text search, advanced analytics (e.g., pattern recognition, forecasting, anomaly detection), visualization, scheduling, orchestration, automation, and many more native capabilities.
The Azure Data Explorer connector is a custom target for NLog that allows you to send your log messages to an ADX cluster. The plugin is built on top of the Azure-Kusto-Data library and provides a simple and efficient way to sink your logs to ADX.
Now we will discuss how to get started with configuring NLog and Azure Data Explorer, by using the NLog-kusto sample application provided in the ADX-Nlog repository. You will learn how to write log messages and view them in Azure Data Explorer(ADX) cluster.
.create table ADXNLogSample (Timestamp:datetime, Level:string, Message:string, FormattedMessage:dynamic, Exception:string, Properties:dynamic)
git clone https://github.com/Azure/azure-kusto-nlog-sink.git
Step 1: Install the ADX Target for NLog
The ADX Target for NLog is available as a NuGet package. To install it, open the Package Manager Console and enter the following command:
Step 1.1: Configure the ADX Target for NLog
Alternatively, If you do not want to use environment variables, you can configure the NLog.config file. Open the configuration file and replace the existing target configuration one with the following:
<target name="adxtarget" xsi:type="ADXTarget" IngestionEndpointUri="<ADX connection string>" Database="<ADX database name>" TableName="<ADX table name>" ApplicationClientId="<AAD App clientId>" ApplicationKey="<AAD App key>" Authority="<AAD tenant id>" />
Step 2: Build the application and run it
ADXNLogSample | take 10
In this blog post, we have seen how NLog and Azure Data Explorer can work together to provide a powerful and scalable solution for logging and analyzing .NET applications. By using the Azure Data Explorer custom target connector with NLog, you can easily send your log messages to an ADX cluster and take advantage of its fast and flexible query capabilities and data visualization tools. You can also leverage ADX’s advanced analytics features such as time series analysis, anomaly detection, forecasting, and pattern recognition to gain deeper insights into your application performance and behavior.
If you are interested in trying out this solution, you can follow the steps mentioned this post to set up your own ADX cluster and database, create a table to store your logs, install the ADX target for NLog, and run the sample application. You can also explore the ADX-Nlog repository on GitHub for more details and examples.
We hope you find this solution useful, and we welcome your feedback and suggestions.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.