Getting started with Serilog and Azure Data Explorer

[tanmayapanda]

Logging is an essential component of any good software development process. Through logging, we can collect sufficient information to detect, do root cause analysis and fix breaking issues. Sometimes, we may also be able to utilize logs to understand the behavior of systems, users, or software components and obtain valuable insights to maintain business continuity. Logging however, may be difficult to implement without a proper logging framework, as a lot of time could be spent in deciding what to log, where to log and how to log. By employing a logging framework such as Serilog, you can save a lot of time. This can allow you to focus your efforts on improving your app and delivering value for your users, while the framework does all the heavy lifting from logging perspective.

What is Serilog?

Serilog is a diagnostic logging library for .NET applications. It provides a flexible and extensible platform for generating log messages used in .NET development, and it is particularly popular for building cloud-based applications that require real-time monitoring and logging capabilities.

It is easy to set up, has a clean API, and runs on all recent .NET platforms. While it's useful even in the simplest applications, Serilog's support for structured logging shines when instrumenting complex, distributed, and asynchronous applications and systems.

What is Azure Data Explorer?

ADX is a big data analytics platform that is highly optimized for all types of logs and telemetry data analytics. It provides low latency, high throughput ingestions with lightning-speed queries over extremely large volumes of data. It is feature-rich in time series analytics, log analytics, full-text search, advanced analytics (e.g., pattern recognition, forecasting, anomaly detection), visualization, scheduling, orchestration, automation, and many more native capabilities. 

In this blog, we will discuss how to get started with Serilog and Azure Data Explorer, including how to set up a Serilog sink for Azure Data Explorer and how to use Kusto Query Language (KQL) to query and analyze log data.

Let's get started with the installation and configuration of the Serilog-ADX sink.

Step 1: Installing the Serilog sink for Azure Data Explorer

The first step in ingesting log data into Azure Data Explorer is to install the Serilog sink for Azure Data Explorer. The sink provides a way to send log data from your .NET application to Azure Data Explorer in real-time. To install the sink, you can use the following NuGet package:

Install-Package Serilog.Sinks.AzureDataExplorer

Once the package is installed, you can configure the sink using the following code:

var log = new LoggerConfiguration()
                .MinimumLevel.Verbose()
                .WriteTo.AzureDataExplorerSink(new AzureDataExplorerSinkOptions
                {
                    IngestionEndpointUri = "<ADXIngestionURL>",
                    DatabaseName = "<databaseName>",
                    TableName = "<tableName>",
                    BufferBaseFileName = "<bufferFileName>",
                    ColumnsMapping = "<mappingName>" ,
                    }
                }.WithAadApplicationKey("<appId>", "<appKey>", "<tenant>"))
                .CreateLogger();

Replace the placeholders with the appropriate values for your Azure Data Explorer cluster. You can find these values in the Azure portal.

Setting up our Serilog ADX Demo Application

Serilog-ADX sink provides a demo/sample application that can be used to quickly get started with producing logs that can be ingested into the ADX cluster. 

Follow these steps to setup your demo application:

• Create Azure Data Explorer cluster and database from here.
• Create Azure Active Directory App registration and grant it permissions to the database from here. (don't forget to save the app key and the application ID for later.)
• Create a table in Azure Data Explorer which will be used to store log data. For example, we have created a table with the name "Serilogs".

.create table Serilogs (Timestamp: datetime, Level: string, Message: string, Exception: string, Properties: dynamic, Position: dynamic, Elapsed: int) 

•  Clone the Serilog-ADX sink git repo
• The following are the set of parameters which needs to be set as environment variables
  
  1. IngestionEndPointUri : Ingest URL of ADX cluster created.
  2. DatabaseName : The name of the database to which data should be ingested into.
  3. TableName : The name of the table created (in our case Serilog)
  4. AppId : Application Client ID required for authentication.
  5. AppKey : Application key required for authentication.
  6. Tenant : Tenant Id
  7. BufferBaseFileName : If we require durability of our logs(ie we don't want to lose our logs incase of any connection failure to ADX cluster), Ex: C:/Users/logs/Serilog
• The above-mentioned parameters need to be set as environment variables in the respective environments.
  For Windows, in PowerShell set the following parameters
  

  $env:ingestionURI="<ingestionURI>"
  $env:databaseName="<databaseName>"
  $env:tableName="<tableName>"
  $env:appId="<appId>"
  $env:appKey="<appKey>"
  $env:tenant="<tenant"

  For Linux-based environments, in the terminal set the following parameters

  export ingestionURI="<ingestionURI>"
  export databaseName="<databaseName>"
  export tableName="<tableName>"
  export appId="<appId>"
  export appKey="<appKey>"
  export tenant="<tenant"​

•  Open a Powershell window, navigate to Serilog-ADX sink base folder, and run the following command:

dotnet build src

• Navigate to src/Serilog.Sinks.AzureDataExplorer.Samples/ folder and run the following command

dotnet run

• The Sample/Program.cs contains predefined logs which will start getting ingested to ADX.
• The ingested log data can be verified by querying the created log table (Serilogs in our example) by using the following KQL command:
  

Serilogs | take 10

In conclusion, the ADX-Serilog sink is a simple and effective way to ingest log data into Azure Data Explorer. With this integration, you can get real-time insights into your application's performance and behavior and use this information to make data-driven decisions or debug issues when they occur.