Azure Sentinel: Creating Custom Connectors

Note!
This blog post is now part of the official Azure Sentinel documentation, and updated there. 
This blog post is obsolete.

 

Before you go the custom connector way

 

Want to include your custom connector in Azure Sentinel? contant us. 

 

If the Sentinel data connectors page does not include the source you need, you may still not need a custom connector. Review the following blog posts for additional sources that can be used with Sentinel without a custom connector:

• Azure Sentinel Agent: Collecting telemetry from on-prem and IaaS server
• Collecting logs from Microsoft Services and Applications
• Syslog, CEF, Logstash, and other 3rd party connectors grand list

 

If you still can't find your source in any of those, custom connectors are the solution.

 

Importantly, the options described below can be used to ingest vent data and import context and enrichment data such as threat intelligence, user or asset information.

 

Using the Log Analytics Agent

 

Custom file collection

 

The Log Analytics agent can collect events stored in files. This is the easiest way to collect events from any source that delivers events in files. See collecting Custom logs in Azure Monitor.

 

Fluentd

 

The most direct way to create a custom connector is to use the Log Analytics agent. The Log Analytics agent is based on Fluentd and can use any Fluentd input plugin bundled with the agent to collect events and then forward them to an Azure Sentinel workspace. You can find an example of how to do that in the documentation.

 

Note that the agent can do this alongside its other collection roles as described here. 

 

Using Fluentd or Fluent Bit

 

If the agent is not flexible enough, you may want to consider using Fluentd or Fluent Bit, its lighter sibling, directly:

• Fluent Bit and its Azure Sentinel output plug-in
• Fluentd and its Azure Sentinel output plug-in

 

Using Logstash

An alternative to using the Log Analytics agent and Fluentd plugins is using Logstash. It is architecturally similar, but if you know Logstash, this might be your best bet. To do so, use the Logstash output plugin for Sentinel, which enables you to use Azure Sentinel as the output for a Logstash pipeline. Now you can use all your GROK prowess and any Logstash input plugin to implement your connector.

 

See, for example, Collecting AWS CloudWatch using Logstash.

 

To scale Logstash, you may want to use a load-balanced Logstash VM scale set as described here, or a container cluster: read about doing this here.

 

 

 

Using Logic Apps

A serverless alternative eliminates the need to maintain VMs and uses Logic Apps to get events or context data to Sentinel. To do that, build a playbook with the following elements:

• Use one of these triggers to start the playbook:

1. Recurring task - schedule the connector, for example, for retrieving data from files, databases, or external APIs.
2. On-demand triggering - for manual upload and testing​​​​​​.
3. HTTP/S endpoint - if the source system can initiate the transfer and for streaming. 

• Read the data using one of the following connectors:​​​​​​

1. Using a REST API
2. Read SQL Server data.
3. Read a file
4. Note that those connectors support retrieving data on-premises.

• Prepare the information, for example, using the Parse JSON action.
• Write the data to Log Analytics using the Logic Apps connector for writing data to Log Analytics. 

 

There are many examples out there for doing so:

• A simple example in the API documentation.
• Create Custom Log Analytics logs with Logic Apps walks you through the steps and provides you with an excellent example of using parse JSON
• Getting MDATP alerts into Sentinel using either Paul Huijbregts or Tom Lilly's variants provides a real-world use case, and so will Sending Proofpoint TAP logs to Azure Sentinel.

 

Note that while convenient, this method may be costly for large volumes of data and should be used only for low volume sources or for context and enrichment data upload.

 

The PowerShell cmdlet

The Upload-AzMonitorLog PowerShell script enables you to use PowerShell to stream events or context information to Sentinel from the command line. For example, this command will upload a CSV file to Sentinel: 

Import-Csv .\testcsv.csv 
| .\Upload-AzMonitorLog.ps1 
-WorkspaceId '69f7ec3e-cae3-458d-b4ea-6975385-6e426'
-WorkspaceKey $WSKey
-LogTypeName 'MyNewCSV'
-AddComputerName 
-AdditionalDataTaggingName "MyAdditionalField" 
-AdditionalDataTaggingValue "Foo"

 

The script takes the following parameters:

• WorkspaceId - The Workspace ID of the workspace that would be used to store this data
• WorkspaceKey - The primary or secondary key of the workspace that would be used to store this data. It can be obtained from the Windows Server tab in the workspace Advanced Settings.
• LogTypeName - The name of the custom log table that would store these logs. This name will be automatically concatenated with "_CL."
• AddComputerName - If this switch is indicated, the script will add to every log record a field called Computer with the current computer name
• TaggedAzureResourceId - If exists, the script will associate all uploaded log records with the specified Azure resource. This will enable these log records for resource-context queries as well as adhere to resource-centric role-based access control.
• AdditionalDataTaggingName - If exists, the script will add to every log record an additional field with this name and the value that appears in AdditionalDataTaggingValue. This happens only if AdditionalDataTaggingValue is not empty.
• AdditionalDataTaggingValue - If exist, the script will add to every log record an additional field with this value. The field name would be as specified in AdditionalDataTaggingName. If AdditionalDataTaggingName is empty, the field name will be "DataTagging."

 

The Data Collection API

All the methods above use the Log Analytics Data Collector API  to stream events to Azure Sentinel. You can use the API directly to ingest any data to Sentinel. While it would require programming, it naturally offers the most flexibility.

 

To use the API, you can directly call the RESTful endpoint using C#, Python 2, Java, PowerShell, or any other language, or utilize the available client libraries. You can find an end to end example for a C# based connector here.

 

Azure Functions

Naturally, you need to run your API code somewhere. In traditional on-premises computing, this requires a server to run the connector and challenges it brings with it: monitoring, management, patching, etc.

 

Using Azure Functions to implement a connector using the API connector is especially valuable as it keeps the connector serverless. You can use any language, including PowerShell, to implement the function. To get started with implementing a custom connector using Azure Functions, refer to the C# example in the documentation or the real-world implementation:

• Getting Office 365 Management API data into Azure Sentinel
• Universal app logging for LISA App

 

Parsing

Therefore, the API and all the other options described above allow defining the fields that will be populated in Azure Sentinel. Use your connector parsing technique to extract relevant information from the source and populate it in designated fields, for example, grok in Logstash and Fluentd parsers in the Log Analytics agent.

 

However, Sentinel allows parsing at query time, which offers much more flexibility and simplifies the import process. Query time allows you to push data in at the original format and parse on demand when needed. Updating a parser will apply to already ingested data.

 

Query time parsing reduces the overhead of creating a custom connector as the data's exact structure does not have to be known beforehand. Nor do you need to identify the vital information to extract. Parsing can be implemented at any stage, even during an investigation, to extract a piece of information Adhoc and will apply to already ingested data.

 

JSON, XML, and CSV are especially convenient as Sentinel has built-in parsing functions for those and a UI tool to build a JSON parser as described in the blog post Tip: Easily use JSON fields in Sentinel. 

 

To ensure parsers are easy to use and transparent to analysts, they can be saved as functions and be used instead of Sentinel tables in any query, including hunting and detection queries. Read more about query time function parsers and the available KQL operators for parsing here.