Azure Sentinel: Creating Custom Connectors

[Ofer_Shezaf]

Before you go the custom connector way

If the Sentinel data connectors page does not include the source you need, you may still not need a custom connector. Review the following blog posts for additional sources that can be used with Sentinel without a custom connector:

• Collecting telemetry from on-prem and IaaS server
• Collecting Azure PaaS services logs
• The Syslog and CEF source configuration grand list

If you still can't find your source in any of those, custom connectors are the solution.

The HTTP Data Collector API

The fundamental way to get custom data to your Sentinel workspace is using the HTTP Data Collector API. The API will enable you to write code to ingest any data to Sentinel. Importantly, this can be used not just for event data, but also for context and enrichment data such as threat intelligence, user or asset information.

To use the API, you can utilize those client libraries. Java does not have a client library. However, this Java client example can help you get going with Java also.

Using Azure Functions to implement the API connector is especially valuable as it keeps the connector serverless. The example How Azure Functions and Log Analytics provided easy and universal app logging for LISA App can help you to learn how to implement the API base connector using Azure functions.

Using PowerShell

Too much programming? The Upload-AzMonitorLog PowerShell script enables you to use PowerShell to stream events or context information to Sentinel. While it uses the same API behind the scenes, it is much simpler.

For example, this command will upload a CSV file to Sentinel: 

Import-Csv .\testcsv.csv 
| .\Upload-AzMonitorLog.ps1 
-WorkspaceId '69f7ec3e-cae3-458d-b4ea-6975385-6e426'
-WorkspaceKey $WSKey
-LogTypeName 'MyNewCSV'
-AddComputerName 
-AdditionalDataTaggingName "MyAdditionalField" 
-AdditionalDataTaggingValue "Foo"

The script takes the following parameters

• WorkspaceId - The Workspace ID of the workspace that would be used to store this data
• WorkspaceKey - The primary or secondary key of the workspace that would be used to store this data. It can be obtained from the Windows Server tab in the workspace Advanced Settings
• LogTypeName - The name of the custom log table that would store these logs. This name will be automatically concatenated with "_CL."
• AddComputerName - If this switch is indicated, the script will add to every log record a field called Computer with the current computer name
• TaggedAzureResourceId - If exist, the script will associate all uploaded log records with the specified Azure resource. This will enable these log records for resource-context queries as well as adhere to resource-centric role-based access control.
• AdditionalDataTaggingName - If exist, the script will add to every log record an additional field with this name and with the value that appears in AdditionalDataTaggingValue. This happens only if AdditionalDataTaggingValue is not empty
• AdditionalDataTaggingValue - If exist, the script will add to every log record an additional field with this value. The field name would be as specified in AdditionalDataTaggingName. If AdditionalDataTaggingName is empty, the field name will be "DataTagging."

While discussing PowerShell, the MaxPatrol connector PowerShell example shows an alternative implementation using PowerShell.

Using Logic Apps

Another alternative is to use Logic Apps to get events or context data to Sentinel. To do that, build a playbook with the following elements:

• Use one of these triggers to start the playbook:

1. Recurring task - schedule the connector, for example, for retrieving data from files, databases, or external APIs.
2. On-demand triggering - for manual upload and testing​​​​​​.
3. HTTP/S endpoint - if the source system can initiate the transfer and for streaming. 

• Read the data using one of the following connectors:​​​​​​

1. Using a REST API
2. Read SQL Server data
3. Read a file
4. Note that those connectors support retrieving data on-premises.

• Prepare the information, for example, using the Parse JSON action.
• Write the data to Log Analytics using the Logic Apps connector for writing data to Log Analytics. 

There are many examples out there for doing so:

• Create Custom Log Analytics logs with Logic Apps walks you through the steps and provides you with an excellent example of using parse JSON
• Getting MDATP alerts into Sentinel using either Paul Huijbregts or Tom Lilly's variants provides a real-world use case, and so will Sending Proofpoint TAP logs to Azure Sentinel

Note that while convenient, this method may be costly for large volumes of data and should be used only for low volume sources or for context and enrichment data upload.

Logstash

If you know Logstash, this might be your best bet. Logstash has an output plugin for Sentinel which enables you to use LogStash as a collector for Sentinel. Now you can use all your GROK prowess as well as any Logstash input plugin to implement your connector. The challenge is that unlike all other methods above, this would require a VM and cannot be serverless.

Parsing

The API and therefore all the other methods which utilize it allow defining fields. Using this feature, you can parse the information as part of the custom parser withing Logstash, Logic App, or your custom code.

However, Sentinel allows parsing at query time which offers much more flexibility and simplifies the import process. Query time allows you to push data in at the original format and parse on demand when needed. Updating a parser will apply to already ingested data.

Query time parsing reduces the overhead of creating a custom connector as the exact structure of the data does not have to be known beforehand. Nor do you need to identify the vital information to extract. Parsing can be implemented at any stage, even during an investigation to extract a piece of information Adhoc and will apply to already ingested data.

JSON, XML, and CSV are especially convenient as Sentinel has built-in parsing functions for those as well as a UI tool to build a JSON parser as described in the blog post Tip: Easily use JSON fields in Sentinel. 

To ensure parsers are easy to use and transparent to analysts, they can be saved as functions and be used instead of Sentinel tables in any query, including hunting and detection queries. The blog post Using KQL functions to speed up analysis in Azure Sentinel describes how to do that.

The full documentation for Sentinel parsing can be found here.