What's New: Create your own codeless data connector

[DanielZatakovy]

Today we're announcing the general availability of the Codeless Connectors Platform (CCP) in Microsoft Sentinel that provides partners, advanced users, and developers the ability to create custom connectors for ingesting data to Microsoft Sentinel.

How is this CCP different from the previous version?

The initial version of the CCP was announced in January of 2022. Since then, we've improved upon the platform and the legacy release is no longer recommended. This new version of the CCP has the following key improvements:

1. Better support for various authentication and pagination types.

2. Supports standard data collection rules (DCRs).

3. The user interface and connection configuration portions of the codeless connector are separate now. This allows the creation of connectors with multiple connections which wasn't possible previously.

At present, CCP allows connections to any data source with a public REST API endpoint. 

Key benefits of using CCP:

• Eliminates the need to write code for connecting to public REST APIs.
• Offers a scalable, built-in Poller as a service.
• Provides configurable UI components for your connector.
• Enables monitoring of your connectors; CCP integrates with Sentinel Connector Health messages, allowing for troubleshooting and health status updates.

Currently, CCP support connectors that are based on RestAPI, S3 and GCP.

 

Getting started

Prerequisites

Before building a connector, understand your data source and how Microsoft Sentinel needs to connect.

1. Data Collection Endpoint (DCE)

   A DCE is a requirement for a DCR. Only one DCE is created per log analytics workspace DCR deployment. Every DCR deployed for a Microsoft Sentinel workspace uses the same DCE. For more information on how to create one or whether you need a new one, see Data collection endpoints in Azure Monitor.

2. Schema of the output table(s).

   It's important to understand the shape of your data stream and the fields you want to include in the output table. Reference your data source documentation or analyze sufficient output examples.

Research the following components and verify support for them in the Data Connector API reference:

1. HTTP request and response structure to the data source

2. Authentication required by the data source.
   For example, if your data source requires a token signed with a certificate, the data connector API reference specifies cert authentication isn't supported.

3. Pagination options to the data source

We also recommend a tool like Postman to validate the data connector components. For more information, see Use Postman with the Microsoft Graph API.

Build the data connector

There are 4 components required to build the CCP data connector.

1. Output table definition
2. Data Collection Rule (DCR)
3. Data connector user interface
4. Data connector connection rules

Follow the steps for creating each component before creating the deployment template.

Deploy the connector

1. Copy the contents of the ARM deployment template.
2. Follow the Edit and deploy the template instructions from the article, Quickstart: Create and deploy ARM templates by using the Azure portal.

 

Verify the codeless connector

View your codeless connector in the data connector gallery. Open the data connector and complete any authentication parameters required to connect . Once successfully connected, the DCR and custom tables are created. View the DCR resource in your resource group and any custom tables from the logs analytics workspace.

Refer to the following illustration as an example of one of the connectors: this connector requires token endpoint, authorization token, user activity logs endpoint, client ID and client secret.

Note that It may take up to 30 minutes to see data begin ingesting.

What to Expect in the Future:

1. Platform extensibility:

1. Support APIs that are based on nested API calls.
2. Support for push type connectors.
3. Allow selection of endpoint in multi rule connectors.
4. KQL transformation in ingest time.
5. and more...

2. Dedicated webinar for creating a codeless data connectors.

3. UI that will assist creating codeless data connectors.

Learn more:

Create a codeless connector documentation:  Create a codeless connector for Microsoft Sentinel 

Data connector API reference: Data connector connection rules reference

Data connector definition API reference: Data connector UI definitions reference