In today's data-driven world, it's essential to collect, analyze, and gain insights from various logs and data sources. Azure Data Explorer (aka Kusto) is a cloud-based data analytics platform designed for analyzing and visualizing large volumes of data in real time and is particularly well-suited for log time series and telemetry data. Its real-time capabilities make it a valuable resource for organizations looking to gain insights from their data such as forecasting, anomaly detection, and prediction among various other capabilities. More on leveraging the power of Kusto can be found here.
Splunk Universal Forwarder, commonly referred to as "Splunk UF," is a lightweight version of the Splunk Enterprise software. It is specifically designed for the sole purpose of collecting and forwarding log data and machine data from various sources to a central Splunk Enterprise server or a Splunk Cloud deployment. Splunk Universal Forwarder serves as an agent that simplifies the process of data collection and forwarding, making it an essential component in a Splunk deployment. More details on Splunk Universal Forwarder can be found here
In this article, we will talk about how we forward logs using Splunk Universal Forwarder to Azure Data Explorer (Kusto) using Kusto Splunk Univeral Forwarder Connector.
To forward logs or Splunk logs to Kusto, we need the Splunk Universal Forwarder to be installed in the same machine where the logs are located. The Splunk Universal Forwarder will be configured to send messages via TCP 9997 port (default port) to another system that will be hosting the Kusto Splunk Universal Forwarder Connector. This connector will be listening to the messages arriving on the TCP 9997 port and will forward the logs to Kusto. The following diagram represents the setup.
Before we begin, make sure you have the following prerequisites in place:
Splunk Universal Forwarder: Install Splunk Universal Forwarder on the machine where your logs originate. For the Installation guide please refer here.
Azure Data Explorer (Kusto): Set up an Azure Data Explorer cluster and database to receive the logs. More on this can be found here.
Azure Active Directory Application: Create an Azure AD application and service principal to enable communication between Splunk and Kusto.
[default]
index = default
disabled = false
[monitor://C:\Program Files\Splunk\var\log\splunk\modinput_eventgen.log*]
sourcetype = modinput_eventgen
[tcpout]
defaultGroup = default-autolb-group
sendCookedData=false
[tcpout:default-autolb-group]
server = 127.0.0.1:9997
[tcpout-server://127.0.0.1:9997]
cd .\SplunkADXForwarder\
ingest_url : <ingest URL>
client_id : <azure-client-id>
client_secret : <azure-client-secret>
authority : <authority>
database_name : <kusto-database-name>
table_name : <kusto-table-name>
table_mapping_name :
data_format : csv
docker build -t splunk-forwarder-listener
docker run -p 9997:9997 splunk-forwarder-listener
Once the docker starts running, data is sent to your Azure Data Explorer table. You can verify that the data is ingested by running a query in the web UI query editor.
In conclusion, we have demonstrated how to ingest data using Splunk Universal Forwarder to route data to Azure Data Explorer, a powerful cloud-based data analytics platform. By using the Kusto Splunk Universal Forwarder Connector, we can easily forward logs from various sources to Kusto and perform real-time analysis and visualization. This enables us to gain insights from our data and leverage the capabilities of Kusto for various scenarios such as forecasting, anomaly detection, and prediction. We hope this article helps you get started with Splunk and Kusto integration and explore the benefits of both technologies.
This connector can be used in Real-Time Analytics in Microsoft Fabric. Use the instructions in this article with the following exceptions:
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.