Real-time data streams (events) from Azure Event Hubs to Azure Data Explorer (ADX) can be easily processed, before the ingestion to ADX, using Azure Stream Analytics (ASA). The ASA no-code editor allows you to develop a Stream Analytics job without writing a single line of code.
The experience provides a canvas that allows you to connect to Event Hub as an input sources, and quickly see your streaming data. Then, you can transform the data before writing to your Azure Data Explorer cluster.
You can:
- Modify input schemas of the events
- Perform data preparation operations like joins and filters
- Approach advanced scenarios like time-window aggregations (tumbling, hopping, and session windows) for group-by operations
When to use Azure Data Explorer and Azure Stream Analytics
Azure Stream Analytics:
- Stream Processing Engine - Continuous/Streaming real-time analytics
- Job based
- ASA has a lookback window period of 1ms to 7 days for in-memory temporal analytics/stream processing
Azure Data Explorer:
- Analytical Engine - On-demand/Interactive real-time analytics
- Streaming data ingestion into persistent data store along with super powerful querying capabilities
- Ingest data from Event Hubs, IoT Hub, Blob, Data Lake, Kafka, Logstash, Spark, ADF, and more.
- 10 seconds to 5 minutes latency for high throughput workloads
- Simple data transformation can be done with ADX's update policy, during ingestion
You can significantly grow the scope of real-time analytics by leveraging ASA and ADX together. Below are a few scenarios:
- ASA identifies anomalies in real time and ADX helps determine how and why it occurred through interactive exploration.
Note that ADX also offers out-of-the-box anomaly detection capabilities (post-ingestion).
- ASA deserializes incoming data stream for use in ADX (E.g. ingest Protobuff format by using custom deserializer, custom binaries formats etc.)
- ASA can aggregate, filter, enrich, and transform incoming event data streams, before ingesting it to ADX
How to access the ASA editor?
-
In the Azure portal, go to your Event Hub instance page.
-
Select Features > Process Data and then select Start on the Filter and store data to Azure Data Explorer card.
An Azure Data Explorer tile will be presented. On its configuration panel, fill out your ADX cluster parameters and hit connect.
Tip
To easily create the destination table in ADX, based on the actual events' schema, you can use the ADX One-Click ingestion wizard which can infer the schema of the events.
Anyway, the ADX One-Click ingestion wizard can be used if you don't require any pre-ingestion event processing and only wish to ingest data from Event Hub into ADX. Post-ingestion data processing can be done in ADX using an Update Policy.
Learn more
For the full step by step guide of using the ASA no-code editor, please visit: Filter and ingest to Azure Data Explorer using the Stream Analytics no code editor