Azure Data Explorer Insights (ADX Insights) provides a unified view of your clusters' usage, performance, and health. Now, you can monitor batching ingestion operations from ADX using the new "Ingestion" tab.
In the batching ingestion process, Azure Data Explorer optimizes data ingestion for high throughput by batching incoming small chunks of data into batches based on a configurable ingestion batching policy. The batching policy allows you to set the trigger conditions for sealing a batch (data size, number of blobs, or time passed). These batches are then optimally ingested for fast query results.
In this post, you will learn how to use ADX Insights to monitor batching ingestion.
On the Azure portal, go to the ADX cluster page > "Insights" blade ()> "Ingestion" tab.
The number of failed and successful ingestions is the number of blobs that were ingested or failed to be ingested. (The ingestion process is performed in blobs. Event Hub and IoT Hub ingestion events are aggregated into a single blob (multiple events per blob) and then processed as a single blob (source) blob for ingestion)
Click on the "Successful" or "Failures" tabs to drill down and see more details per database and table, including:
Table level monitoring is based on diagnostic logs. To see table-level details ,make sure to enable the ingestion diagnostic logs, according to your monitoring needs, and send them to Log-Analytics.
On the other two tabs, you will find information about:
In the batching ingestion process, Azure Data Explorer optimizes data ingestion for high throughput by batching incoming small chunks of data into batches based on a configurable ingestion batching policy. The batching policy allows you to set the trigger conditions for sealing a batch to be ingested (the conditions are: data size, number of blobs, or time passed. More possible conditions that can’t be configured in the batching policy, can be found here: batching types). These batches are then optimally ingested for fast query results.
There are four stages to batching ingestion, and there are specific components for each step:
You can monitor your data connections (per event hub or IoT hub) and track the "received data size" by each data connection.
You can also monitor the "discovery latency" – this is the time frame from data enqueue until data is discovered by ADX. This time frame is upstream to Azure Data Explorer. Discovery latency is available only for data connections (Event Hub, IoT Hub, or Event Grid ingestion) and it measures the time until data is discovered by the data connection.
When you see a long latency until data is ready for query, analyzing the discovery Latency and the next stage latencies (in the next steps) can help you understand whether the long latency is because of long latency in ADX, or is upstream to ADX.
When applying Event Hub, IoT Hub, or Event Grid ingestion, it can be useful to compare the number of events arriving at ADX data connection with the number of events sent for the next steps of the ingestion process (in other words, they were processed successfully by the data connection stage). The tiles Events Received, Events Processed, and Events Dropped allow you to make this comparison.
The second component of the batching ingestion proceeds is the Batching Manager, which optimizes ingestion throughput by batching data based on the ingestion batching policy.
This step allows you to monitor aspects such as:
Moreover, you can view "per-table" data: batching duration per table, the batching size per table, and how the batches were sealed per table (as determined by the ingestion batching policy details.)
The 3ed and 4th components are Ingestion Manager and Engine Storage, respectively. In the Engine Storage, you can see the accumulative latency per database - the time from the moment ADX accepts the data until the data is received by the Engine Storage, and it is available for query.
The "Amount of data processed" tile shows the number of blobs received, blobs processed (== successfully), and blobs dropped. Blobs that have been processed by the engine storage are ready for query.
Each definition described here can be found throughout the experience!
The definitions of the batching steps are hidden by default, but can be shown by using the "Show help" toggle.
Feel free to comment on this blog post. You can also use the feedback button () on the top of the "Insights" page.
More information about the metrics of the batching ingestion can be found here.
More information about other tabs of ADX Insights can be found here.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.