Introduction:
In the ever-evolving landscape of cloud computing, Log Analytics Workspace is used as a tool in Azure to collect logs, edit/run log queries and interactively analyze query results.
As organizations scale their infrastructure and applications, the volume of observability data naturally increases.
A query running in Log Analytics workspace can return a maximum of 30,000 records. However, there are several instances where huge amount of data needs to be extracted and analyzed. Some of the scenarios are:
- Data over a long period of time: Organization needs several months of data which is high in volume and number of records.
- Monitoring Solution Design: Sometimes there is a need to capture all logs under one workspace for Security/Compliance team to review. Hence, some organizations monitor all their subscriptions under one tenant in one workspace. This causes causing centralization of a huge volume of data in one workspace.
- Larger Scope while querying data: Some organizations have monitoring data spread across several workspaces. A query can be run to fetch the data across workspaces. This results in large number of records.
This limitation of the 30,000 records in the workspace leads to writing the same query in a shorter time range, running it multiple times to get the data in batches and combining that data at the end. Hence causing re-running/re-writing query with additional efforts and taking more time than expected to fetch logs.
To address this challenge and to empower customers with the ability to query and fetch data in one go, Azure Data explorer service in Azure can be utilized.
What is Azure Data Explorer?
Azure Data Explorer is a platform for high-performance which helps to analyze high volumes of data in near real time. The Azure Data Explorer provides an end-to-end solution for data ingestion, query, visualization, and management. Azure Data Explorer is ideal for enabling interactive analytics capabilities over high velocity, diverse raw data.
Getting Started: Query data in Azure Monitor using Azure Data Explorer
- Open https://dataexplorer.azure.com and click on “Query” from left pane.
Then click on “Add+” and select “Connection”.
2. You will see an option to add a Connection. Put the following values in:
- Connection URI: Put Log Analytics Workspace URI in the format of
https://ade.loganalytics.io/subscriptions/<subscription-id>/resourcegroups/<resource-group-name>/pro...>
Note: You will find the above details in under Log Analytics Workspace -> Properties -> Resource ID
- Display Name: Name of the workspace or anything as per convenience
Click on Add to establish the connection between Azure Data explorer and the workspace.
- Once the connection is established, you can run a query and fetch the records. You need to select the database from the left pane before running the query.
Advantages:
- Fetch records in one go thus saving time and manual efforts of re-writing/re-running query.
- No overhead of creating a data cluster in Azure Data explorer thus reducing cost and complex setup.
- Delivers high performance along with variety and volume of data.
- Larger query scope and time range.