Azure Monitor Logs Next Evolution: Multi-tier logging

Today we’re announcing the public preview of Auxiliary Logs, a new inexpensive Azure Monitor plan for verbose logs used in compliance and security scenarios.  Together with the recent public preview of Summary Rules and improved capabilities of Basic Logs, Azure Monitor Logs is evolving into a new multi-tier logging vision.

Most organizations have many different needs for logging.  This is because not all logs are the same - some are very frequently accessed, some are needed for investigation when issues arise, and some are kept mostly for audit and compliance purposes. When we talk to customers, we often hear that they use multiple logging services or products side-by-side to serve their needs, and this is slowing them down. They need to deploy, maintain and learn several different technologies just to observe their services. Customers are also telling us that they need to log much more data than before, which is ballooning their logging cost.

Azure Monitor is focused on addressing these problems with a single solution that includes multiple logging plans that cover a broad variety of scenarios. With Azure Monitor Logs, customers have a one-stop shop for observability.

These log plans are:  

• Auxiliary Logs – Our new, inexpensive log plan that enables ingestion and management of verbose logs needed for auditing and compliance scenarios. These may be queried with KQL on an infrequent basis and used to generate summaries.
• Basic Logs – Improved to support even richer troubleshooting and incident response with fast queries while saving costs. Now available with a longer retention period and the addition of KQL operators to aggregate and lookup.
• Analytics Logs – This plan is designed for frequent, concurrent access and supports interactive usage by multiple users. This plan drives the features in Azure Monitor Insights and powers Microsoft Sentinel. It is designed to manage critical and frequently accessed logs optimized for dashboards, alerts, and business advanced queries.

For detailed capabilities comparison between the plans, see our documentation. For the pricing of these plans, see here.

All these logs can be retained for up to 12 years, while being accessible using Search Jobs, that can scan Petabytes of data to find specific records. 

Since all these logs are in Azure Monitor, it means that they have the same KQL query language, API, query and admin experiences. Furthermore, we're not only integrating these logs under one roof, but also are providing additional capabilities to ensure that logs function cohesively:

• Summary Rules – Continuous aggregation of raw data into compact summaries that are easier to analyze and cheaper to retain
• Ingestion transformations – Enable filtering, enriching, and splitting data between log plans during ingestion
• Search Job – An orchestrated long-running query that can scan Petabytes of historical data to specific records into Analytics tables for further analysis

With this, it is easy to switch or mix-and-match between the different log plans. Customers can start with a table that is configured with the Analytics plan, then can check if Basic Logs works for them through a simple configuration change. If they find that they need more capabilities, they can revert to Analytics.

Here are two examples how these plans work together to create an improved solution:

• Firewall logs can be huge, 100s of TB per day of highly verbose data. Firewalls are optimized to omit all communications details. While most consumers don’t need the raw data for day-to-day use, organizations do need to keep it for auditing. In Azure Monitor, customers can now send all firewall logs to Auxiliary Logs and retain them per compliance requirements. On this data, customers can run a summary rule that creates hourly aggregations. Investigators can use these aggregations for their day-to-day work and if they need to drill down, they can easily query the relevant records from Auxiliary Logs.
• Some logs contain highly important information mixed with less important data. Customers can send these logs using our pipeline Data Collection Rules and split the data between Analytics and Basic Logs. The highly important records will have all analytics capabilities, while the less important information will be available for troubleshooting at a lower.

We are working closely with our customers to collect feedback and will continue to add more functionality to the service. We are always interested in hearing your thoughts and understanding how you apply Azure Monitor Logs to your environment. You can contact us using lafeedback@microsoft.com or via our feedback community forum.

Video provider requires cookies to play the video. Accept to continue or watch it directly on the provider's site.

Accept

For more details on how to configure Auxiliary Logs see here.

For more details on how to query Auxiliary Logs see here.