Announcing Azure Firewall Structured Logs General Availability

Microsoft

Jun 14, 2023

As part of Azure Firewall continues strive to improve its troubleshooting capabilities, we have recently announced New flow logs and latency metrics. Today, we are happy to announce the general availability of Azure Firewall Structured Logs capability.

Azure Firewall is a Cloud-native Firewall as a Service offering that enables customers to centrally govern and log all their traffic flows using a DevOps approach. The service supports both application and network-level filtering rules and is integrated with the Microsoft Defender Threat Intelligence feed to filter known malicious IP addresses and domains. Azure Firewall is highly available with built-in auto-scaling.

Recently, Azure Firewall added a new capability called ‘Structured Logs’ that provides an enhanced logging experience for firewall events. In this blog post, we'll explore what structured logs are and how they can benefit your organization.

What are Structured Logs?

Structured logs are a type of log data that are organized in a specific format. They use a predefined schema to structure log data in a way that makes it easy to search, filter, and analyze. Unlike unstructured logs, which consist of free-form text, structured logs have a consistent format that can be parsed and analyzed by machines.

Azure Firewall's structured logs provide a more detailed view of firewall events. They include information such as source and destination IP addresses, protocols, port numbers, and action taken by the firewall. They also include additional metadata, such as the time of the event and the name of the Azure Firewall instance.

Azure Firewall legacy log categories are using Azure diagnostics mode. In this mode, the entire data from any diagnostic setting will be collected in the AzureDiagnostics table.

With this new feature, customers will be able to choose using Resource Specific Tables instead of the existing AzureDiagnostics table. In case both sets of logs are required, at least two diagnostic settings would need to be created per firewall.

When Resource specific mode is selected by the user, ‘Structured Logs’ are enabled and individual tables in the selected workspace are created for each category selected in the diagnostic setting.

This is the recommended method since it makes it easier to work with the data in log queries, provides better discoverability of schemas and their structure, improves performance across both ingestion latency and query times, and the ability to grant Azure RBAC rights on a specific table.

Azure Firewall Structured logs allow users to utilize the following newly added categories:

Network rule log - Contains all Network Rule log data. Each match between data plane and network rule creates a log entry with the data plane packet and the matched rule's attributes.
NAT rule log - Contains all DNAT (Destination Network Address Translation) events log data. Each match between data plane and DNAT rule creates a log entry with the data plane packet and the matched rule's attributes.
Application rule log - Contains all Application rule log data. Each match between data plane and Application rule creates a log entry with the data plane packet and the matched rule's attributes.
Threat Intelligence log - Contains all Threat Intelligence events.
IDPS log - Contains all data plane packets that were matched with one or more IDPS (Intrusion Detection and Prevention System ) signatures.
DNS proxy log - Contains all DNS (Domain Name System) Proxy events log data.
Top Flows - The Top flows log (known in the industry as Fat Flows), shows the top connections that are contributing to the highest throughput through the firewall.
Flow Trace - Contains flow information, flags and the time period when the flows were recorded.
Internal FQDN resolve failure log - Contains all internal Firewall FQDN (Fully Qualified Domain Name) resolution requests that resulted in failure.
Application rule aggregation log - Contains aggregated Application rule log data for Policy Analytics.
Network rule aggregation log - Contains aggregated Network rule log data for Policy Analytics.
NAT rule aggregation log - Contains aggregated NAT rule log data for Policy Analytics

Benefits of Structured Logs

The benefits of structured logs include improved searchability, easier analysis, and better integration with other tools. Because structured logs have a consistent format, they can be easily searched and filtered to find specific information. This makes it easier to troubleshoot issues and identify patterns in network traffic.

Structured logs also enable easier analysis using tools such as Azure Monitor and Azure Sentinel. These tools can parse the structured logs and provide insights into network traffic patterns and potential security threats. This allows organizations to proactively detect and respond to security threats, improving their overall security posture.

In addition, structured logs can be easily integrated with other tools in your organization's security stack. They can be exported to a variety of destinations, such as Azure Blob Storage or Azure Event Hubs, and then ingested into other tools for further analysis.

Structured logs can also be useful in troubleshooting network performance issues. As noted in a recent Microsoft blog post, structured logs can be used to identify network bottlenecks, detect configuration errors, and troubleshoot network connectivity issues. By analyzing structured logs, you can gain a better understanding of your network traffic patterns and identify potential issues before they become major problems.

Enabling Structured Logs in Azure Firewall

To enable structured logs in Azure Firewall, you must first configure a Log Analytics workspace in your Azure subscription. This workspace is used to store the structured logs generated by Azure Firewall.

Once you have configured the Log Analytics workspace, you can enable structured logs in Azure Firewall by navigating to the Firewall's ‘Diagnostic settings’ blade in the Azure portal. From there, you are required to select ‘resource specific’ destination table and to select the types of events you want to log, as see in the below diagram.

Structured Logs Queries

A list of predefined queries is available in Azure Firewall Portal. This list consists of a predefined (Kusto Query Language) log query for each category as well as joined query showing the entire Azure firewall logging events in single view.

New Azure Firewall Workbook

Azure Firewall Workbook provides a flexible canvas for Azure Firewall data analysis. You can use it to create rich visual reports within the Azure portal. You can tap into multiple Firewalls deployed across Azure and combine them into unified interactive experiences.

To deploy the new workbook, which utilize Azure Firewall Structured Logs, go to Azure Monitor Workbook for Azure Firewall and follow the instructions on the page.

Conclusion

Azure Firewall's new capability, Structured Logs, provides enhanced logging experience for firewall events. By using a consistent format, structured logs make it easier to search, filter, and analyze network traffic data. This enables organizations to proactively detect and respond to security threats, troubleshoot network performance issues, and improve their overall security posture. If you're using Azure Firewall, consider enabling structured logs to get a more detailed view of your network traffic and enhance your security monitoring capabilities.

Learn more