As you might already know, there are a couple of ways of filtering traffic in Azure Virtual Networks: Network Security Groups (NSGs) and Azure Firewall. NSGs offer unlimited performance for Layer 4 filtering, while Azure Firewall is more powerful with features like deep packet inspection or application-level intelligence.
However, even if these solutions follow a common goal (drop undesired network traffic), they are based on completely different technologies and have very disparate logging formats. However, what if you suspect that “something” is dropping your legitimate traffic, but you don’t know whether that “something” is an Azure Firewall or an NSG? You would have to look into both sets of logs.
You can probably do this in Azure Sentinel, but due to my lack of knowledge there (shame on me!) I will use the tools I know: Kusto queries on Log Analytics and processing logs in storage accounts: Welcome to “unified” network security log analysis!
Logs? What logs?
If you don’t know yet what I am talking about, this section is for you. In Azure you can configure Network Security Groups to allow or deny traffic to a virtual machine or a complete subnet, and those operations (allow or deny) can optionally be recorded in so called “Flow Logs”:
These flow logs are sent to a Storage Account, and optionally to a Log Analytics workspace where they will be enriched with additional information for Traffic Analytics. Note that this export to Traffic Analytics can happen either every 10 minutes or every 1 hour, with certain cost implications of that decision (see Network Watcher Pricing for more on that).
The way of configuring Azure Firewall logs is completely different, this is done (as for most other Azure services) with the Diagnostic Settings of the Azure Firewall:
In the previous screen you can see some differences already: sending the logs to a Storage Account is optional, and there is no “collection process” that takes place every 10 minutes or every hour. Hence, the firewall logs ingestion time for Log Analytics will typically be lower than for NSG flow logs, but eventually both will end up there.
The Workbook
Some time ago I blogged here about how to extract additional insights from data in Traffic Analytics, one of my favorite Azure services. In that blog I showcased a custom Azure Workbook with tabs for Anomaly Detection and NSG Hit analysis. I will use and expand the same workbook (which you can find in GitHub here) with a third tab, “Firewall+FlowLogs”:
The screenshot above is rather small, but you might see that I have applied some filters looking for traffic in the last 5 minutes going to or from the IP address 192.168.0.5:
The returned table shows me logs coming from both the Azure Firewall and NSGs, as the column OperationName shows, giving me the proverbial single pane of glass for a combined analysis of packet drops in either Azure Firewall or Network Security Groups.
Getting the logs from Azure Storage
You might say that you actually cannot wait for the around 5-10 minutes in average that it takes Traffic Analytics to ingest NSG Flow Logs. No worries, I have you covered: some years ago I created a Python script that retrieves the latest flow logs from a storage account, which has a lower ingestion latency (and a lower price point) than Traffic Analytics: https://github.com/erjosito/get_nsg_logs.
When running the script you can now select the logs that you want to retrieve from the storage account with the flag --mode: either only NSG flow logs (the default), only Firewall logs, or both.
Consider though that if you don’t use Traffic Analytics you will lose all the enriched information that it gives you, plus you will not be able to run complex queries for example to detect anomalies or security breaches.
Enjoy!
Hopefully this post has given you some ideas to save some time when you are troubleshooting your next connectivity problem!