Using Packet Capture for troubleshooting Azure Firewall flows

This blog is written in collaboration with @GustavoModena

Introduction

Azure Firewall is a cloud-native and intelligent network firewall security service that provides best of breed threat protection for your cloud workloads running in Azure. It’s a fully stateful firewall as a service with built-in high availability and unrestricted cloud scalability. Azure Firewall provides both east-west and north-south traffic inspection, and it is offered in three SKUs: Basic, Standard and Premium.

Azure Firewall also brings powerful logs and metrics to monitor your traffic and operations within the firewall. These logs and metrics include Traffic Analysis, Performance and Health Metrics, and Audit Trail. However, there are situations where you may need a comprehensive network packet capture to troubleshoot and investigate an incident reported by users. We are happy to announce that Microsoft just released the new Packet capture feature and it is Generally Available for Azure Firewall.

The Packet capture feature in Azure Firewall is intended for troubleshooting purposes and will allow customers and engineers to debug connectivity issues by tracing packets passing through their Azure Firewall.  Azure Firewall Packet Capture shows two packets per flow, one for incoming direction and one for outgoing direction, so you can accurately correlate requests and responses during troubleshooting.

What is a network packet capture?

Network packet capture is a process that involves capturing network packets as they traverse a network interface. It's a valuable tool for network troubleshooting, analysis, and security monitoring.

A network packet capture involves intercepting Internet Protocol (IP) packets for analysis and then saving the packets captured to output files, typically saved in the “.pcap” file extension. Network engineers often utilize packet capturing for troubleshooting and monitoring network traffic to identify security threats. In the event of a data breach or other incident, packet captures offer essential forensic evidence that supports investigations. From a malicious actor’s viewpoint, packet captures can be used to steal passwords and other sensitive data. Unlike active reconnaissance techniques like port scanning, packet capturing can be conducted covertly, leaving no trace for investigators.

How Does a Packet Capture Work?

Packet captures can be performed using networking equipment like routers, firewalls or switches, or even an engineer’s laptop or desktop. Regardless of the method, packet capture involves creating copies of some or all packets passing through a particular point in the network.

Capturing packets from a specific device on the network is the simplest way to start troubleshooting, but there are a few caveats. By default, network interfaces only monitor traffic destined for them. For a more comprehensive view of network traffic, you’ll need to set the interface to promiscuous mode or monitor mode. Many routers, firewalls and other network devices have embedded packet capture functions that can be used to quickly troubleshoot directly from the device's admin console. This capability is now available in Azure Firewall.

Scenario (VNET to VNET)

In this blog we have VM-1 (10.10.0.4) unsuccessfully trying to establish HTTP (TCP 80)/HTTPS (TCP 443) connection to VM-2 (10.10.0.132) via Azure Firewall.

 

Using Azure Firewall Packet Capture to investigate the connection issue

In this section, we will use Azure Firewall Packet Capture to understand why an HTTP/HTTPS connection between VM-1 and VM-2 is not working properly. For this demonstration, we are not going to review the rules and Azure Firewall logs, as the purpose of the blog is to demonstrate the new Packet Capture feature, and we are assuming that the Azure Firewall is configured correctly.

Let’s start by making sure that we have all the required resources to take the packet captures from Azure Firewall:

• Azure Firewall with Management NIC enabled
• Storage account with a container in which you can store the packet captures

Once you have all the required resources available, follow the next steps to start running a Packet Capture via Azure Firewall:

 

1. Create a SAS URL to the container in the storage account: In the Azure Portal go to Storage Account > Containers and select the 3 ellipses at the very right side of the name of the container that you want to use to store the packet captures and select “Generate SAS”.

 

2. When defining the parameters of the SAS select “Write” under Permissions, so Azure Firewall will be able to successfully save the packet captures. Then click on “Generate SAS token and URL”.

 

3. Now, we must go to the Azure Firewall > Packet Capture (under Help) to start running the packet capture.
4. On the Packet Capture page, provide the following information:

• • Packet capture name - the name of one or more capture files.
  • Output SAS URL - the SAS URL of the storage container you created previously.

5. Next, complete the Basic settings for the packet capture:

• • Maximum number of packets - You should limit the packet capture to a set number of packets.
  • Time limit (seconds) - Since the packet capture is intended for troubleshooting purposes, you should limit the capture time.
  • Protocols - the protocols you want the capture to save (values: Any, TCP, UDP, ICMP).
  • TCP Flags - if TCP or Any is selected, you can select which types of packets to save (values: FIN, SYN, RST, PSH, ACK, URG)

If both the Maximum number of packets and Time limit are set, the capture ends when the earliest condition is met. So, either when the maximum number of packets is received or when the time limit is reached.

 

6. In the Filtering section, you can add the source, destination, and destination ports to include in the capture. You must add at least one filter.

• • The packet capture saves bidirectional traffic that matches each row in the filter section.
  • For the source and destination fields you can list multiple commas separated values in a single filter including IP addresses and IP blocks.

7. Select Run Packet Capture after you're done with your configuration.

 

8. Once the packet capture is complete, you will navigate to the container used in the storage account and download the pcap files. Note that you will see multiple pcap files, this is because each virtual machine in the backend of the firewall has its own file.

Analyzing the Packet Captures

When using Azure Firewall Packet Capture, you will always see two packets for every single packet in the flow. This is because the firewall captures both the incoming and outgoing directions of the traffic. Understanding this behavior is critical for accurate troubleshooting, as it ensures you can correlate the original request with its corresponding response. The additional scenarios below will explain how to match these incoming and outgoing flows effectively.

To analyze the pcap files you need a network protocol analyzer tool. In this blog we are using Wireshark.

Note: The intent of this blog is not to show how to use it nor to do advanced troubleshooting using Wireshark.

 

With the pcap files downloaded to your computer, open the files to start your investigation. Since we have multiple files due to the number of active Azure Firewall instances at the time of the packet capture, it may be easier to merge the files. To merge the pcap files, first open one of them using Wireshark and then go to File > Merge and select the second file. There are different ways to merge them, but here we are using “Merge packets chronologically”.

 

Once the pcap files are merged, you will start your investigation by using filters. In this scenario, we want to investigate why an HTTP request from VM-1 to VM-2 on port TCP 80 is not working, and we are using the following filter:

• Wireshark filter: tcp.port==80 && tcp.port==50245 && ip.addr==10.10.0.132 (VM-2’s IP address)

Ok, so here we can see that VM-1 (10.10.0.4) sends a SYN packet from port 53945 to VM-2 (10.10.0.132) on port 80, then VM-2 sends a reset back to VM-1. This behavior shows us that the traffic is successfully passing through Azure Firewall (allowed), and the issue may possibly be something on VM-2. After involving the application team, they have found an issue related to the IIS configuration and it is now fixed as we can see the TCP request being established on ports 80 and 443 in the screenshot below.

Other Scenarios

DNAT (Inbound traffic)

In this scenario we are connecting from a client via Internet to the Azure Firewall’s public IP, using DNAT rules on port 8443. You can see in the screenshot below the incoming request (TCP 3-way handshake) and all the hops until it gets to the Web Server. L3 (and source IP) differs from the incoming packet since its SNATed at L3 while L4 remains the same. For taking the packet capture in this scenario, we are using the following filters:

• Source: 71.28.90.56,52.176.62.243,10.10.0.64/26,10.10.0.128/26
• Destination: 71.28.90.56,52.176.62.243,10.10.0.64/26,10.10.0.128/26
• Destination ports: 8443,443

Check below to understand what each one of the IP/IP ranges and ports are used as filters:

• Client Public IP: 71.28.90.56
• Azure Firewall Public IP: 52.176.62.243
• Azure Firewall Instance Private IP: 10.10.0.69 (this IP is included in the IP range 10.10.0.64/26)
• Web Server Private IP: 10.10.0.132 (this IP is included in the IP range 10.10.0.128/26
• Azure Firewall Listening Port: 8443
• Web Server Listening (translated) Port: 443

In DNAT scenarios, you will notice two SYN packets for the same flow. SYN 1 represents the incoming packet with its original 5-tuple (source IP, destination IP, source port, destination port, protocol), while SYN 2 corresponds to the same flow but with a different 5-tuple after translation by Azure Firewall. This behavior contrasts with VNET-to-VNET flows, where the 5-tuple remains unchanged.

 

 

When you are SNATing, connecting to/from the Internet, or processing application rules, to see both incoming and outgoing packets you need to make sure that both Public IP address and subnet address space are included.

Internet Access (Outbound traffic)

In this scenario, we are connecting from an Azure VM to the public IP via Azure Firewall using Network rules. The screenshot illustrates the TCP three-way handshake followed by the HTTP GET request. Notice two SYN packets: one originating from the client to the destination and another from the Azure Firewall instance IP to the destination. In the first two lines, packets flow from the Azure VM IP to the external public IP, followed by the SNATed packet from the Azure Firewall instance IP to the same external address. For this packet capture, the following filters were applied:

• Source: 10.10.0.132, 10.10.0.0/26
• Destination: 151.101.195.5
• Destination ports: 80,443

Check below to understand what each one of the IP/IP ranges and ports are used as filters:

• Azure VM: 10.10.0.132
• Azure Firewall Subnet: 10.10.0.0/26 (10.10.0.5 is the instance IP)
• External Public IP: 151.101.195.5
• External Public IP Port: 80

Application Rule Traffic:

In this scenario, we are connecting from an Azure VM to the public IP via Azure Firewall using Application rules. While the original request originates from the VM with source IP 10.0.2.4, the Layer 4 details differ from the incoming packet because, during application rule evaluation, the firewall establishes a new outbound connection acting as a proxy. As shown in the image, the SNAT IP of the Azure Firewall instance (10.0.0.5) initiates the connection to the public IP 140.82.112.4. HTTP or TLS keys can be used to match incoming and outgoing packets. L7 remains the same. For packet capture in this scenario, the following filters are applied:

• Source: 10.0.2.4, 10.0.0.0/24
• Destination: 140.82.112.4
• Destination ports: 80,443

Check below to understand what each one of the IP/IP ranges and ports are used as filters:

• Azure VM: 10.0.2.4
• Azure Firewall Subnet: 10.0.0.0/24 (10.10.0.5 is the instance SNAT IP)
• External Public IP: 140.82.112.4
• External Public IP Port: 80,443

 

VNET to VNET with SNAT:

In this scenario, the client VM 10.1.0.4 initiates the connection to the server VM 10.0.2.4 but we have enabled SNAT to happen by default. So, the Firewall’s Private IP 172.16.0.5 (SNAT) will initiate a connection with the destination web server as we can see in the below image. For packet capture in this scenario, the following filters are applied:

• Source: 10.1.0.4, 172.16.0.0/24
• Destination: 10.2.0.4
• Destination ports: 80,443

Check below to understand what each one of the IP/IP ranges and ports are used as filters:

• Azure VM: 10.1.0.4
• Azure Firewall Subnet: 172.16.0.0/24 (172.16.0.5 is the instance SNAT IP)
• Web Server Private IP: 10.2.0.4
• Web Server Port: 80

 

 

Conclusion

The availability of Azure Firewall Packet Capture is crucial for effective network and security troubleshooting. It allows network administrators and security professionals to monitor, analyze, and diagnose network traffic in real-time, providing invaluable insights into potential issues and vulnerabilities. By capturing and examining data packets, they can identify anomalies, detect malicious activities, and ensure the integrity and performance of the network. This proactive approach not only enhances the overall security posture but also minimizes downtime and improves the reliability of network services, making packet capture an indispensable tool in the modern IT landscape.