Introduction to Network Trace Analysis Part 1: Asking Questions and Collecting Data

Published Jul 18 2022 07:55 AM 2,296 Views
Microsoft

 

Howdy everyone! Today’s post will be a bit shorter but will cover how we collect network traces and what information we need to make use of the data we collect. 

 

Asking Questions 

 

I know it can be tempting to spin up WireShark and jump right into looking at traces, but asking questions is just as important, if not more important than the traces themselves. 

 

I usually like to group these questions into two groups: technical and general. 

 

Note: I will be using the terms client and server to refer to the sender and receiver. The client is always the sender, the server is always the receiver. 

 

Technical Questions: 

 

  • Who is talking? 
  • What is the client’s IP address? 
  • What is the server’s IP address? 
  • How are we getting there? 
  • What is the network path between these two endpoints? 
  • What language are they speaking? 
  • What protocol is the traffic using and over what port? 

It may seem like we are getting really into the networking weeds with these questions, but they are primarily about context.  By clarifying who is talking we can cut out noise, clarifying how we get there helps us avoid rabbit holes, clarifying the language helps us understand what the rules are for their conversation. 

 

General: 

 

  • Does this always happen or just sometimes? 
  • Has this ever worked? 
  • Does this work with a different source or destination? 

By combining these two sets of questions we can start understanding which locations are our primary suspects for the cause of the issue. 

 

For example: 

 

  • Q: Does this work with a different source or destination? 
  • A: Yes! It works with the old server 

What this tells us is that it is less likely that the issue is when the packet is leaving the sender. 

 

  • Q: What is the network path between these two endpoints? 
  • A: These devices are in the same subnet. (Think back to the post office metaphor from part 0) 

This tells us that it is less likely that the issue occurs along the network path. 

Given this new information, we would want to start by looking on the destination to see if anything could be interrupting the communication. 

 

Running tests 

 

Another benefit of asking the questions above is that it allows us to understand how to perform specific tests or what information we need to flesh out ourselves. 

For the purposes of this article, all testing is done via Windows PowerShell. 

 

What is the clients IP address / What is the servers IP address? 

 

Finding the relevant IP addresses is essential and we can quickly determine this information by running ipconfig /all 

 

Here is an example output from a client: 

 

 

PS C:\> ipconfig /all 
 
Windows IP Configuration 
 
   Host Name . . . . . . . . . . . . : Client01 
   Primary Dns Suffix  . . . . . . . : contoso.com 
   Node Type . . . . . . . . . . . . : Hybrid 
   IP Routing Enabled. . . . . . . . : No 
   WINS Proxy Enabled. . . . . . . . : No 
   DNS Suffix Search List. . . . . . : contoso.com 
                                       tailspin.com 
                                       fabrikam.com 
 
Ethernet adapter Ethernet 2: 
 
   Connection-specific DNS Suffix  . : fabrikam.com 
   Description . . . . . . . . . . . : Microsoft Hyper-V Network Adapter 
   Physical Address. . . . . . . . . : 00-15-5D-CA-40-0E 
   DHCP Enabled. . . . . . . . . . . : Yes 
   Autoconfiguration Enabled . . . . : Yes 
   Link-local IPv6 Address . . . . . : fe80::d5cf:204:2294:daeb%8(Preferred) 
   IPv4 Address. . . . . . . . . . . : 192.168.1.22(Preferred)                 // This is the client IPv4 address 
   Subnet Mask . . . . . . . . . . . : 255.255.255.0 
   Lease Obtained. . . . . . . . . . : Thursday, May 12, 2022 11:35:18 PM 
   Lease Expires . . . . . . . . . . : Friday, May 13, 2022 9:30:55 AM 
   Default Gateway . . . . . . . . . : fe80::215:5dff:feca:4009%8 
                                       192.168.1.1 
   DHCP Server . . . . . . . . . . . : 192.168.1.1 
   DHCPv6 IAID . . . . . . . . . . . : 117445981 
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-29-49-9F-7D-00-15-5D-CA-40-0E 
   DNS Servers . . . . . . . . . . . : 192.168.1.10                            // This is the client primary DNS server 
   NetBIOS over Tcpip. . . . . . . . : Enabled 
   Connection-specific DNS Suffix Search List : 
                                       fabrikam.com 
                                       tailspintoys.com 

 

 

And we can run the same command on the server to understand its IP address. 

There is a TON of great info in this output, but the keys are going to be: 

  • IPv4 Address 
  • DNS Server 
  • Default Gateway 

 

What is the network path between these two endpoints? 

 

A network path is the path a packet will take when being transported between the client and server. 

If the client and server are on different subnets one or more routers will be needed to route the traffic between subnets. 

To determine what the network path looks like you can use tracert. 

 

Here is an example of a network path between a client and server where the machines are in two different subnets: 

 

 

PS C:\> tracert dc01.contoso.com 
 
Tracing route to dc01.contoso.com [192.168.1.10] 
over a maximum of 30 hops: 

// Here we can see the IP address of the router or gateway that was traversed 
  1    <1 ms    <1 ms    <1 ms  172.16.10.1        
  2     3 ms     1 ms    <1 ms  192.168.1.10 

 

 

 

This allows us to understand what points we may need to investigate. If they are in the same subnet, we can likely limit the scope of the issue to the two machines instead of any network devices along the path. And on the flip side, if there are lots of network devices along the path we may need to investigate based off where the last place we saw the traffic in question. 

 

 

What port & protocol is communication taking place over? 

 

Port and protocol are the most network-y part of these questions. The protocol tells us what we can expect from the conversation on the network. Are any rules that this traffic should be following? Where in the lifecycle of this traffic are we? How can we zero in on the specific conversation we care about? 

 

For example, let’s say that I am having issues connecting to an external webserver. 

WillAftring_0-1658154995686.png

 

 

 

We can see that we are waiting for iis.contoso.com. What happens if we try to remove one of the layers we are using?  

Given that this is HTTP traffic we know: 

 

  • HTTP sits on top of TCP 
  • HTTP by default will use port 80 

We are going to use the PowerShell cmdlet Test-NetConnection to test TCP connectivity. 

 

 

PS C:\> Test-NetConnection iis.contoso.com -Port 80 
 
ComputerName     : iis.contoso.com 
RemoteAddress    : 192.168.1.24    
RemotePort       : 80 
InterfaceAlias   : Ethernet  
SourceAddress    : 192.168.1.10 
TcpTestSucceeded : True 

 

 

 

The result of this command tells us a few things. 

 

  1. We were able to resolve the name iis.contoso.com 
  2. We were able to successfully able to establish a TCP connection to 192.168.1.24 on port 80 

This means that if the Test-NetConnection succeeds, but the web browser fails then we have eliminated basic connectivity as the cause of the issue. It allows us to re-scope the issue to the web browser of the web server. 

 

Collecting Network Traces 

 

There are a few ways to collect and review network traces. For the sake of this series, we will be reviewing all network traces in WireShark as it is the gold standard of trace analysis. 

 

As a quick summary, Wireshark is a packet capture and analysis tool created by the WireShark foundation. It is open source and cross-platform and my favorite tool for reviewing network traces. For more information, please visit Wireshark · Go Deep. 

When collecting data, I always recommend collecting a simultaneous two-sided network trace while reproducing the issue. 

Tracing on only one machine is like listening to only half of a phone call. You can kind of guess what the other person is saying but, I’d rather hear it directly from them to prevent any confusion. 

 

Collecting captures in WireShark 

 

Collecting captures in WireShark is very straightforward. 

 

  • Install WireShark 
  • Launch WireShark 
  • Select the capture icon 

WillAftring_1-1658154995688.png

 

 

 

These captures can be saved and reviewed on other machines. 

 

WillAftring_2-1658154995690.png

 

 

Alternatively, you can start a capture using dumpcap.exe (a tool shipped with Wireshark). 

 

Example: 

 

 

PS C:\Program Files\Wireshark> .\dumpcap.exe -w Example.pcapng 
Capturing on 'Ethernet' 
File: Example.pcapng 
Packets captured: 32 
Packets received/dropped on interface 'Ethernet': 32/0 (pcap:0/dumpcap:0/flushed:0/ps_ifdrop:0) (100.0%) 

 

 

 

This created pcapng file can then be opened in WireShark. 

 

WillAftring_3-1658154995693.png

 

 

 

Collecting captures using netsh 

 

Starting in Windows 7 SP2, the operating system added functionality to collect packet captures with the inbox tool netsh. 

 

The tool is very powerful and provides lots of configuration options, but the simplest configuration uses the following parameters.

 

 

PS C:\> netsh trace start capture=yes tracefile=C:\netsh.etl 
 
Trace configuration: 
------------------------------------------------------------------- 
Status:             Running 
Trace File:         C:\netsh.etl 
Append:             Off 
Circular:           On 
Max Size:           250 MB 
Report:             Off 
 
PS C:\> netsh trace stop 
Merging traces ... done 
Generating data collection ... done 
The trace file and additional troubleshooting information have been compiled as "C:\netsh.cab". 
File location = C:\netsh.etl 
Tracing session was successfully stopped. 

 

 

 

There are two important things to keep in mind: 

 

  • The netsh command will be in the etl format and will need to be converted to pcapng using etl2pcapng 
  • The netsh command also collects a cab file with a handful of wonderfully insightful data 
  • IPconfig data 
  • Operating system versions 
  • etc… 

 

Conclusion 

 

I’m sure you are itching to dig into captures but let’s remember there is more than looking at network traces. As a reminder: 

 

  • Ask questions, ask questions, ask questions. 
  • Run your tests for additional context. 
  • Collect your traces. 

Starting in the next post we will dig into reviewing these traces and what you can expect to see in different failure scenarios. Until then have fun poking around with network captures, I’ll catch you later. 

 

Previous posts in the series 

Co-Authors
Version history
Last update:
‎Jul 18 2022 12:25 PM
Updated by: