Howdy everyone! Today’s post will be a bit shorter but will cover how we collect network traces and what information we need to make use of the data we collect.
I know it can be tempting to spin up WireShark and jump right into looking at traces, but asking questions is just as important, if not more important than the traces themselves.
I usually like to group these questions into two groups: technical and general.
Note: I will be using the terms client and server to refer to the sender and receiver. The client is always the sender, the server is always the receiver.
It may seem like we are getting really into the networking weeds with these questions, but they are primarily about context. By clarifying who is talking we can cut out noise, clarifying how we get there helps us avoid rabbit holes, clarifying the language helps us understand what the rules are for their conversation.
By combining these two sets of questions we can start understanding which locations are our primary suspects for the cause of the issue.
What this tells us is that it is less likely that the issue is when the packet is leaving the sender.
This tells us that it is less likely that the issue occurs along the network path.
Given this new information, we would want to start by looking on the destination to see if anything could be interrupting the communication.
Another benefit of asking the questions above is that it allows us to understand how to perform specific tests or what information we need to flesh out ourselves.
For the purposes of this article, all testing is done via Windows PowerShell.
What is the clients IP address / What is the servers IP address?
Finding the relevant IP addresses is essential and we can quickly determine this information by running ipconfig /all
Here is an example output from a client:
PS C:\> ipconfig /all Windows IP Configuration Host Name . . . . . . . . . . . . : Client01 Primary Dns Suffix . . . . . . . : contoso.com Node Type . . . . . . . . . . . . : Hybrid IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No DNS Suffix Search List. . . . . . : contoso.com tailspin.com fabrikam.com Ethernet adapter Ethernet 2: Connection-specific DNS Suffix . : fabrikam.com Description . . . . . . . . . . . : Microsoft Hyper-V Network Adapter Physical Address. . . . . . . . . : 00-15-5D-CA-40-0E DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes Link-local IPv6 Address . . . . . : fe80::d5cf:204:2294:daeb%8(Preferred) IPv4 Address. . . . . . . . . . . : 192.168.1.22(Preferred) // This is the client IPv4 address Subnet Mask . . . . . . . . . . . : 255.255.255.0 Lease Obtained. . . . . . . . . . : Thursday, May 12, 2022 11:35:18 PM Lease Expires . . . . . . . . . . : Friday, May 13, 2022 9:30:55 AM Default Gateway . . . . . . . . . : fe80::215:5dff:feca:4009%8 192.168.1.1 DHCP Server . . . . . . . . . . . : 192.168.1.1 DHCPv6 IAID . . . . . . . . . . . : 117445981 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-29-49-9F-7D-00-15-5D-CA-40-0E DNS Servers . . . . . . . . . . . : 192.168.1.10 // This is the client primary DNS server NetBIOS over Tcpip. . . . . . . . : Enabled Connection-specific DNS Suffix Search List : fabrikam.com tailspintoys.com
And we can run the same command on the server to understand its IP address.
There is a TON of great info in this output, but the keys are going to be:
What is the network path between these two endpoints?
A network path is the path a packet will take when being transported between the client and server.
If the client and server are on different subnets one or more routers will be needed to route the traffic between subnets.
To determine what the network path looks like you can use tracert.
Here is an example of a network path between a client and server where the machines are in two different subnets:
PS C:\> tracert dc01.contoso.com Tracing route to dc01.contoso.com [192.168.1.10] over a maximum of 30 hops: // Here we can see the IP address of the router or gateway that was traversed 1 <1 ms <1 ms <1 ms 172.16.10.1 2 3 ms 1 ms <1 ms 192.168.1.10
This allows us to understand what points we may need to investigate. If they are in the same subnet, we can likely limit the scope of the issue to the two machines instead of any network devices along the path. And on the flip side, if there are lots of network devices along the path we may need to investigate based off where the last place we saw the traffic in question.
What port & protocol is communication taking place over?
Port and protocol are the most network-y part of these questions. The protocol tells us what we can expect from the conversation on the network. Are any rules that this traffic should be following? Where in the lifecycle of this traffic are we? How can we zero in on the specific conversation we care about?
For example, let’s say that I am having issues connecting to an external webserver.
We can see that we are waiting for iis.contoso.com. What happens if we try to remove one of the layers we are using?
Given that this is HTTP traffic we know:
We are going to use the PowerShell cmdlet Test-NetConnection to test TCP connectivity.
PS C:\> Test-NetConnection iis.contoso.com -Port 80 ComputerName : iis.contoso.com RemoteAddress : 192.168.1.24 RemotePort : 80 InterfaceAlias : Ethernet SourceAddress : 192.168.1.10 TcpTestSucceeded : True
The result of this command tells us a few things.
This means that if the Test-NetConnection succeeds, but the web browser fails then we have eliminated basic connectivity as the cause of the issue. It allows us to re-scope the issue to the web browser of the web server.
Collecting Network Traces
There are a few ways to collect and review network traces. For the sake of this series, we will be reviewing all network traces in WireShark as it is the gold standard of trace analysis.
As a quick summary, Wireshark is a packet capture and analysis tool created by the WireShark foundation. It is open source and cross-platform and my favorite tool for reviewing network traces. For more information, please visit Wireshark · Go Deep.
When collecting data, I always recommend collecting a simultaneous two-sided network trace while reproducing the issue.
Tracing on only one machine is like listening to only half of a phone call. You can kind of guess what the other person is saying but, I’d rather hear it directly from them to prevent any confusion.
Collecting captures in WireShark
Collecting captures in WireShark is very straightforward.
These captures can be saved and reviewed on other machines.
Alternatively, you can start a capture using dumpcap.exe (a tool shipped with Wireshark).
PS C:\Program Files\Wireshark> .\dumpcap.exe -w Example.pcapng Capturing on 'Ethernet' File: Example.pcapng Packets captured: 32 Packets received/dropped on interface 'Ethernet': 32/0 (pcap:0/dumpcap:0/flushed:0/ps_ifdrop:0) (100.0%)
This created pcapng file can then be opened in WireShark.
Collecting captures using netsh
Starting in Windows 7 SP2, the operating system added functionality to collect packet captures with the inbox tool netsh.
The tool is very powerful and provides lots of configuration options, but the simplest configuration uses the following parameters.
PS C:\> netsh trace start capture=yes tracefile=C:\netsh.etl Trace configuration: ------------------------------------------------------------------- Status: Running Trace File: C:\netsh.etl Append: Off Circular: On Max Size: 250 MB Report: Off PS C:\> netsh trace stop Merging traces ... done Generating data collection ... done The trace file and additional troubleshooting information have been compiled as "C:\netsh.cab". File location = C:\netsh.etl Tracing session was successfully stopped.
There are two important things to keep in mind:
I’m sure you are itching to dig into captures but let’s remember there is more than looking at network traces. As a reminder:
Starting in the next post we will dig into reviewing these traces and what you can expect to see in different failure scenarios. Until then have fun poking around with network captures, I’ll catch you later.
Previous posts in the series
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.