Howdy everyone! Today’s post will be a bit shorter but will cover how we collect network traces and what information we need to make use of the data we collect.
Asking Questions
I know it can be tempting to spin up WireShark and jump right into looking at traces, but asking questions is just as important, if not more important than the traces themselves.
I usually like to group these questions into two groups: technical and general.
Note: I will be using the terms client and server to refer to the sender and receiver. The client is always the sender, the server is always the receiver.
Technical Questions:
- Who is talking?
- What is the client’s IP address?
- What is the server’s IP address?
- How are we getting there?
- What is the network path between these two endpoints?
- What language are they speaking?
- What protocol is the traffic using and over what port?
It may seem like we are getting really into the networking weeds with these questions, but they are primarily about context. By clarifying who is talking we can cut out noise, clarifying how we get there helps us avoid rabbit holes, clarifying the language helps us understand what the rules are for their conversation.
General:
- Does this always happen or just sometimes?
- Has this ever worked?
- Does this work with a different source or destination?
By combining these two sets of questions we can start understanding which locations are our primary suspects for the cause of the issue.
For example:
- Q: Does this work with a different source or destination?
- A: Yes! It works with the old server
What this tells us is that it is less likely that the issue is when the packet is leaving the sender.
- Q: What is the network path between these two endpoints?
- A: These devices are in the same subnet. (Think back to the post office metaphor from part 0)
This tells us that it is less likely that the issue occurs along the network path.
Given this new information, we would want to start by looking on the destination to see if anything could be interrupting the communication.
Running tests
Another benefit of asking the questions above is that it allows us to understand how to perform specific tests or what information we need to flesh out ourselves.
For the purposes of this article, all testing is done via Windows PowerShell.
What is the clients IP address / What is the servers IP address?
Finding the relevant IP addresses is essential and we can quickly determine this information by running ipconfig /all
Here is an example output from a client:
PS C:\> ipconfig /all
Windows IP Configuration
Host Name . . . . . . . . . . . . : Client01
Primary Dns Suffix . . . . . . . : contoso.com
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : contoso.com
tailspin.com
fabrikam.com
Ethernet adapter Ethernet 2:
Connection-specific DNS Suffix . : fabrikam.com
Description . . . . . . . . . . . : Microsoft Hyper-V Network Adapter
Physical Address. . . . . . . . . : 00-15-5D-CA-40-0E
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::d5cf:204:2294:daeb%8(Preferred)
IPv4 Address. . . . . . . . . . . : 192.168.1.22(Preferred) // This is the client IPv4 address
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Lease Obtained. . . . . . . . . . : Thursday, May 12, 2022 11:35:18 PM
Lease Expires . . . . . . . . . . : Friday, May 13, 2022 9:30:55 AM
Default Gateway . . . . . . . . . : fe80::215:5dff:feca:4009%8
192.168.1.1
DHCP Server . . . . . . . . . . . : 192.168.1.1
DHCPv6 IAID . . . . . . . . . . . : 117445981
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-29-49-9F-7D-00-15-5D-CA-40-0E
DNS Servers . . . . . . . . . . . : 192.168.1.10 // This is the client primary DNS server
NetBIOS over Tcpip. . . . . . . . : Enabled
Connection-specific DNS Suffix Search List :
fabrikam.com
tailspintoys.com
And we can run the same command on the server to understand its IP address.
There is a TON of great info in this output, but the keys are going to be:
- IPv4 Address
- DNS Server
- Default Gateway
What is the network path between these two endpoints?
A network path is the path a packet will take when being transported between the client and server.
If the client and server are on different subnets one or more routers will be needed to route the traffic between subnets.
To determine what the network path looks like you can use tracert.
Here is an example of a network path between a client and server where the machines are in two different subnets:
PS C:\> tracert dc01.contoso.com
Tracing route to dc01.contoso.com [192.168.1.10]
over a maximum of 30 hops:
// Here we can see the IP address of the router or gateway that was traversed
1 <1 ms <1 ms <1 ms 172.16.10.1
2 3 ms 1 ms <1 ms 192.168.1.10
This allows us to understand what points we may need to investigate. If they are in the same subnet, we can likely limit the scope of the issue to the two machines instead of any network devices along the path. And on the flip side, if there are lots of network devices along the path we may need to investigate based off where the last place we saw the traffic in question.
What port & protocol is communication taking place over?
Port and protocol are the most network-y part of these questions. The protocol tells us what we can expect from the conversation on the network. Are any rules that this traffic should be following? Where in the lifecycle of this traffic are we? How can we zero in on the specific conversation we care about?
For example, let’s say that I am having issues connecting to an external webserver.
We can see that we are waiting for iis.contoso.com. What happens if we try to remove one of the layers we are using?
Given that this is HTTP traffic we know:
- HTTP sits on top of TCP
- HTTP by default will use port 80
We are going to use the PowerShell cmdlet Test-NetConnection to test TCP connectivity.
PS C:\> Test-NetConnection iis.contoso.com -Port 80
ComputerName : iis.contoso.com
RemoteAddress : 192.168.1.24
RemotePort : 80
InterfaceAlias : Ethernet
SourceAddress : 192.168.1.10
TcpTestSucceeded : True
The result of this command tells us a few things.
- We were able to resolve the name iis.contoso.com
- We were able to successfully able to establish a TCP connection to 192.168.1.24 on port 80
This means that if the Test-NetConnection succeeds, but the web browser fails then we have eliminated basic connectivity as the cause of the issue. It allows us to re-scope the issue to the web browser of the web server.
Collecting Network Traces
There are a few ways to collect and review network traces. For the sake of this series, we will be reviewing all network traces in WireShark as it is the gold standard of trace analysis.
As a quick summary, Wireshark is a packet capture and analysis tool created by the WireShark foundation. It is open source and cross-platform and my favorite tool for reviewing network traces. For more information, please visit Wireshark · Go Deep.
When collecting data, I always recommend collecting a simultaneous two-sided network trace while reproducing the issue.
Tracing on only one machine is like listening to only half of a phone call. You can kind of guess what the other person is saying but, I’d rather hear it directly from them to prevent any confusion.
Collecting captures in WireShark
Collecting captures in WireShark is very straightforward.
- Install WireShark
- Launch WireShark
- Select the capture icon
These captures can be saved and reviewed on other machines.
Alternatively, you can start a capture using dumpcap.exe (a tool shipped with Wireshark).
Example:
PS C:\Program Files\Wireshark> .\dumpcap.exe -w Example.pcapng
Capturing on 'Ethernet'
File: Example.pcapng
Packets captured: 32
Packets received/dropped on interface 'Ethernet': 32/0 (pcap:0/dumpcap:0/flushed:0/ps_ifdrop:0) (100.0%)
This created pcapng file can then be opened in WireShark.
Collecting captures using netsh
Starting in Windows 7 SP2, the operating system added functionality to collect packet captures with the inbox tool netsh.
The tool is very powerful and provides lots of configuration options, but the simplest configuration uses the following parameters.
PS C:\> netsh trace start capture=yes tracefile=C:\netsh.etl
Trace configuration:
-------------------------------------------------------------------
Status: Running
Trace File: C:\netsh.etl
Append: Off
Circular: On
Max Size: 250 MB
Report: Off
PS C:\> netsh trace stop
Merging traces ... done
Generating data collection ... done
The trace file and additional troubleshooting information have been compiled as "C:\netsh.cab".
File location = C:\netsh.etl
Tracing session was successfully stopped.
There are two important things to keep in mind:
- The netsh command will be in the etl format and will need to be converted to pcapng using etl2pcapng
- More details on etl2pcapng: Converting ETL Files to PCAP Files - Microsoft Tech Community
- The netsh command also collects a cab file with a handful of wonderfully insightful data
- IPconfig data
- Operating system versions
- etc…
Conclusion
I’m sure you are itching to dig into captures but let’s remember there is more than looking at network traces. As a reminder:
- Ask questions, ask questions, ask questions.
- Run your tests for additional context.
- Collect your traces.
Starting in the next post we will dig into reviewing these traces and what you can expect to see in different failure scenarios. Until then have fun poking around with network captures, I’ll catch you later.
Previous posts in the series