First published on TechNet on Aug 18, 2014
Hi folks, Lakshman Hariharan here again with the first of what my peers and I intend to be a new series of posts introducing how to read network traces using Microsoft Message Analyzer (henceforth referred to as MA) to go along with our Real World Example Series of posts, which can be found here , here and here in reverse chronological order.
We have found that network trace analysis happens to be one of the key skills required to troubleshoot many issues we see in the field on a day to day basis, thus the reason for starting this series of posts.
I intend to follow the general outline for this particular post.
1. How to capture a network trace on a Windows machine.
2. TCP concepts explained using network traces
3. Various TCP flags, session setup, teardown explained using a network trace
This post is intended to be quite basic, introducing the concepts that we will build upon in subsequent posts. Concepts that you may have most likely come across in any Networking 101 class you have had the chance to take, but using practical and visual examples to illustrate those concepts. Once we use this post to lay the foundations, so to speak, we will then use subsequent posts to discuss various other protocols and their behavior as observed on the wire. Protocols such as HTTP (with various forms of authentication such as Basic, Forms Based, and Integrated Windows), Server Message Block (SMB), Authentication (NTLM, Kerberos). We can then move on to (somewhat) more complex and involved scenarios such as a domain logon, Kerberos constrained delegation and such.
Let’s get started!
How to capture a network trace on a Windows machine
As many of you reading this are probably aware of, since the release Windows Server 2008 the ability to capture a network trace on a machine has been included within the netsh.exe utility. So, by simply executing a netsh command with specific parameters, one can capture a trace without having to install any tools or network capture utilities. The screenshot below, shows the netsh.exe command that can be used to capture a trace.
The first screenshot shows the netsh command used to start the trace and the second screenshot shows the command used to stop the trace. So, the sequence of events would be
1. Start the network trace by executing the command in the first screenshot.*
2. Reproduce the problem or behavior you are trying to analyze.
3. Stop the trace by executing another netsh.exe command
Note the ping.exe commands right after starting the trace and right before the command to stop the trace. These ping.exe commands are what we like to call “breadcrumbs”, ala Hansel of Hansel and Gretel, to find our way through the trace. Although not necessary in a trace that is not too big, in a potentially large trace, to focus on a general area it is good to leave such breadcrumbs so you can focus your analysis efforts on the frames in between the ICMP packets the ping.exe commands generate on the wire.
This results in an Event Trace Log (ETL) file that can be opened in MA.
Before moving to the next topic in this post, TCP and UDP basics, I would like to share another cool feature. The ability (since Windows Server 2008 R2 and Windows 7) to be able to capture a complete trace of the boot process.
If you have ever tried to capture a trace of a machine booting, it was quite a challenge with having to either mirror the switch port that the machine being started up was connected to, or by chaining a hub to the switch and capturing all the traffic.
But on Windows Server 2008 R2 (or Windows 7 and above in the case of a client OS) by adding the “persistent=yes” parameter to the netsh command we executed previously will instruct netsh to continue capturing even across a reboot. Once the machine is restarted executing the netsh command to stop the trace will create the ETL file that included the entire startup (boot and logon) process.
This is illustrated in the screenshots below.
Now that we have shown how to start and stop a network trace without the use of any other tools but a built-in command line utility, let’s proceed to topic 2.
UDP and TCP Basics
Anyone that has ever taken a Networking 101 lesson has probably heard of the description of Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) as connection oriented and connectionless respectively. It basically means that TCP establishes a connection (session) and after establishing the session starts the process of transmitting and receiving data between the two endpoints in the conversation. TCP also ensures by the use of flags known as Acknowledgements (Acks) that packets intended for the other endpoint have indeed made it to the other endpoint. Thus TCP is connection oriented.
UDP, on the other hand doesn’t require setup of sessions and such. It transmits the data and is pretty much done. It doesn’t use any mechanisms to keep track of whether the packets it transmitted have made it to the other endpoint successfully or not. Thus UDP is connectionless.
Now let us observe that on the wire.
UDP is pretty straightforward since it is connectionless. Let us observe the behavior of TCP, specifically session setup, the various flags and session teardown parts.
Test TCP Connectivity To A Specific Port Using Telnet.exe
To start learning the basics of TCP we will use a trace captured while connecting to a TCP port on a server. The easiest way to test TCP connectivity to a destination port is to use the Telnet.exe** client.
Once you know the destination port a service or application listens on, you can use the telnet.exe <port number> command to test connectivity. Some of the most common ports are 80 (HTTP), 445(SMB), 3389(RDP) and so forth.
In this case we will test connectivity to the port used by RDP (3389) using telnet.exe. The sequence of events is the following:
a. Launch a command window as Administrator
b. Start the trace by executing the netsh.exe command, as shown in the first screenshot, highlighted in red.
c. Use Telnet.exe to connect to port 3389 on the server, as shown in the first screenshot highlighted in green.
d. Once connection is established successfully you will receive a blank window with a cursor as seen in the second screenshot
e. Terminate the connection by using the “Ctrl+]” combination keystrokes as shown in the third screenshot, highlighted in red.
f. Stop the trace by executing the netsh.exe trace stop command.
Open Trace Using Message Analyzer
Now let us start analyzing the trace using Message Analyzer, which can be downloaded here .
Once MA is opened we will navigate to the File menu and click on Quick Open, browse to the directory where the file is stored, select the file and click Open.
This is illustrated in the screenshots below.
Applying a Basic Filter To See TCP Traffic
Once the trace is open in a new session we will apply a filter to only view TCP traffic since that is what we are interested in. To do so, simply type TCP in the Filter window and click on “Apply Filter” or hold the Ctrl key and hit the Enter key. This is shown in the screenshot below and highlighted in Red.
Once the filter has been applied, we are viewing a sum total of six packets, the ones we are interested in, that include the session setup and teardown pieces. Frames 132 through 134 show the session setup and frames 161 through 163 show the teardown.
A cursory glance at the Summary filed shows the various TCP flags, highlighted in Red (Syn, Syn/Ack, A, Ack/Fin, Ack/Reset) we will explain further in the rest of this post.
Analyzing TCP Session Setup
Let’s start by exploring the session setup piece first.
The first three frames in the screenshot above with the Syn , Syn/Ack and Ack flags depicted as S, A..S, and A respectively is what is collectively referred to as the TCP three way handshake , as most folks reading this are probably aware. Every TCP session is always established with the three way handshake.
Let’s now analyze the first part of the three way handshake, the initial Syn packet. For that, we will select frame 132 in the Summary pane and expand the Flags field in the Details pane, as shown in the screenshot below.
The only flag that is set on this frame is the Syn flag, from all the possible flags that can be set but aren’t. A description of all TCP flags and what they mean can be found here.
Another item to observe in the screenshot above, highlighted in green are the source and destination ports. The client uses a port from a range of ports called ephemeral ports (the range being 49152-65535 in Windows Vista and above) to connect to the port the application is listening on (in this case 3389).
Here is an intuitive way to find the beginning of all TCP sessions in a trace. We know that every TCP session begins with the Syn flag, so we can build a filter that says TCP.Flags.Syn (highlighted in Red) and apply that filter. This will result in the following result, as shown in the screenshot below.
Note that this results in displaying ALL packets with the Syn flag set, but not the packets with ONLY the Syn flag set, as highlighted in green in the screenshot above.
To display just the frames with the Syn Flag set we could build intuitively, another condition to the filter and add “AND NOT(tcp.flags.Ack)”. This will result in the screenshot below.
I realize this is rather cumbersome but I wanted to quickly demonstrate how to build a filter intuitively and on the fly.
Sequence And Acknowledgement Numbers
Let us further explore the three way handshake. For that let us focus on a different set of fields in frame 132, the Sequence and Acknowledgement number fields.
When we started explaining TCP basics we had mentioned that TCP is connection oriented. Which means that it has to have a mechanism to keep track of what packets have made their way to the other endpoint or not. TCP does that via the use of Acknowledgements (or Acks) and thus the use of Sequence and Acknowledgement numbers.
When a client sets out to start a TCP session, it sends its own Sequence number in the SequenceNumber field, which is a 32 bit number and uses an Acknowledgement number of 0 in the AcknowledgementNumber field. The number 0 in the AcknowledgementNumber field indicating that it is the beginning of the TCP session.
So, alternative to the cumbersome filter that I demonstrated in the previous step, if you wanted to find the beginning of all TCP sessions in a trace you could just use the filter TCP.AcknowledgementNumber==0
That will result in the second screenshot below, again the filter highlighted in red and the flags highlighted in green. The screenshot used below is from a trace that has more than one TCP session going, as you can tell.
So returning to the analysis of the telnet.exe trace we captured earlier. Let us note the Sequence number of 465092260 and Acknowledgement number of 0 in frame 132 and move on now to frame 133, which is part two of the three way handshake.
Notice that the server acknowledges the receipt of the original Syn frame sent in frame 132 by setting the Syn and the Ack flag (as depicted as A..S and highlighted in red), sends its own Sequence number (of 2393426318), increments the Sequence number it received in frame 132 by a value of 1 (to 465092261) and uses it in its own AcknowledgementNumber field, as highlighted in green in the screenshot below.
The last part of the three way handshake is frame 134. The client in turn acknowledges the receipt of frame 133 from the server by setting the Ack flag (highlighted in red in the screenshot below) and uses the Sequence number (of 465092261, received as the AcknowledgemenetNumber field in frame 133) and increments the Sequence number it received in frame 132 by a value of 1 (to 2393426319, highlighted in green) for the server (or the other endpoint) to use as its Sequence number in the next frame.
At this point the TCP session has been established and the application can use the session to transfer data between the two endpoints.
While all this may sound like too much detail to be bothered with, these Sequence and Acknowledgement numbers are crucial to how TCP keeps track of what packets have made their way to the other endpoint, and are thus crucial to understanding TCP.
In subsequent posts we will describe what happens after the session is setup and before it is torn down, as in the application behavior HTTP, SMB etc. but for the rest of this post let us concentrate on the session teardown.
Session Teardown
For that, we will focus on frames 161 through 163. A TCP session can be torn down several different ways depending on the application. Regardless of which way a session is torn down, the TCP flags used to teardown a session invariably includes either the Fin (depicted as F in the screenshot below) or the Reset (depicted with the A.R in the screenshot below) flag.
So in this case, the application telnet.exe chose to use a three way close. Once I executed the keystrokes of “Ctrl + ]” in the telnet session, the application gave the go-ahead to TCP to tear down the session. The client indicated its desire to teardown the session with the Fin (depicted as A…F in the screenshot above), also acknowledging the receipt of the previous packet (thus the A flag in A..F)
The server in turn acknowledges the client’s packet and the client then proceeds to abruptly terminate the session with the Reset (depicted as A.R..) in the screenshot above, also acknowledging the receipt of the previous pact from the server (thus the A in A.R..)
I use the word abruptly because of the use of the Reset flag. A four way session termination would look something like depicted in the screenshot below, taken from another trace.
The client indicates its desire to terminate the session with the Fin flag in frame 30
The server acknowledges the client’s packet in frame 31
In the subsequent packet the server also indicates its desire to terminate the session in frame 32
In frame 33 the client acknowledges the previous packet and the session teardown is complete.
In the next post we will briefly discuss what happens when the client attempts to connect to a server on a port it is not listening on. In the interest of brevity, we will also save for the next post the UDP behavior on the wire as well.
Stay tuned for part 2 of this series, coming soon.
Thank you. Lakshman Hariharan
*To execute the command successfully one would have to launch the command window as an administrator.
**On Windows Vista and above clients the Telnet.exe client is not installed as part of the default installation of the operating system and would have to be installed under Control Panel - Programs and Features - Turn Windows features on or off - Telnet Client.