First published on TechNet on Jan 22, 2009
Hi all, Mark from Directory Services again. This time I would like to talk about one of the many tools that we use in troubleshooting network issues. At times you may see errors such as The RPC server is unavailable or There are no more endpoints available from the endpoint mapper (These error messages can be DNS related at times). Of course you may see an error code such as 1722 or 1753. So you ask, “What does the error code mean?” If you get an error such as 1722 or 1753, open a command prompt and type in the following:
Net helpmsg <error number>
An example would be net helpmsg 1722 and after hitting enter you will get The RPC server is unavailable. If the error number is something such as a hex number then they will not resolve using the net helpmsg command.
Before we get into talking about using portqry we need to understand a little bit more about RPC (Remote Procedure Call). When a server starts up applications and services that are going to be listening for other clients connecting, they may register with the RPC Endpoint Mapper (EPM). EPM will keep a database of all these registrations and what ports they are listening on. Ports are used so that applications can communicate between client and server. By default EPM will listen on port 135. When a client needs to know what port an application is listening on it queries RPC to find out. This is much like going to the hospital to see a sick friend. The address of the hospital would be the IP address of the server. When you go in the door to the information desk to find out what room your friend is in would be the same as querying RPC on port 135. Then once you’re in the room with your friend you now connected with the application you wanted to talk to. Some ports such as LDAP (389), SMB (445), etc. are hard coded so they are consistent across all Windows clients and servers. To see a list of common ports go to KB article here . Applications may register a port on what is referred to as ephemeral ports. An ephemeral port is a port above 1024 and less than 65536. Take LSASS (Local Security Authority Subsystem Service) for an example, it will usually listen on a couple of ports just above 1024 such as 1025, 1026 or 1027. An example of the output is below:
Note: I am querying port 135 against a Domain Controller in my examples.
UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_ip_tcp:192.168.0.30[1025]
The UUID stands for Universally Unique Identifier, no 2 applications should ever have the same UUID, and is followed by an annotated name if one exists. On the second line is the protocol the application uses and the IP address of the server. Then in the brackets will be the port number the application is listening on. The example above was taken from the LSASS service but you can see multiple entries for a single application as well. Such as this:
UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_np:\\server1[\PIPE\lsass]
Now note that the UUID’s are the same, the annotated name is the same if one I present, but the protocol is different and now we have the server name path as this is the named pipe instance. So how do we query the RPC to find out what applications are registered? Glad you asked! First we need to download the portqry command line tool from here or the GUI tool from here . I prefer using the command line tool and piping the results to a text file. Once you have the tool downloaded and installed open a command prompt. Let’s say you have issues connecting to another server (call it Server1) across a WAN where there is a firewall in place. At the command prompt run the following command:
Portqry –n server1 –e 135 > port135.txt & port135.txt
This will query the RPC EPM service (-e switch) on server1 (-n switch) and pipe the results to a text file. The use of the”&” and referring to the text file again will open the text file after the command is done. Now look at the text file. The first portion is simply taking the server name that you are querying against and resolving to an IP address:
Querying target system called:
Server1
Attempting to resolve name to IP address...
Name resolved to 192.168.0.30
The next line is very critical:
TCP port 135 (epmap service): LISTENING
Querying Endpoint Mapper Database...
Server's response:
If something other than LISTENING is returned then there could be a problem with a port being blocked somewhere. If the port shows to be FILTERED then a firewall or VLAN could be blocking that port, if the port returns NOT LISTENING then we got to the machine but the machine is not listening on that port number. If you get a LISTENING or FILTERED response, check and see whether we are checking TCP or UDP, most likely it was attempting to use UDP and this would be a normal response as UDP is connectionless. An example of this would be if you query port 88 for Kerberos against a DC and use the following syntax:
Portqry –n server1 –e 88 –p both
Querying target system called:
Server1
Attempting to resolve name to IP address...
Name resolved to 192.168.0.30
TCP port 88 (kerberos service): LISTENING
UDP port 88 (kerberos service): LISTENING or FILTERED
By default we will only query the port on TCP. By using the –p switch we can tell the portqry tool which protocol we want to use. Using the both after the –p we can tell the utility to query both TCP and UDP. In the above example we use UDP by default for Kerberos so that is why I wanted to check both protocols.
Now back to querying the endpoint mapper. Let’s say you have two Domain Controllers with a firewall separating them and they are not replicating. You believe the networking admin has all the ports open on the firewall. From one DC query the other DC on port 135 and look at the output. Search the text file for lsass then note the UUID. Scroll up or down in the text file until you find the port number in brackets such as this, [1025]. Now run the portqry again using this:
Portqry –n server1 –e 1025
Does it return LISTENING ? If so then good, if it shows FILTERED then the port is blocked on the firewall. Keep searching for lsass as you will find it will listen on more than one port, usually 2 different ones. Once you find the second port run the command again using the second port number and verify it is listening as well. You can also search the text file of FRS (File Replication Service) and query its port. If you query port 389 (LDAP) then you will get a stream of information from the DC such as below:
currentdate: 12/16/2008 23:37:23 (unadjusted GMT)
subschemaSubentry: CN=Aggregate,CN=Schema,CN=Configuration,DC=domain,DC=com
dsServiceName: CN=NTDS Settings,CN=server1,CN=Servers,CN=Corp,CN=Sites,CN=Configuration,DC=domain,DC=domnamingContexts: DC=domain,DC=com
defaultNamingContext: DC=domain,DC=com
schemaNamingContext: CN=Schema,CN=Configuration,DC=domain,DC=com
configurationNamingContext: CN=Configuration,DC=domain,DC=com
rootDomainNamingContext: DC=domain,DC=com
supportedControl: 1.2.840.113556.1.4.319
supportedLDAPVersion: 3
supportedLDAPPolicies: MaxPoolThreads
highestCommittedUSN: 1541417
supportedSASLMechanisms: GSSAPI
dnsHostName: server1.domain.com
ldapServiceName: domain.com:server1$@DOMAIN.COM
serverName: CN=SERVER1,CN=Servers,CN=Corp,CN=Sites,CN=Configuration,DC=domain,DC=com
supportedCapabilities: 1.2.840.113556.1.4.800
isSynchronized: TRUE
isGlobalCatalogReady: TRUE
domainFunctionality: 2
forestFunctionality: 0
domainControllerFunctionality: 2
You can use the –p both switch with this command and it will query both UDP and TCP and it will return the same information each time. Note the UDP will show up as LISTENING or FILTERED as discussed before. If you want to query multiple ports you can use a switch such as this: -o 88,389,445 All you have to do is separate the port numbers with a comma then it will run it against all the ports you specify one right after another. If you want to do a range of ports say between 1100 and 1105 use –r 1100:1105 .
If you prefer to use a GUI based tool instead of the command line download the GUI version from the link above or here . Extract the files then run the portqueryui.exe to open the tool. Enter the IP address or server name in the first field then in the Query Type you will notice you have a drop down box for Service to query . This is useful as these ports will be fined in the config.xml file that is located in the same directory. This is useful to gather information against multiple ports that are commonly queried and outputs the results to a window you can simply copy and paste into a text file for review later. It also allows you to manually enter the port numbers that you want to query on and the protocols.
You can use portqry to troubleshoot several different issues. I had a case where when checking the NTFS permissions on a folder it would take up to 20 seconds to resolve the SID to a name. In a network trace we saw the traffic going from the server to a DC. In the trace we queried port 135 to find out what ports LSASS was listening on. It returned 2 different ports. In the trace we could see that the server would send a SYN packet to the first port and we would never get a response from the server. It kept trying to use that port for almost 20 seconds then it would fail over to the second port on the list and resolve the SID to a name. We used portqry using the first port number and showed the customer that port was being filtered on the VLAN which he later confirmed was true. Another time we used portqry to confirm a problem with a trust between 2 domains which is common scenario for us.
Another note is that you cannot run portqry on a server back to itself. To do this run the following command on the server itself:
Netstat –ano > netstat.txt & netstat.txt
This will list out the protocols (TCP and UDP) the local address, foreign address (this would be a remote server that has a connection to your server) it’s state then the PID of the service that is listening locally. Here is a sample:
Active Connections
Proto Local Address Foreign Address State PID
TCP 0.0.0.0:53 0.0.0.0:0 LISTENING 1424
TCP 0.0.0.0:88 0.0.0.0:0 LISTENING 400
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 748
TCP 0.0.0.0:389 0.0.0.0:0 LISTENING 400
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4
TCP 0.0.0.0:464 0.0.0.0:0 LISTENING 400
TCP 0.0.0.0:593 0.0.0.0:0 LISTENING 748
TCP 0.0.0.0:636 0.0.0.0:0 LISTENING 400
TCP 0.0.0.0:1025 0.0.0.0:0 LISTENING 400
TCP 0.0.0.0:1027 0.0.0.0:0 LISTENING 400
To find what process is listening open Task Manager, go to View – Select Columns – put a check mark in the box for PID then ok. You can click on the PID column to sort it by numerical order to look up the number easier. As in my example above the port 53, which is DNS, 1424 resolves to DNS.exe, 400 is lsass.exe, 748 is svchost.exe (one of many svchost.exe’s), etc.
I hope I have not confused anyone too much and hope this helps in any troubleshooting network issues. Thanks!
References:
816103 HOW TO: Use Portqry to Troubleshoot Active Directory Connectivity Issues
http://support.microsoft.com/default.aspx?scid=kb;EN-US;816103
Troubleshooting networks without Netmon (Ned Pyle has some good information here as well)
http://blogs.technet.com/askds/archive/2007/12/18/troubleshooting-networks-without-netmon.aspx
Hi all, Mark from Directory Services again. This time I would like to talk about one of the many tools that we use in troubleshooting network issues. At times you may see errors such as The RPC server is unavailable or There are no more endpoints available from the endpoint mapper (These error messages can be DNS related at times). Of course you may see an error code such as 1722 or 1753. So you ask, “What does the error code mean?” If you get an error such as 1722 or 1753, open a command prompt and type in the following:
Net helpmsg <error number>
An example would be net helpmsg 1722 and after hitting enter you will get The RPC server is unavailable. If the error number is something such as a hex number then they will not resolve using the net helpmsg command.
Before we get into talking about using portqry we need to understand a little bit more about RPC (Remote Procedure Call). When a server starts up applications and services that are going to be listening for other clients connecting, they may register with the RPC Endpoint Mapper (EPM). EPM will keep a database of all these registrations and what ports they are listening on. Ports are used so that applications can communicate between client and server. By default EPM will listen on port 135. When a client needs to know what port an application is listening on it queries RPC to find out. This is much like going to the hospital to see a sick friend. The address of the hospital would be the IP address of the server. When you go in the door to the information desk to find out what room your friend is in would be the same as querying RPC on port 135. Then once you’re in the room with your friend you now connected with the application you wanted to talk to. Some ports such as LDAP (389), SMB (445), etc. are hard coded so they are consistent across all Windows clients and servers. To see a list of common ports go to KB article here . Applications may register a port on what is referred to as ephemeral ports. An ephemeral port is a port above 1024 and less than 65536. Take LSASS (Local Security Authority Subsystem Service) for an example, it will usually listen on a couple of ports just above 1024 such as 1025, 1026 or 1027. An example of the output is below:
Note: I am querying port 135 against a Domain Controller in my examples.
UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_ip_tcp:192.168.0.30[1025]
The UUID stands for Universally Unique Identifier, no 2 applications should ever have the same UUID, and is followed by an annotated name if one exists. On the second line is the protocol the application uses and the IP address of the server. Then in the brackets will be the port number the application is listening on. The example above was taken from the LSASS service but you can see multiple entries for a single application as well. Such as this:
UUID: 12345778-1234-abcd-ef00-0123456789ac
ncacn_np:\\server1[\PIPE\lsass]
Now note that the UUID’s are the same, the annotated name is the same if one I present, but the protocol is different and now we have the server name path as this is the named pipe instance. So how do we query the RPC to find out what applications are registered? Glad you asked! First we need to download the portqry command line tool from here or the GUI tool from here . I prefer using the command line tool and piping the results to a text file. Once you have the tool downloaded and installed open a command prompt. Let’s say you have issues connecting to another server (call it Server1) across a WAN where there is a firewall in place. At the command prompt run the following command:
Portqry –n server1 –e 135 > port135.txt & port135.txt
This will query the RPC EPM service (-e switch) on server1 (-n switch) and pipe the results to a text file. The use of the”&” and referring to the text file again will open the text file after the command is done. Now look at the text file. The first portion is simply taking the server name that you are querying against and resolving to an IP address:
Querying target system called:
Server1
Attempting to resolve name to IP address...
Name resolved to 192.168.0.30
The next line is very critical:
TCP port 135 (epmap service): LISTENING
Querying Endpoint Mapper Database...
Server's response:
If something other than LISTENING is returned then there could be a problem with a port being blocked somewhere. If the port shows to be FILTERED then a firewall or VLAN could be blocking that port, if the port returns NOT LISTENING then we got to the machine but the machine is not listening on that port number. If you get a LISTENING or FILTERED response, check and see whether we are checking TCP or UDP, most likely it was attempting to use UDP and this would be a normal response as UDP is connectionless. An example of this would be if you query port 88 for Kerberos against a DC and use the following syntax:
Portqry –n server1 –e 88 –p both
Querying target system called:
Server1
Attempting to resolve name to IP address...
Name resolved to 192.168.0.30
TCP port 88 (kerberos service): LISTENING
UDP port 88 (kerberos service): LISTENING or FILTERED
By default we will only query the port on TCP. By using the –p switch we can tell the portqry tool which protocol we want to use. Using the both after the –p we can tell the utility to query both TCP and UDP. In the above example we use UDP by default for Kerberos so that is why I wanted to check both protocols.
Now back to querying the endpoint mapper. Let’s say you have two Domain Controllers with a firewall separating them and they are not replicating. You believe the networking admin has all the ports open on the firewall. From one DC query the other DC on port 135 and look at the output. Search the text file for lsass then note the UUID. Scroll up or down in the text file until you find the port number in brackets such as this, [1025]. Now run the portqry again using this:
Portqry –n server1 –e 1025
Does it return LISTENING ? If so then good, if it shows FILTERED then the port is blocked on the firewall. Keep searching for lsass as you will find it will listen on more than one port, usually 2 different ones. Once you find the second port run the command again using the second port number and verify it is listening as well. You can also search the text file of FRS (File Replication Service) and query its port. If you query port 389 (LDAP) then you will get a stream of information from the DC such as below:
currentdate: 12/16/2008 23:37:23 (unadjusted GMT)
subschemaSubentry: CN=Aggregate,CN=Schema,CN=Configuration,DC=domain,DC=com
dsServiceName: CN=NTDS Settings,CN=server1,CN=Servers,CN=Corp,CN=Sites,CN=Configuration,DC=domain,DC=domnamingContexts: DC=domain,DC=com
defaultNamingContext: DC=domain,DC=com
schemaNamingContext: CN=Schema,CN=Configuration,DC=domain,DC=com
configurationNamingContext: CN=Configuration,DC=domain,DC=com
rootDomainNamingContext: DC=domain,DC=com
supportedControl: 1.2.840.113556.1.4.319
supportedLDAPVersion: 3
supportedLDAPPolicies: MaxPoolThreads
highestCommittedUSN: 1541417
supportedSASLMechanisms: GSSAPI
dnsHostName: server1.domain.com
ldapServiceName: domain.com:server1$@DOMAIN.COM
serverName: CN=SERVER1,CN=Servers,CN=Corp,CN=Sites,CN=Configuration,DC=domain,DC=com
supportedCapabilities: 1.2.840.113556.1.4.800
isSynchronized: TRUE
isGlobalCatalogReady: TRUE
domainFunctionality: 2
forestFunctionality: 0
domainControllerFunctionality: 2
You can use the –p both switch with this command and it will query both UDP and TCP and it will return the same information each time. Note the UDP will show up as LISTENING or FILTERED as discussed before. If you want to query multiple ports you can use a switch such as this: -o 88,389,445 All you have to do is separate the port numbers with a comma then it will run it against all the ports you specify one right after another. If you want to do a range of ports say between 1100 and 1105 use –r 1100:1105 .
If you prefer to use a GUI based tool instead of the command line download the GUI version from the link above or here . Extract the files then run the portqueryui.exe to open the tool. Enter the IP address or server name in the first field then in the Query Type you will notice you have a drop down box for Service to query . This is useful as these ports will be fined in the config.xml file that is located in the same directory. This is useful to gather information against multiple ports that are commonly queried and outputs the results to a window you can simply copy and paste into a text file for review later. It also allows you to manually enter the port numbers that you want to query on and the protocols.
You can use portqry to troubleshoot several different issues. I had a case where when checking the NTFS permissions on a folder it would take up to 20 seconds to resolve the SID to a name. In a network trace we saw the traffic going from the server to a DC. In the trace we queried port 135 to find out what ports LSASS was listening on. It returned 2 different ports. In the trace we could see that the server would send a SYN packet to the first port and we would never get a response from the server. It kept trying to use that port for almost 20 seconds then it would fail over to the second port on the list and resolve the SID to a name. We used portqry using the first port number and showed the customer that port was being filtered on the VLAN which he later confirmed was true. Another time we used portqry to confirm a problem with a trust between 2 domains which is common scenario for us.
Another note is that you cannot run portqry on a server back to itself. To do this run the following command on the server itself:
Netstat –ano > netstat.txt & netstat.txt
This will list out the protocols (TCP and UDP) the local address, foreign address (this would be a remote server that has a connection to your server) it’s state then the PID of the service that is listening locally. Here is a sample:
Active Connections
Proto Local Address Foreign Address State PID
TCP 0.0.0.0:53 0.0.0.0:0 LISTENING 1424
TCP 0.0.0.0:88 0.0.0.0:0 LISTENING 400
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 748
TCP 0.0.0.0:389 0.0.0.0:0 LISTENING 400
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4
TCP 0.0.0.0:464 0.0.0.0:0 LISTENING 400
TCP 0.0.0.0:593 0.0.0.0:0 LISTENING 748
TCP 0.0.0.0:636 0.0.0.0:0 LISTENING 400
TCP 0.0.0.0:1025 0.0.0.0:0 LISTENING 400
TCP 0.0.0.0:1027 0.0.0.0:0 LISTENING 400
To find what process is listening open Task Manager, go to View – Select Columns – put a check mark in the box for PID then ok. You can click on the PID column to sort it by numerical order to look up the number easier. As in my example above the port 53, which is DNS, 1424 resolves to DNS.exe, 400 is lsass.exe, 748 is svchost.exe (one of many svchost.exe’s), etc.
I hope I have not confused anyone too much and hope this helps in any troubleshooting network issues. Thanks!
References:
816103 HOW TO: Use Portqry to Troubleshoot Active Directory Connectivity Issues
http://support.microsoft.com/default.aspx?scid=kb;EN-US;816103
Troubleshooting networks without Netmon (Ned Pyle has some good information here as well)
http://blogs.technet.com/askds/archive/2007/12/18/troubleshooting-networks-without-netmon.aspx
- Mark Ramey
Updated Nov 09, 2023
Version 3.0
NedPyle
Microsoft
Microsoft