Summary
In order to clean up Routing Engine’s Cache, we no longer have to recommend customers’ to shut down all their Exchange Servers simultaneously in the entire org. This was required to force concurrent destruction of Org object that lives inside reapi.dll, by forcing unload of reapi.dll from all machines in the organization at the same time. This was extremely painful for the customers, & other alternatives (e.g., un-registering xlsasink.dll were complex & resource intensive, & may not work in some situations). With this hotfix / tool release, we have a much easier solution now.
Microsoft has created a tool / hotfix which will fix stale routes problems ( [Routing groups not found in DS] or object_not_found_in_DS ) without requiring that all servers need to be down at the same time & then rebooted. To read more about this problem, please check this KB article. This blog post includes detailed documentation of REMonitor tool, its injection mode capabilities, and its usage. It is recommended that this tool should be run under the guidance and supervision of Support Services, especially under injection mode. However, because we have sent this tool to customers in the past, we wanted to provide documentation on it.
REMonitor tool has two main functions. First, it can monitor the change in routing packet in a given Exchange server. Second, it can inject custom built routing packet into a given Exchange server to reduce foot print of a deleted routing group. The tool can work with both Exchange Server 2000 and Exchange Server 2003 when proper access rights are granted under monitoring or injecting mode.
Hotfix / Tool details
The tool has packet injection capability. This would allow us to clear out the connector and server container of deleted RG entry to eliminate the possibility of routing to deleted connector as well as reducing the size of the packet drastically.
This version of REMonitor will work with E2000 & E2003, as the hotfix enable REMonitor to do authentication with RESVC:691 in order to monitor. The tool would examine all RG entries, identifies the deleted ones (by querying DS), and inject a version that clears out connector/server container. This injected version cannot be overridden because its major/minor version is increased by 1. Also, the injected version will have deleted key word in its RG address triplet. Therefore, the tool wouldn't inject against already inject-processed entries.
Where to obtain REMonitor.exe
You can get REMonitor tool by calling Microsoft Support Services.
Where to install REMonitor.exe
Because of dependency issues, you must copy REMonitor.exe to your exchange bin directory (i.e. exchsrvr\bin).
Working under monitoring mode in Exchange Server 2000
It is straightforward to monitor against an Exchange Server 2000. You do not need to run under account with special access right because there is no access checks for debug request in Exchange Server 2000. By simply typing “REMonitor”, you will see detailed parameter usage for both monitoring and injecting mode. By default, the tool displays the monitoring result to screen.
Working under monitoring mode in Exchange Server 2003
Monitoring under Exchange Server 2003 requires running the tool under account with ADS_RIGHT_ACTRL_DS_LIST right. This is the result of our security push to prevent unauthorized information disclosure. Usually, the easiest choice is domain administrator.
Working under injecting mode in Exchange Server 2003
Note: The steps below show how to launch the REMonitor "directly" from an Exchange Server 2003, and then inject packet against a remote E2000 Server or an E2003 server.
Injecting packets accomplishes two objectives. First, it reduces the size of deleted routing group entries by injecting a modified version without server and connector entries. These two containers constitute the bulk of deleted entry size. Second, zombie connector entries in deleted routing group may cause misroute or delay in delivering under some rare circumstances. The tool completely eliminates that possibility. In addition, the modified version of deleted routing group will increase the major version by one (or minor version if under 5.5 RG) to prevent any possible further update. And finally, the tool will modify the RG address triplet to include “deleted” key word. As a result, the tool can detect already modified entries and will ignore them in future injections.
Injecting packets requires account with Send_As rights for both Exchange Server 2000 and 2003. The obvious choice is to run under local system account. You need to follow the following steps to create a command window under such account:
1. In any command window, type “at 5:55 /interactive “cmd.exe”. This will add a new task to your server’s scheduled task list.
2. Open up “Scheduled Task” window and locate the new task you just created. Right click on the task and select “Run”. This will create a new command window under local system account.
3. Go to exchsrvr\bin to run REMonitor for injection.
Because Exchange Routing Service uses mutual authentication and runs under local system account (same account as the tool), it is not possible to inject against the same server where you run the tool from.
As a final important point, you should run injection only when sufficient time has passed since you deleted your last routing group. This will allow the deletion to be replicated to all DCs to prevent false positives. In case the tool deletes a legitimate routing group, you can recover by simply recycling the routing service of routing master server. Please note that we have tested this unlikely scenario in our test pass.
Working under injecting mode in Exchange Server 2000
Note: The steps below show how to launch the REMonitor "directly" from an Exchange 2000 Server, and then inject packet against a remote E2000 Server or an E2003 server.
REMonitor can also be launched from an E2000 server. In order to run REMonitor from an Exchange 2000 server, the following steps must be followed. Please note, these steps ONLY apply to running REMonitor on an E2000 server!
1. Create a separate dir (e.g. REMon2003) and copy all the required binaries along with REMonitor.exe to this dir. The required binaries are Exchmem.dll, Gwart.dll, reapi.dll, & REMonitor.exe. We require these dlls as these are the dlls whose export signature has changed in E2003. You should get these binaries along with the tool from Support Services.
2. Add the bin folder to the OS path. To add x:\program files\exchsrvr\bin to the OS path, please go to control panel-> system -> advanced-> environment variable and locate “PATH”. Add x:\program files\exchsrvr\bin to the current path. You may have to reboot the server to see this change. This is needed because REMonitor depends on dsaccess.dll and only one copy could run in the server (security restriction).
3. To run the tool now on E2000 server for injecting packets, follow the steps to run under local system account as mentioned in the section above titled “Working under injecting mode in Exchange Server 2003” However, beware of one important exception in Step 3 - REMonitor now MUST be run from the new separate Dir and not from Exchsrvr/bin folder. Tool can now inject packets to either a remote E2000 server or a remote E2003 server.
If the above steps are not followed, When tool is launched directly from an E2000 server, the tool returns a popup error complaining about a call to function ExchMHeapFreeEx in exchmem.dll, which only exists in Exchange 2003.
Injecting example step by step
The following is a typical Winroute Snapshot of a deleted routing group. In our example, it only has two deleted servers and connectors. Roughly, its size is 1200 bytes.
Then you run the tool under injection mode. Please note that the tool detects one deleted routing group entry out of total of 2 (one healthy).
The following is the snapshot of injected entry. Please note the following
1. Both server and connector containers are empty now.
2. RG address triplet is altered with “deleted” key word.
3. Major version of deleted entry has increased by one (from 3 to 4).
The final processed entry has a fixed size around 280 bytes. In our example, the deleted entry size is reduced by 77%. The reduction rate will be higher when there are more servers and connectors in the deleted routing group, which is likely the case for our customers.
Finally, if you run the injection again, the tool will detect previously processed entries and skip them. If injection is not needed (i.e. no deleted routing group is detected), the tool will exist without doing anything. (See below)
Frequently Asked Questions
Question: Do we need to run the tool on each server in the org individually? Or running it on just one server will take care of the problem on all the Servers in the org?
Answer: Injection will be propagated by routing. So you only need to run against one server whose routing service is running in normal condition.
Question: Will the tool \ hotfix stop the future occurrence of [Routing groups not found in DS] whenever an existing RG is deleted at some point in future (i.e. after running the tools once)?
Answer: If you delete a RG after running the tool, you have to run it again in order empty the newly deleted item. But since running this tool is extremely simple, it’s really not a big deal compared to the old method of shutting down all servers at the same time.
Question: Will the tool clean up the Winroute view completely?
Answer: Tool will keep the [Routing groups not found in DS] node in Winroute. Display change in Winroute will require a separate fix on Winroute part. In order to completely clear the view in Winroute, we still have to shut down ALL their Exchange 2000 /2003 Servers in their org simultaneously and then reboot them. The tool can not completely erase deleted RG from routing packet and there won't be a solution other than organization-wise reboot that can accomplish that goal. The foot print of injection processed deleted routing group entry will be around 280 bytes. The size may be a few bytes more if you have long versions. But generally, you should consider it having a fixed size. After injection is processed foot print is reduced to such insignificant size that rebooting all servers are no longer necessary.
Question: Will the tool work just for E2003 Servers?
Answer: This tool will work with for both E2000 & E2003 Servers.
Question: Can REMonitor inject packets on the local Server (i.e. where it's running from)?
Answer: Since Exchange Routing Service uses mutual authentication and runs under local system account (same account as the tool), it is not possible to inject against the same server where you run the tool from. This is by design. When you inject data, the tool runs under localSystem account. The RESVC also runs under localSystem account. You can't do mutual authentication against yourself. That is why you can not inject against the same server where you run the tool. If you have to, you need to run the tool under an account with send-As right.
Question: How can I find where the “Scheduled tasks” are?
Answer: Start menu > All Programs > Accessories > System Tools > Scheduled tasks
Question: How to use the tool?
Answer: If you go to exchsrvr\bin folder, and type REMonitor, you will see the REMonitor usage as follows:
E:\Program Files\Exchsrvr\bin>REMonitor
Usage under monitor mode: REMonitor -m [server] [duration] [interval] [version]
[server] -- Netbios name of the target server to run the tool against.
[duration] -- Monitoring duration in minutes. The value is ignored under injection mode.
[interval] -- Polling interval in seconds. The value is ignored under injection mode.
[version] -- Either Ex2000 or Ex2003. The value is ignored under injection mode.
Usage under inject mode: REMonitor -i [server]
[server] -- Netbios name of the target server to run the injection against.
Question: Can REMonitor inject packets between RGs (i.e. source Server running REMonitor is in one RG and target server is in another RG)?
Answer: Yes, as long as there is no firewall blocking port 691. If firewall blocks port 691 or connection cannot be made to port 691 for other reasons, then it will not work, and you can have bigger issues with Routing.
Question: Should REMonitor injection feature work to / from one cluster node to another or from one cluster node to / from another standalone Exchange Server?
Answer: It doesn’t matter.
Question: Should REMonitor injection feature works through a Terminal Server / remote Control session?
Answer: If REMonitor is launched through terminal server or remote control session, the localsystem account probably isn't going to be used (depending on the configuration of the Terminal Server / remote control software), and as such it will not work. With Windows 2003 or XP you will have to use the /console switch with RDP client to get an interactive session through TS. Win2k does not have this functionality.
REMonitor is an exceptional tool to help correct stale linkstate issue, but it won't work in a Terminal Services session because when you use the AT command to bring up cmd.exe it opens on the console, right? Well not quite. If you start the Terminal Services session in console mode, you are at the console. This feature is new to Windows Server 2003 and Windows XP Professional. Once you have started the Terminal Services session in console mode, you are free to start the REMonitor procedures.
So how do you do start the Terminal Services session in console mode?
Use one of the following command line switches to invoke console mode:
mstsc /console
mstsc -console
Help for the command line switches for the RDP client is available by entering
mstsc /? from the Run dialog (select the Start button, then Run).
For more information on running Terminal Services in console mode, please refer to these KB articles:
- Hao Zhang and Mohammad Nadeem