Restores from Online backup fail to Recovery Storage Groups configured on USB storage

You may find that recovery operations from online backup fail when the target of a restore is a Recovery Storage Group that exists on an External USB storage device. Most commonly External USB storage is implemented as a temporary measure due to lack of physical disk space on the server. Thus, the storage is added, an RSG is configured and linked, and a restore is attempted. However, there are a number of "gotchas" you need to be made aware of to allow these operations to complete successfully. The "general inspiration" for this post was taken from a real case I worked within Customer Support Services. Primary Issue & Symptoms: As mentioned above, I recently worked with a customer who was attempting to restore an Exchange 2003 Mailbox Database to a Recovery Storage Group that had been configured on an External USB Hard-Drive. This was done because the available disk space on the production server was not sufficient for restoring the complete mailbox database from backup. Once restored, the customer was going to perform any number of additional recovery operations (single item recovery, complete mailbox, complete database, etc.). What we found was that the restore job would initialize successfully but the byte count never increased above 238 bytes. The restore operation would eventually timeout and fail. A review of the restore log contained the following information:

Job Completion Status Job Ended: <time stamp> Completed Status:  Failed Final Error:  0xa0008488 -- Access is Denied

Troubleshooting: At first glance the "Access is Denied" error would suggest a permissions related problem. I initially began the call with a review of the Recovery Storage Group's configuration. The configuration clearly showed that the linked mailbox store was being written to the USB drive, which was being provided to the system as the f:\ volume. I performed a compare and contrast of the permissions that had been written to the volume with the other drives and did not notice any specific NTFS permissions anomalies. I then proceeded to perform the following physical tests:

• Verified that I could traverse the f:\ via Windows Explorer as theuser performing the restore.
• Verified that I could manually create files and folders on the f:\ as this user.
• I tested building a new mailbox store on the f:\, built a single mailbox, sent a test message and then verified I could backup the mail store on the f:\.

All of these operations completed successfully. I then proceeded to add the "Test Mailbox Store" to the RSG as a linked database (I actually built "Test Mailbox Store" within the same production storage group that he was trying to restore), and attempted to restore it to the f:\ (e.g. the USB Storage Device). This failed with the identical errors as before. I then reviewed the application log which contained the following Error:

Event Type:  Error Event Source: ESE Backup Event ID: 950 Description:  Unexpected file system error 53 encountered while opening the restore environment file.

This event clearly shows that the nature of the failure was during the creation of the Exchange Restore Environment file (restore.env). The System Error Code of 53 equates to "ERROR_BAD_NETPATH" or more commonly referred to as "the network path was not found". I decided to try testing whether or not my "Test Mailbox Store" could be restored to a physical (non-USB device). I did this by removing the RSG from the USB device, recreating it, and testing the restore. The result was that the restore was successful. Troubleshooting Conclusions: 1. The fact that the "Test Mailbox Store" could be restored to an RSG that did not exist on USB storage, immediately made the USB drive suspect. 2. The fact that the restore could complete successfully suggested no problems with the backup/restore agent or with the files actually being restored. So outside of the f:\ simply being a USB device, what made it unique? Backup and Restore Agents: When any valid Backup/Restore agent (e.g. one that abides to the rules and methods of the APIs responsible for Exchange aware backup/restore), actually restores an Exchange Database (regardless of whether or not it resides within a Recovery Storage Group or a production directory), the Extensible Storage Engine needs to be able to physical map the path specified as the Temporary Location for the restore. It does this via a UNC path that includes within the path the Admin Share for the target drive. \\Server Name\<AdminShare of Drive>\Temp Directory Specified For Log Files Example: \\MyServer\f$\MyRestore So to verify that the shares were configured properly, I had the customer open a command line and enter: NET SHARE NET SHARE will list all configured shares on a system. In my case, a review of the output clearly showed that the default Admin Share had not been created for the f:\ yet was present for all other drives. This explained why a restore to a separate physical disk worked in my previous troubleshooting step. The fact that share was not present also provided a logically sound reason for why System Error 53 was being documented in the 950 Event. Once, I had made this observation, I initially tried to create the share "manually". To do so, I shared the f:\ volume as f$ and manually applied all the permissions via a compare and contrast of a properly initialized Admin Share. Once done, I re-ran NET SHARE. Although the share now showed as present, the share itself did have a few noticeable differences (e.g. was not a Default Admin share), etc. (which in this instance would be expected as the share itself had not been actually created by the SERVER service). I proceeded to retry a restore, which failed with the identical errors. Troubleshooting Conclusions 2: The fact that the Default Admin share for the f:\ was missing suggested the following:

• An Administrator had manually removed the Admin Share (seemed very unlikely).
• The USB drive was not fully initialized when theSERVER service had completed startup (seemed very likely).
• "Manually" attempting to create the share has no net effect on providing aresolution to the issue.

As previously noted, Admin Shares (drive$) are created during startup when the SERVER service fully initializes. My suspicion was that when the SERVER service created the shares the USB device itself had not been fully initialized by the Disk Management System. Thus, no Admin Share would have been configured for the f:\ because it would not have been technically valid at the time of assignment by the SERVER service. I tested this hypothesis on my own system and witnessed the identical behavior. If you simply attach an external USB drive you will notice that a Default Admin Share is not created because the SERVER service is already started. Taking this a step further, I restarted the SERVER service on my own workstation then re-ran NET SHARE once the service was back online. The result was that a Default Share had been created for the USB drive on my system. The next logical step was to restart the SERVER service on the Exchange Server. There are two primary Exchange Services that maintain dependencies on the SERVER service (as well as the Computer Browser service which is also dependent on the SERVER service):

• The Exchange Information Store service (MSExchangeIS) 
• The Exchange System Attendant service (MSExchangeSA)

Prior to restarting the SERVER service, I had the customer manually dismount all production stores which was done to force all pending writes into the store. Once done, I manually stopped the Information Store and System Attendant services, then restarted the SERVER service: From command line (cmd): net stop srv net start srv With the SERVER service back online, I proceeded to bring the Computer Browser, Information Store and System Attendant services back online. Once done, I re-ran NET SHARE to verify the Default Admin share had been created successfully: NET SHARE Share name   Resource                        Remark ---------------------------------------------------------- F$           F: \ Default share We then rebuilt our Recovery Storage Group, linked the original database back to the RSG on the f:\, placed our Database, Log and TEMP paths onto the f:\ and restored successfully! Conclusion: It would appear that when External USB Storage is used, Windows does not automatically create the Admin Shares for the USB drive.  As previously discussed, Admin Shares are created during system startup when the SERVER service fully initializes.  My "professional opinion" (I am after all an Exchange Specialist not a Windows specialist), is that most likely this occurs because the External Storage isn't fully initialized by the Disk Manager System when the SERVER service has completed startup.  This is why a "reboot" of the server wouldn't correct the issue (it would simply recreate the issue).  By restarting the SERVER service with the USB Drive fully initialized, you effectively force the shares to be re-evaluated and recreated. So what is important to remember here is that if you ever have to perform a similar operation (restoring an Exchange Database to an External Drive), you need to ensure that the Admin Share for the Drive is configured properly prior to attempting the restore (as Exchange needs to directly map to the UNC path specified via the Admin Share). The Admin Share itself is not needed for a backup, but it should go without saying that no production Exchange Database should ever by physically located on a USB drive.  Otherwise, you and I will probably be working on an Exchange Performance related issue in the near to distant future (hello RPC pop-up). Hope you find this interesting. Happy Trails. - Eric Norberg