There have been some blogs on VSS backups in the past. Over last few months, we have worked on several cases where troubleshooting VSS was required. Seeing that the documentation around how to troubleshoot this is somewhat lacking, we decided to put together this blog post. This post will give you VSS troubleshooting tips using publicly available tools. Introduction Exchange-aware VSS backups are supported for both the active and passive storage groups and databases. The passive copy backup solution we provide is VSS-only, and it is implemented by the Exchange Replica VSS Writer that is part of the Replication service. Streaming backups are only supported from the active storage groups. You cannot use streaming backup APIs to back up the replica database. A VSS backup program must be used with a VSS Requestor for the Exchange writer. With Exchange Server 2007, it is possible to run two separate VSS backup jobs against the same Exchange server. With Streaming Backups in Exchange Server 2003, while it was possible to run two backups at the same time versus two different Storage Groups, it was not possible with VSS. With VSS in Exchange Server 2003 it was not possible to back up SG2 until the backup completed on SG1. The Exchange 2007 Writer also allows you to restore Exchange data to an alternative location. The Exchange 2003 Writer did not allow retargeting VSS restores to a location other than the original path. VSS backups cannot be restored to the replica location using the Exchange Writer, but they can be restored to an alternate location which could the replica copy folder directory. In Exchange 2007 the store VSS writer allows for restores to ANY storage group, including RSG. It also allows for restores of databases to a folder location (not associated with an SG), where Jet is then used to replay logs into the database to bring it to a consistent and mountable state. Then it can be ported anywhere. Troubleshooting Volume Shadow Copy Service The Volume Shadow Copy Service is installed by default on a Windows Server 2003 server. It is set to start manually and will run if a requester (Backup program) can use the writers in the system. Windows Server 2003 Service Pack 2 addresses some critical problems affecting the performance of VSS. KB 914962 provides a list of all the updates on the service pack. You can then install just the updates for VSS. However, it is recommended that the full service pack be installed. An update rollup package 940349 is available to resolve some Volume Shadow Copy Service (VSS) snapshot issues in Windows Server 2003. Additionally, depending on the symptoms, there are many hotfixes/articles that are available to correct top VSS issues. Some of them are listed below:
- http://support.microsoft.com/kb/833167
- http://support.microsoft.com/kb/915331
- http://support.microsoft.com/kb/930800
- http://support.microsoft.com/?id=910250
- http://support.microsoft.com/kb/923628
- http://support.microsoft.com/kb/933779
The output of this command should provide all writers registered with VSS. Check the State of the Writers. If the state is anything other than Stable, it might be a cause of concern; however, it does not necessarily have to be a cause of backup failure. If the Exchange writer is not in stable state (in "Retryable error" state as shown below) then we have to restart the information store service on the exchange server to get the exchange server in stable state.
Upon restarting the information store service, you should see "No error" as shown below.
vssadmin list providers - This command lists the registered volume shadow copy providers. It should ideally be "Microsoft Software Shadow Copy provider 1.0". You can change it through the registry, if you need to. Here is the screenshot obtained by running the command on a working server:
By default, we would see only the Microsoft Software Shadow Copy Provider. However, some Backup software use their own providers and we may see an additional provider if such Backup software is installed.
A good result on both commands is extremely necessary. For more information on the use of VSSADMIN, try running vssadmin /? on the command prompt for a list of other tests available.
In case you find that the output of the vssadmin list writers command is blank, there might be registry corruption that is preventing the system from picking up and displaying the correct information. This is documented on KB 940184. It explains how you can recreate the list of VSS writers
If we still have problems, we can enable VSS tracing. To do so, refer the below article http://support.microsoft.com/?id=887013
Make sure you have installed the latest version of the backup client.
BETEST
We can run the BETest tool to take VSS backup of the Exchange storage group. BETest is a test requestor that can be used to test the Exchange VSS writer. Betest can perform most of the operations that a VSS requestor will perform. With Betest you can take a VSS Snapshot of your active and replica Databases on an Exchange 2007 Server.
BETest can be obtained by downloading and installing the Volume Shadow Copy Service SDK 7.2 from
http://www.microsoft.com/downloads/details.aspx?FamilyID=0b4f56e4-0ccc-4626-826a-ed2c4c95c871&displa...
The default location for the i386 version of BETest is:
C:\Program Files(x86)\Microsoft\VSSSDK72\TestApps\betest\obj\i386
Note: There is also an AMD64 compiled version available. Always change to the directory of your operating system version before running BETest.
1. Once the tool is installed, we need to verify which writers are available, by running the following command:
Betest.exe >AvailableWriters.txt
2. In addition to the BETest.exe file you will need to create a Components.txt file which will contain information that BETest requires.
Components selection file format:
"<WriterId>": "<component-logical-path>" {"target" # "new target", ...}, ..."<component-logical-path>" : '"<subcomponent-logical-path>,...";
- Open the AvailableWriters file using Notepad. Go to Edit | Find, type: Microsoft Exchange Writer
For Exchange Server 2007, you can use the following cmdlet
Get-StorageGroup <SG name> |fl Guid
4. One thing to keep in mind is that streaming backups are only supported from the active storage groups. If you want to backup your passive copy the only solution is to run a VSS backup. If this is CCR cluster or if you are using LCR and want to backup the replica copy, the path will look similar to the following:
For CCR Replica Copy:
"{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}":"Microsoft Exchange Server\Microsoft Information Store\Replica\<CMS Name>\<GUID of Storage Group>
For LCR Replica Copy:
"{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}":"Microsoft Exchange Server\Microsoft Information Store\Replica\<Server Name>\<GUID of Storage Group>
NOTE: The success or failure of a streaming backup is not indicative of the Exchange Writer. A streaming backup might be a valid workaround for failing VSS backups, but its success or failure is not indicative of the health of the Exchange Writer.
- After the file Components.txt is created, we can start doing the backup using BETest by running the following command:
In this example, the backup will be created under the C:\Betest directory.
If this backup fails with an error, there is a problem with the Exchange writer. In order to troubleshoot, we will need the Output.txt file for us to review the errors.
You can also run the following command without the /E switch.
BETEST.exe /B /T 1 /S output.XML /C components.txt /D c:\betest > Output1.txt
- There is also a newer version of BETEST available with the Windows SDK. Instead of being text-based, the components.txt is XML-based.