~ Chris Butcher | Senior Support Escalation Engineer
Troubleshooting issues encountered when configuring SharePoint protection in Data Protection M...
In this installment we’ll cover the backup process and some of the more common issues you may see when backing up SharePoint. This is broken down into three areas:
- Backing up SharePoint Configuration and content databases from SQL
- Creating SharePoint metadata
- Creating the SharePoint catalog
Similar to part 1, this scenario assumes that DPM 2012 is installed, storage is provisioned, agents have been deployed and protection has been enabled SharePoint.
SharePoint backup
SharePoint backups will touch the SharePoint Web Front End (WFE) server as well as the SQL server that hosts the SharePoint Configuration/Content databases. This means that both servers need to be accessible for SharePoint backup to complete successfully.
The screenshot below is from Monitoring/Jobs after a successful recovery point. Note that Configuration and Content database jobs were executed on the SQL server and SharePoint Farm\SharePoint_Config ran on the SharePoint server
SharePoint backup is a three steps process and each is examined below.
Step One: Backup SharePoint Configuration/Content Databases from SQL ServerThis step doesn’t require access to SharePoint WFE. It is simply SQL Database Protection and will just use the SQL writer to back up the content databases (as well as SharePoint_Config and SharePoint_AdminContent_*).
If you encounter errors in this area, it should be looked at on the SQL server where the content databases are located. The basic approach to troubleshooting is as follows:
1. Repro the failure
2. Gather sys and app logs from the SQL server computer
3. Gather the most recent *.errlog files from the SQL server computer
4. Gather the most recent *.errlog files from the DPM server
5. Troubleshoot the problem as a standard SQL backup/protection issue
In the screenshot above, the metadata generation occurs in the SharePoint Farm\SPSQL\SharePoint_Config job and is the last job to be completed in a Recovery Point. During metadata enumeration, the SharePoint WFE server connects to the SQL server. If the SharePoint WFE server cannot communicate to the SQL server, the end result will be “Backup metadata enumeration failed” as shown below.
You will also see an error similar to the following:
Error text: DPM could not resolve the SQL alias <serverName> on the SharePoint front-end web server. (ID 31250 Details: Unknown error (0x80131534))
OK, so we’re talking about SharePoint metadata, but what is that exactly? Good question, and I’ll explain it this way: So there is one command that we are very familiar with which is VSSADMIN LIST WRITERS . We run this command to check if a writer is available in a system as well as to get the state of that writer (stable, failed, etc). However, VSSADMIN LIST WRITERS doesn’t tell us everything we need to know about writers. If we need more details, we need to use other tools. In Windows Server 2008 or newer, the tool we use is DISKSHADOW.EXE which is built into Windows. For Windows Server 2003 we use VSHADOW.EXE .
When you launch DISKSHADOW.EXE and type in LIST WRITER , you are requesting every single writer to show its metadata information. The metadata information returned by those writers is what they will be backing up if there is a backup request for that writer. Since we are talking about SharePoint, the metadata information from a SharePoint VSS Writer contains the following:
- The Configuration Database
- Content databases
- Application databases
With SharePoint protection, DPM only cares about the Configuration Database and the Content databases.
Below is a snippet of a SharePoint metadata. You can use the following to pipe the output into a text file to be reviewed and searched:
C:\> DISKSHADOW /l writer.txt
If you run DISKSHADOW> LIST WRITER then you should see output similar to the snippet I have below.
* WRITER "SharePoint Services Writer"
- Writer ID = {da452614-4858-5e53-a512-38aab25c61ad}
- Writer instance ID = {d36db7cf-c2b4-4977-a18e-a642987f9c85}
- Supports restore events = TRUE
- Writer restore conditions = VSS_WRE_ALWAYS
- Restore method = VSS_RME_RESTORE_AT_REBOOT_IF_CANNOT_REPLACE
- Requires reboot after restore = FALSE
- Excluded files:
+ Component "SharePoint Services Writer:\SPSQL\SharePoint_Config"
- Name: SharePoint_Config
- Logical path: SPSQL
- Full path: \SPSQL\SharePoint_Config
- Caption: Configuration Database SharePoint_Config
- Type: VSS_CT_DATABASE [1]
- Is selectable: TRUE
- Is top level: TRUE
- Notify on backup complete: FALSE
- Paths affected by this component:
- Volumes affected by this component:
- Component Dependencies:
- Dependency to "{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}:\\WILSON-SQL\WILSON-SQL\SharePoint_Config"
+ Component "SharePoint Services Writer:\SPSQL\WSS_Content"
- Name: WSS_Content
- Logical path: SPSQL
- Full path: \SPSQL\WSS_Content
- Caption: Content Database WSS_Content
- Type: VSS_CT_DATABASE [1]
- Is selectable: TRUE
- Is top level: TRUE
- Notify on backup complete: FALSE
- Paths affected by this component:
- Volumes affected by this component:
- Component Dependencies:
- Dependency to "{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}:\\WILSON-SQL\WILSON-SQL\WSS_Content"
+ Component "SharePoint Services Writer:\SPSQL\SharePoint_AdminContent_abda792e-1e17-4119-99b1-34f6fc61a6c1"
- Name: SharePoint_AdminContent_abda792e-1e17-4119-99b1-34f6fc61a6c1
- Logical path: SPSQL
- Full path: \SPSQL\SharePoint_AdminContent_abda792e-1e17-4119-99b1-34f6fc61a6c1
- Caption: Content Database SharePoint_AdminContent_abda792e-1e17-4119-99b1-34f6fc61a6c1
- Type: VSS_CT_DATABASE [1]
- Is selectable: TRUE
- Is top level: TRUE
- Notify on backup complete: FALSE
- Paths affected by this component:
- Volumes affected by this component:
- Component Dependencies:
Highlighted is what DPM uses to figure what each database returned by the writer is used for. And by the way, DPM won’t populate the DPMDB with the output above. Instead, it will generate an XML file based on that output and insert it into DPMDB via the stored procedure dbo.prc_RM_RecoverySource_UpdateWithPreBackupDetails .
You can retrieve the metadata information from the database (if one was created) by running the SQL query below:
from tbl_RM_RecoverySource where WriterId = 'DA452614-4858-5E53-A512-38AAB25C61AD' order by BackupTime
where WriterID = 'DA452614-4858-5E53-A512-38AAB25C61AD' = SharePoint VSS Writer
So with all of that in mind, we can conclude that if SharePoint metadata is what the SharePoint VSS Writer is able to return, a failed metadata means that the SharePoint VSS Writer wasn’t able to return anything. Below are some common reasons why that may occur.
- The SharePoint VSS Writer is in failed state.
- The SharePoint VSS Writer isn’t running.
- The SharePoint farm administrator don’t have SQL Permissions.
- The account used by configuresharepoint –enablesharepointprotection doesn’t have SQL permissions. The account used needs to be SharePoint Farm administrator and hold a sysadmin role on the SQL server that is hosting the SharePoint Farm databases.
- A SQL alias is used but the alias can’t be ‘translated’ to <SQLServerName>\<InstanceName>.
- A SQL alias can be translated to <SQLServerName>\<InstanceName> but SQL Browser on the SQL Server side can’t redirect that string to the port used by that SQL named instance.
- SharePoint 2010 has Search configured and the registry key below exists on the SharePoint WFE Server:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Agent\2.0\SharePointSearchEnumerationEnabled
The reason for the error is that SharePoint Search protection for SharePoint 2010 has been deprecated in DPM 2010 and later.
So now that we know what SharePoint Metadata is, the next question is where DPM will use it.
When we start a Recovery Point or Consistency Check, several individual jobs will kick in: One for each content database of that farm as well as the one for the metadata itself.
DPM relies on the metadata to know which content database it needs to run a backup job for.
NOTE If metadata fails and a new content database is added to the farm, this new content database won’t be added to SharePoint protection when the autoprotect job kicks in at midnight.
Metadata information can only be added – it cannot be removed from an existing job. This means that if a content database is removed from the farm, you can only remove that reference from DPMDB if you stop/restart protection for that farm.
Step Three: Create the SharePoint CatalogThis is what allows DPM to drill down through the SharePoint content database, site collection, sites, etc. for Item Level Recover (ILR). Things to note:
- The SharePoint catalog job is executed on the SharePoint WFE.
- The SharePoint VSS Writer isn’t needed for Catalog generation.
- A connection to the SQL server is required for the catalog to complete.
- The SharePoint catalog needs access to the MTATempStore$ share (C:\Program Files\Microsoft DPM\DPM\Temp\MTA) from the SharePoint WFE.
NOTE: If DPM cannot access MTATempStore$ you will see the following:
Type: SharePoint Catalog Task
Status: Completed
Description: The job completed successfully with the following warning:
File copy failed due to an I/O error.
Source location: \\wilson-sp.wsouza.local\MTATempStore$\{242f99e3-a60e-40f1-ab16-9ab7616ed87d}\2.rocatalogDesti...
location: c:\Program Files\Microsoft DPM\DPM\Temp\60423b5c-780f-4162-bde1-7ee980fbe093\2.rocatalog.
(The network name cannot be found.)
Exception trace :
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite)
at Microsoft.Internal.EnterpriseStorage.Dls.ArmCommon.FileCopyBlock.DoFileCopy(Object msgObject) (ID 30123)
End time: 5/15/2012 8:47:28 PM
Start time: 5/15/2012 8:47:24 PM
Time elapsed: 00:00:04
Data transferred: 0 MB (0 bytes)
Source details: wilson-sp.wsouza.local
Target details: wilson-sp.wsouza.local
Cluster node -
NOTE : The Catalog task will run 3 hours after a recovery point, however you can manually trigger a catalog task by running the PowerShell command Start-CreateCatalog
Start-CreateCatalog or a scheduled catalog job performs the following:
1. On the SharePoint Web Front End Server you will find a subfolders under the MTATempStore$ share (C:\Program Files\Microsoft Data Protection Manager\DPM\Temp\MTA).
The GUID above is the DatasourceID for the SharePoint farm. This folder will be created at the first time the SharePoint catalog task is executed.
Underneath the GUID folder shown above we have one or more of the following files:
Every file represents a SharePoint Configuration/content database. As we can see, we have three X.rocatalog files. So the question is “where did the numbers (2, 3 and 4) come from?” Every data source that is protected by DPM gets a Datasource sequence number. This sequence number can be found in table tbl_RM_RecoverableObject. Whatever sequence number the SharePoint database gets on that table will reflect on the rocatalog file name.
We can see this by running the following:
select DatasourceSequenceNumber, LogicalPath, ComponentName from tbl_RM_RecoverableObject order by RecoverableObjectId
That gives us the following:
DatasourceSequenceNumber LogicalPath ComponentName
-----------------------------------------------------------------
1 Sharepoint Farm SPSQL\SharePoint_Config
2 WILSON-SQL SharePoint_AdminContent_abda792e-1e17-4119-99b1-34f6fc61a6c1
3 WILSON-SQL WSS_Content
4 WILSON-SQL SharePoint_Config
If there was a change within a content database (like new site collection, new site, new item, removed site, removed item, etc), the rocatalog file will have that change in it. Here is a snippet from 3.rocatalog :
FullCatalog
SiteCollection*http://wilson-sp/sites/Blog/*1c29c4ec-a676-41c4-9737-9988378fd87a
Site*http://wilson-sp/sites/Blog/documents3/*43f6e062-b5c5-4f9e-a4ac-4a5ebdaa92f3
List*http://wilson-sp/sites/Blog/documents3/_catalogs/*6a368c11-20da-4630-8cff-5a7c03c92a15
List*http://wilson-sp/sites/Blog/documents3/_catalogs/masterpage/*3687946b-17ba-4918-966f-ffe447fcaa09
List*http://wilson-sp/sites/Blog/documents3/_catalogs/masterpage/Forms/*63077659-a5e3-4e45-90bf-0ee8f4db686e
List*http://wilson-sp/sites/Blog/documents3/_catalogs/masterpage/Forms/MasterPage/*b697225e-2d8b-4472-9512-d10982af5a3e
If there is no change, this is what will be within the file (snippet from SharePoint_Config)
NoCatalog
If the catalog ran was an incremental, the file should look like this:
IncrementalCatalog
ADD*ListItem*http://wilson-sp/sites/Blog/documents3/Shared Documents/sammui.log*57c11095-c0da-4011-a3a2-3ee786396f63
After the catalog job is complete, all rocatalog files will be removed. This is why it can be tricky to troubleshoot SharePoint catalog issues since we don’t necessarily have a file to look into. The only way we can see catalog content is via DPM verbose tracing.
1. Folder SPCatalogDump is created while the SharePoint catalog is being generated. This folder is created under C:\Program Files\Microsoft Data Protection Manager\DPM\Temp on the SharePoint WFE server.
Folder SPCatalogDump will only be used for Incremental catalogs, and for every full/incremental backup, a folder called SPXXXXXXXXXXXXXXXXXX will be created and left behind (its content will be deleted though).
When an incremental backup takes place and there is a change in it, two files will be created in that location.
The rocatalog file is similar to the one mentioned earlier, however it will only include the changed items
IncrementalCatalog
ADD*ListItem*http://wilson-sp/sites/Blog/documents3/Shared Documents/DownloadCenter.xml*c2a7341b-3d35-4d48-a32e-125cd4e485c9
The spdump file contains a map for the newly added SharePoint recoverable item (I added document DownloadCenter.xml to my SharePoint site)
I:1c29c4ec-a676-41c4-9737-9988378fd87a:43f6e062-b5c5-4f9e-a4ac-4a5ebdaa92f3:c2a7341b-3d35-4d48-a32e-125cd4e485c9:Add:1;0;9d9844a7-7a9b-419c-ab5d-16fae7b346d6;634728378915800000;916
A:c2a7341b-3d35-4d48-a32e-125cd4e485c9
Now even though this information looks like gibberish, it does make sense. Here is the breakdown:
'c2a7341b-3d35-4d48-a32e-125cd4e485c9')
Output:
NOTE: In this output, column ComponentType give us information about the SharePoint content. These are the Component Types in SharePoint:
SiteCollection
: This is the ‘root’ site
Site
: A site can be created underneath a site collection
List
: Is a path within a Site Collection/Site
ListItem
: A file within a Site Collection/Site
Other information we get from spdump is the Token . The Token is created by SharePoint on every change that occurs in it.
To identify these items in DPM you can go to Recovery tab and highlight a SharePoint farm. On the right hand side you will see the content databases
When you double click a content database, if there was no problem creating the catalog, and if there is at least one site collection defined, you will see something like this. In this case, content database WSS_Content has one site collection:
If you double-click a site collection you can see List , ListItem and Sites, if they exist. The yellow icon on the left means that this is a list or a ListItem . If the icon is four little colorful people then it means it is a site.
And if there are ListItems within Lists or Sites , you can double-click them to go down into the tree.
You can get that same information by running this T-SQL query:
Full SharePoint Catalog runs only once: The first time a Catalog is generated against a content database or after a content database is restored.
If you ever need to run a full SharePoint Catalog again, you can run the SQL query below.
OR AppId = 'DA452614-4858-5E53-A512-38AAB25C61AD'
NOTE: The T-SQL query above will reset SharePoint catalog for ALL SharePoint Farms protected by this DPM server. If you want to run it against a specific farm you will need to find the datasource for the desired farm.
There are a few different issues that you might run into after a scheduled backup/Catalog is completed.
- A content database that was being protected by DPM was removed from the farm. This is an easy one to fix; stop SharePoint protection (retaining data) and protect it again.
- Something is blocking metadata enumeration from occurring. If so refer to metadata troubleshooting mentioned previously.
- The SharePoint catalog job doesn’t complete, or it completes with warnings. When a catalog is not successfully completed, you might not be able to drill down through a content database to a site collection/site/item, etc. This doesn’t mean that the data is missing from the backup, it just means that you can’t see them from DPM UI.
In the screenshot below we can’t go further the content database. Double clicking any database on the right hand side won’t do a thing. If you have successful backups but failing catalogs and you need to restore an item. The solution is to restore the whole content database, attach to SQL, then use SharePoint tools to export/import the desired data.
That should cover the most common problems you’re likely to encounter when backing up SharePoint with Data Protection Manager. Next time we’ll take a look at some of the issues you might see when restoring SharePoint.
Chris Butcher | Senior Support Escalation Engineer | Microsoft GBS Management and Security Division
Wilson Souza | Senior Support Escalation Engineer | Microsoft GBS Management and Security Division
Get the latest System Center news on Facebook and Twitter :
System Center All Up: http://blogs.technet.com/b/systemcenter/
Configuration Manager Support Team blog:
http://blogs.technet.com/configurationmgr/
Data Protection Manager Team blog:
http://blogs.technet.com/dpm/
Orchestrator Support Team blog:
http://blogs.technet.com/b/orchestrator/
Operations Manager Team blog:
http://blogs.technet.com/momteam/
Service Manager Team blog:
http://blogs.technet.com/b/servicemanager
Virtual Machine Manager Team blog:
http://blogs.technet.com/scvmm
Microsoft Intune:
http://blogs.technet.com/b/microsoftintune/
WSUS Support Team blog:
http://blogs.technet.com/sus/
The RMS blog:
http://blogs.technet.com/b/rms/
App-V Team blog:
http://blogs.technet.com/appv/
MED-V Team blog:
http://blogs.technet.com/medv/
Server App-V Team blog:
http://blogs.technet.com/b/serverappv
The Surface Team blog:
http://blogs.technet.com/b/surface/
The Application Proxy blog:
http://blogs.technet.com/b/applicationproxyblog/
The Forefront Endpoint Protection blog :
http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog :
http://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog:
http://blogs.technet.com/b/isablog/
The Forefront UAG blog:
http://blogs.technet.com/b/edgeaccessblog/
DPM 2012 R2 SharePoint 2010 SharePoint 2012