Hello, Shane Brasher here again. This article picks up from where DPM Certificate Troubleshooting–Part 1: General Troubleshooting left off. We are going to jump right in and look at a few failed scenarios when your DPM certificate related registry keys are missing or corrupt.
When you have everything setup and working, the certs are in the right store and the proper command syntax has been used, there are specific registry entries placed on both the DPM server and the member server being protected. In this next section we will go over what errors you can expect to see when either the certificates or the registry keys are missing.
Theme
: “Certs check, registry check, DPMCPWrapperService restart check. Rinse. Repeat.”
This troubleshooting theme should be strictly adhered to during your certificate based authentication troubleshooting. In the scenarios below, after I would remove a reg key or cert, in order to get things back into a working state I would have to repeat those steps. This is so important that it warrants repeating. When troubleshooting DPM certificate based authentication:
a.)Check the registry keys on both the DPM server and protected server.
b.) Check the certificate in use
c.) Restart the DPM CPWrapper service.
Registry Entries
We will look at the following scenarios:
DPM Server
Missing DPM cert reg key (it’s own key)
Missing Member cert reg key
Member Server
Missing DPM cert reg key
Missing Member server cert reg key (it’s own key)
We will note the following:
Error in the DPM gui
Error in the DPM alerts event log
Error in the MSDPMCurr.errlog
Errors in the DPMRACurr.errlog
Errors in the DPM CPWrapper log
After running both the SetDPMserver command on the member server or the Attach-ProductionServerWithCertificate command on the DPM server, registry entries are placed on the servers to associate the certificate with the DPM server and the protected server.
The default location is HKLM\Software\Microsoft\MicrosoftTDataProtectionManager\Agent\2.0\Certificates\<DPMServerName> or <ProtectedServerName>
DPM SERVER Registry Keys
DPM Server side registry keys
DPM Server Missing DPM cert reg key (its’ own reg key)
In this example we will look at the errors in the:
a.) DPM management tab
b.) DPM Alerts Event log
c.) MSDPMCurr.errlog
After the Set-DPMCredentail command is run, if the registry key
on the DPM Server for the DPM server
itself is missing or deleted for some reason then you can expect the following error in the DPM GUI:
Reg Key:
HKLM\Software\Microsoft\MicrosoftTDataProtectionManager\Agent\2.0\Certificates\<
DPMServerName
>
DPM Management Agent Status
****************************
Although this error suggests to check the CPWrapper service on the member server, which is not a bad idea, the issue in this case is with the DPM server itself. Remember this error was produced by removing the DPM certificate registry key.
Usually the 33304 indicates an issue with the DPM CPWrapper Service. In this case the related registry keys that bind to that service. (There is a list of the common causes for this error discussed in Part 3 of this series.) If this DPM registry key is missing then you may also see the DPM CPWrapper service in a “starting” state if not then a restart of this service may fail when attempting to bind the service with that missing registry key. You would also see a crash log generated in the following directory: C:\Program Files\Microsoft System Center 2012\DPM\DPM\Temp directory . The crash log name itself will be such as: DPMCPWrapperServiceCurr.errlog.2012-04-30_19-25-50
If the DPM registry key is missing then a consistency check and\or a recovery point on a protected datasource using certificate authentication will fail with the following errors.
DPM Alerts Event Log Error
*************************
You may get one if not all of the alerts listed below.
DPM Alerts Event Log
: Event ID 3122
Warning
DPM Alerts Event Log: Event ID 3115 Warning
DPM Alerts Event Log: Event ID 3170 Critical
MSDPMCurr.errlog
******************
WARNING Failed: Hr: = [0x
80990940
] pDpmCmdProcObject->
SubmitRequest failed on server MEMBERSERVER.Contoso.com
, hrOriginal = 0x80990940, No further retry
WARNING CCommandProcessor::SendOutboundCommandUsingCertificate failed for Server: MEMBERSERVER.Contoso.com
4b0d-8401-d9773b85e7ab" xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd">
WARNING < ErrorInfo ErrorCode="33304 " DetailedCode="-2137454272" DetailedSource="2" ExceptionDetails="" xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd">
C2797F36-E616-4D5C-AC68-D9DA2216CE2D WARNING <Parameter Name="exceptionmessage" Value="The CPWrapper WCF Service encountered an unknown communication error " />
Solution : In this case to where the registry key is missing for the DPM Server itself on the DPM Server, the following needs to be done.
1.) Restore the key via registry backup, if no backup is available for this key and\or you do not feel comfortable with this measure then proceed to the next step.
2.) Verify that a valid certificate is in place on the DPM server. Once done, rerun the Set-DPMCredentials command to recreate that key. This being done taking care to use the proper syntax and correct thumbprint. Please reference the resource link below. Once done make sure the DPM reg key is present.
Example:
Resource: http://blogs.technet.com/b/dpm/archive/2012/04/23/how-to-use-certificates-to-authenticate-compu...
DPM Server Missing the Member Server Reg key
After Attach-ProductionServerWithCertificate command is run on the DPM server , if the registry key on the DPM server for the protected server is missing or corrupted for some reason then you can expect to see the following errors listed below.
In this example we will be noting the errors in the:
a.) DPM monitoring tab
b.) DPM management tab
c.) DPM events alerts tab
d.) MSDPMCurr.errlog
Reg Key:
HKLM\Software\Microsoft\Microsoft Data Protection Manager\Agent\2.0\Certificates\<
Protected ComputerName
>
DPM Management Tab--Agent Status
*******************************
DPM Monitoring Tab
*******************
Agent refresh error
DPM Alerts Event Logs—Event 3122
DPM Monitoring Tab--Protected server Consistency Check failure
MSDPMCurr.errlog
================
2C1A3335-C179-4D87-A993-CBD5B8B8A7C1 WARNING Failed: Hr: = [ 0x80070005 ]
0C9C 0F78 04/30 15:17:21.846 68 RornTaskDef.cs(488) 2C1A3335-C179-4D87-A993-CBD5B8B8A7C1 NORMAL RORN TaskDef: Task 2c1a3335-c179-4d87-a993-cbd5b8b8a7c1 stopped with error code 302
0C9C 0F78 04/30 15:17:21.846 02 EventManager.cs(98) 2C1A3335-C179-4D87-A993-CBD5B8B8A7C1 NORMAL Publishing event from AgentJobs.cs(747): JobProgress, [JobID=9470259c-538c-4e3d-8dc6-aff5bcee9d3c]
0C9C 0F78 04/30 15:17:21.847 07 AgentJobs.cs(751) 2C1A3335-C179-4D87-A993-CBD5B8B8A7C1 NORMAL refresh failed with error AMAgentAccessDenied; -2147024891; WindowsHResult
0C9C 0F78 04/30 15:17:21.847 01 TaskExecutor.cs(843) 2C1A3335-C179-4D87-A993-CBD5B8B8A7C1 FATAL Task stopped (state=Failed, error=AMAgentAccessDenied; -2147024891; WindowsHResult) , search "Task Diagnostic Information" for details.
Solution : In the case of the DPM server missing the proper reg key for the protected member server, the following needs to be done:
1.) Restore the key via registry backup, if no backup is available for this key and\or you do not feel comfortable with this measure then proceed to the next step.
2.) Make sure you have the proper .bin file generated by the member server. Once done, then run the Attach-ProductionServerWithCertificate.ps1 command specifying the correct .bin file. Please reference the resource link below. Once done verify the member server registry key is present.
Example:
Resource: http://blogs.technet.com/b/dpm/archive/2012/04/23/how-to-use-certificates-to-authenticate-compu...
Protected Server Side Registry Keys
Now we will focus on the protected server. We will experiment with both removing the Protected server registry key and the DPM registry key off of the Protected server. Once done we will take note of the common errors shown as a result.
We look at the following:
a.) DPM management tab
b.) DPM events alerts tab
c.) MSDPMCurr.errlog
d.) DPMRACurr.errlog
Member Server with Reg Key for itself missing.
If after running the setdpmserver –dpmCredential command, on the protected server if the registry key for itself is missing or deleted you may see the following errors below:
Reg Key:
HKLM\Software\Microsoft\MicrosoftDataProtectionManager\Agent\2.0\Certificates\<protectedServerName>
DPM Management Tab-Agent Status
DPM Alert Event Log-Event ID 3122
MemberServer Event Log –Event ID 85
MemberServer DPMRA.currerrlog
==========================
schannelutils.cpp(129) 7F9E668E-2A1D-4D55-A498-D7FA318B6068 WARNING Failed: Hr: = [0x80070002] : Error trying to open RegKey [HKLM\Software\Microsoft\Microsoft Data Protection Manager\Agent\2.0\Certificates\MemberServer.Contoso.com]
0EF075F8-504F-48E4-9BAF-85418F0DBD68 WARNING Logging event for error: 33304 , detailed: 0x30bf80
Note: Error 33304 has numerous causes listed at the later on in Part 3 of this series. This is the same indication as we saw when we removed the DPM registry key. In this case it is the member server missing its own registry key.
DPM MSDPMCurr.errlog
===============
034C 0FD4 04/30 15:55:09.481 07 AMUtil_expanded.cs(3590) 92CBF7B2-BA70-4ACF-B0DA-16FE40E43376 WARNING CheckTimeoutMessage: code[0x20000102], detailedCode[0x8099090e], errMgs[Internal error code: 0x8099090E ]
TaskInstance.cs(798) 92CBF7B2-BA70-4ACF-B0DA-16FE40E43376 WARNING <q1: ErrorInfo ErrorCode=”316” DetailedCode=”-2137454322” DetailedSource=”2” ExceptionDetails=”” xmlns:q1=”http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd”>
TaskInstance.cs(798) 92CBF7B2-BA70-4ACF-B0DA-16FE40E43376 WARNING <q1:Parameter Name=”servername” Value=”MEMBERSERVER.Contoso.com ” />
RornTaskDef.cs(488) 92CBF7B2-BA70-4ACF-B0DA-16FE40E43376 NORMAL RORN TaskDef: Task 92cbf7b2-ba70-4acf-b0da-16fe40e43376 stopped with error code 316
92CBF7B2-BA70-4ACF-B0DA-16FE40E43376 FATAL Task stopped (state=Failed, error=AMAgentNotResponding ; -2137454322; WindowsHResult),
Solution: If member server itself is missing it’s own registry key the we will need to perform the following:
1.) Restore the key via registry backup, if no backup is available for this key and\or you do not feel comfortable with this measure then proceed to the next step.
2.) Make sure we have the proper certificate in the computer\personal store on the member server.
3.) Make sure we have the correct .bin file created from the DPM server when you ran the Set-DPMCredentials on the DPM server.
4.) Run the SetDPMServer command on the member server taking care to make sure the correct DPM .bin file is specified along with the correct member server thumbprint from the certificate. Please reference the resource listed below.
Resource : http://blogs.technet.com/b/dpm/archive/2012/04/23/how-to-use-certificates-to-authenticate-compu...
MemberServer with Missing DPM reg Key
If after running the setdpmserver –dpmCredential command, on the protected server if the registry key for the DPM Server is missing or deleted you may see the following errors:
Reg Key:
HKLM\Software\Microsoft\MicrosoftDataProtectionManager\Agent\2.0\Certificates\<DPMServerName>
DPM Management Tab- Agent Status
DPM Monitoring Tab
DPM Alerts Event Log—Event ID 3122
Log Name: DPM Alerts
Source: DPM-EM
Date: 4/28/2012 6:34:44 AM
Event ID:
3122
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: DPM2012.Contoso.com
Description:The DPM protection agent on MEMBERSERVER.Contoso.com could not be contacted. Subsequent protection activities for this computer may fail if the connection is not established. The attempted contact failed for the following reason
: (ID: 3122
)
The DPM CPWrapper Service authorization failed on the MEMBERSERVER.Contoso.com computer. Exception Message = Access is denied.. (ID: 33303 )
Note the 3303 error which indicates that the client was not authorized by the service.
DPM Alerts Event Log—Event ID 3170
MSDPMCurr.errlog
****************
0DF4 0634 04/30 15:30:28.570 01 TaskInstance.cs(798) 51A021F9-8D5E-477B-94D4-73A986EEBDC0 WARNING <q1:
ErrorInfo ErrorCode="33303" DetailedCode="-2146233087
" DetailedSource="2" ExceptionDetails="" xmlns:q1="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd">
0DF4 0634 04/30 15:30:28.570 01 TaskInstance.cs(798) 51A021F9-8D5E-477B-94D4-73A986EEBDC0 WARNING <q1:Parameter Name="servername" Value=" MEMBERSERVER.Contoso.com " />
0DF4 0634 04/30 15:30:28.570 01 TaskInstance.cs(798) 51A021F9-8D5E-477B-94D4-73A986EEBDC0 WARNING <q1:Parameter Name="exceptionmessage" Value=" Access is denied." />
0DF4 0634 04/30 15:30:28.573 01 TaskExecutor.cs(843) 51A021F9-8D5E-477B-94D4-73A986EEBDC0 FATAL Task stopped (state=Failed, error=WCFServiceAuthorizationFailed; -2146233087; WindowsHResult), search "Task Diagnostic Information" for details.
Note: the 33303 error which indicates that the client was not authorized by the service.
Solution:
This will be the same steps done for the member server missing it’s own registry entry.
Those steps will recreate both the DPM server and member server registry key.
Conclusion: As a precautionary measure, a proactive step of backing up the server side and DPM side registry keys are suggested. This can be done via System state or BMR backup but should you not want to rollback to a previous system state or BMR snapshot, then just backing up those keys would work. In addition should you wish to backup the individual keys the restore would be much quicker vs. a system state and BMR restore.
This concludes Part 2 of DPM Certificate Troubleshooting. Part 3 will cover troubleshooting missing or invalid certificates.
Shane Brasher | Senior Support Escalation Engineer
Get the latest System Center news on Facebook and Twitter :
App-V Team blog:
http://blogs.technet.com/appv/
ConfigMgr Support Team blog:
http://blogs.technet.com/configurationmgr/
DPM Team blog:
http://blogs.technet.com/dpm/
MED-V Team blog:
http://blogs.technet.com/medv/
Orchestrator Support Team blog:
http://blogs.technet.com/b/orchestrator/
Operations Manager Team blog:
http://blogs.technet.com/momteam/
SCVMM Team blog:
http://blogs.technet.com/scvmm
Server App-V Team blog:
http://blogs.technet.com/b/serverappv
Service Manager Team blog:
http://blogs.technet.com/b/servicemanager
System Center Essentials Team blog:
http://blogs.technet.com/b/systemcenteressentials
WSUS Support Team blog:
http://blogs.technet.com/sus/
The Forefront Server Protection blog:
http://blogs.technet.com/b/fss/
The Forefront Endpoint Security blog :
http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog :
http://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog:
http://blogs.technet.com/b/isablog/
The Forefront UAG blog:
http://blogs.technet.com/b/edgeaccessblog/