Hello, Shane Brasher here once again, following up DPM Certificate Troubleshooting–Part 1: General Troubleshooting and DPM Certificate Troubleshooting–Part 2: Registry with Part 3 of “DPM Certificate Authentication Troubleshooting”.
In this session we will go over some common symptoms you may see if the certificate is missing or is invalid. This is assuming that after you have installed the certificate, run all the proper commands and even have protection group set up, then later something has happened to the certificate itself.
Member Server with its certificate missing
This error is what you will likely see if AFTER cert protection is setup is done and then the cert is missing or corrupt.
DPM Management Tab-Agent status
MemberServer Application Alerts—Event ID 85
MemberServer DPMRACurr.errlog
****************************
5BD3AD20-B2AF-4D1F-95B6-B73212768440 WARNING Failed: Hr: = [ 0x80092004 ] : Error locating certificate with thumbprint 2ba53e0056bdde64a7fca789c62abd72a3f57610
5BD3AD20-B2AF-4D1F-95B6-B73212768440 WARNING Failed: Hr: = [0x80092004] : Encountered Failure: : lVal : CertificateUtil::GetCertificateContext(hCertStore, ssThumbprint, &pCertContext)
WARNING Failed: Hr: = [0x80092004] : Error locating certificate with thumbprint 2ba53e0056bdde64a7fca789c62abd72a3f57610
WARNING Failed: Hr: = [0x80092004] : Encountered Failure: : lVal : CertificateUtil::GetCertificateContext(hCertStore, ssThumbprint, &pCertContext)
WARNING OuterException of type System.InvalidOperationException from Method = GetCertificateFromStoreCore
WARNING Exception Message = Cannot find the X.509 certificate using the following search criteria: StoreName 'My', StoreLocation 'LocalMachine', FindType 'FindByThumbprint', FindValue '2ba53e0056bdde64a7fca789c62abd72a3f57610'.
Note: The highlighted portion shows that there is an issue with finding the thumbprint for the certificate.
Member Server DPM CPWrapper Log—Cert is missing and the CP Wrapper Service restarted.
******************************
WARNING Exception Message = Cannot find the X.509 certificate using the following search criteria: StoreName 'My', StoreLocation 'LocalMachine', FindType 'FindByThumbprint', FindValue '2ba53e0056bdde64a7fca789c62abd72a3f57610'.
WARNING Exception Stack = at System.ServiceModel.Security.SecurityUtils.GetCertificateFromStoreCore(StoreName storeName, StoreLocation storeLocation, X509FindType findType, Object findValue, EndpointAddress target, Boolean throwIfMultipleOrNoMatch)
WARNING Caught unhandled exception : System.InvalidOperationException: Cannot find the X.509 certificate using the following search criteria: StoreName 'My', StoreLocation 'LocalMachine', FindType 'FindByThumbprint', FindValue '2ba53e0056bdde64a7fca789c62abd72a3f57610'.
CRITICAL Exception Message = Cannot find the X.509 certificate using the following search criteria: StoreName 'My', StoreLocation 'LocalMachine', FindType 'FindByThumbprint', FindValue '2ba53e0056bdde64a7fca789c62abd72a3f57610'. of type System.InvalidOperationException, process will terminate after generating dump
Also if the Protected server cert if removed upon a reboot or the restart of the DPM CPWrapper service you may see the following error.
Solution: If the member server has it’s certificate missing then the following will need to be done.
1.) If the cert is backed up to a safe location, import the certificate into the proper computer\personal store and restart the DPMCPWrapper service. If you do not have a backup of the certificate then proceed to the next step.
2.) Request a new certificate making sure to specify the correct cert attributes and that it is placed into the computer\personal store.
3.) Re-run the SetDPMServer commands to recreate the memberserver bin file. Copy the bin file to the DPM server. Once done re-run the Attach-ProductionServerWithCertificate.ps1 on the DPM server. Please reference the resource link below.
Resource : http://blogs.technet.com/b/dpm/archive/2012/04/23/how-to-use-certificates-to-authenticate-compu...
Important: There may be times to where you may still have to reboot both the member server and the DPM server.
DPM Server With Missing Cert
This scenario will go over symptoms when the DPM server certificate is missing.
If the DPM server is missing its certificate then you will see this in the DPM gui on the agent refresh.
DPM Monitoring tab
******************
Note the 3301 error which means the certificate is invalid.
DPM Alerts Event Log
******************
Note: The 33301 equates to the certificate is invalid.
MSDPMCurr.errlog snippet
***********************
cmdprocforcertificate.cpp(331) [000000001A7F4F50] WARNING CCommandProcessor::
SendOutboundCommandUsingCertificate failed for Server: MemberServer.Contoso.com
WARNING ConfigureProtection.OnFailure.AADeactivationBlock.RAForRead.PT : RADeleteWorkItem, StatusReason = Timeout (StatusCode = -2146233079 , ErrorCode = WCFClientCertificateInvalid, workitem = a1e5773c-a587-4788-a7fb-622f6bf7341e)
5A0AC966-C3A0-4D24-95FF-E96FD0DE04DA WARNING CheckTimeoutMessage: code[0x00008215], detailedCode[0x80131509], errMgs[Unknown error (0 x80131509 ) (0x80131509)]
5A0AC966-C3A0-4D24-95FF-E96FD0DE04DA WARNING <ErrorInfo ErrorCode=" 33301 " DetailedCode="-2146233079" DetailedSource="2" ExceptionDetails="" xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd">
5A0AC966-C3A0-4D24-95FF-E96FD0DE04DA WARNING <Parameter Name="machinename" Value=" DPM2012Backup.Contoso.com " />
5A0AC966-C3A0-4D24-95FF-E96FD0DE04DA WARNING <Parameter Name="exceptionmessage" Value="Cannot find the X.509 certificate using the following search criteria: StoreName 'My', StoreLocation 'LocalMachine', FindType 'FindByThumbprint', FindValue '994b424d93fc08e4fe88c787298c7617ee095cda' ." />
DPMCPWrapperServiceCurr.errlog
=============================
This may be seen upon restarting the DPMCPWrapper service if the cert is missing.
everettexception.cpp(761) CRITICAL Exception Message = Cannot find the X.509 certificate using the following search criteria: StoreName 'My', StoreLocation 'LocalMachine', FindType 'FindByThumbprint', FindValue 'c8ccf847ae8d319691feea1d6f796f0d67fdc7c4'. of type System.InvalidOperationException, process will terminate after generating dump
Note the message in regards to generating a dump. This dump (crash log) will be located in the following directory: %Program Files%\Microsoft System Center 2012\DPM\DPM\Temp. The dump file will have a name of “DPMCPWrapperServiceCurr.errlog.2012-07-11_18_06_16.Crash
Solution : If the DPM Certificate is missing, then please follow the steps below.
1.)If the cert is backed up to a safe location, import the certificate into the proper computer\personal store and restart the DPMCPWrapper service. If you do not have a backup of the certificate then proceed to the next step.
2.)Request a new certificate for the DPM server making sure to specify the correct cert attributes and that it is placed into the computer\personal store.
3.) Re-run the SetDPMCredentils commands to recreate the DPM bin file. Copy the bin file to the member server. Once done re-run the SetDPMServer command on the member server to generate this bin file. Copy the member server bin file to the DPMserver.
4.) On the DPM server re-run the Attach-ProductionServerWithCertificate.ps1 command.
Please reference the resource link below.
Resource : http://blogs.technet.com/b/dpm/archive/2012/04/23/how-to-use-certificates-to-authenticate-compu...
Important: This is considered a very bad situation. As if you have many servers you are protecting via certificate based authentication and the DPM cert is missing, it will be like starting all over again. You will have to:
a.) Generate the DPM bin file
b.) Copy it to each server that you were protecting via cert authentication.
c.) run the setdpmserver command
d.) take each server bin file to the DPM server.
e.) on that DPM server run the attach command.
This will have to be done for each server that you are protecting with certificate authentication. Naturally if you are protecting 100 servers via cert then this can be very labor intensive.
As a precautionary measure I strongly suggest that you export your DPM and member server certificates and save them in a safe location.
Expired Certificate
MemberServer Cert Expired
If the certificate has expired on the protected server then you will see the following errors.
DPM Management Tab-Agent Status
DPM Monitoring Tab
=================
DPMRACurr.errlog
================
415BF1BD-04EF-486C-A8D0-0C6A8E8E0BBD FATAL <Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2146893016 " Reason="Error" CommandID="RAReadDatasetDelta" CommandInstanceID="80b85883-9822-4a64-bea0-1c661101dbe5" GuidWorkItem="856c0da1-fad7-46ba-a215-db95b90de630" TETaskInstanceID="415bf1bd-04ef-486c-a8d0-0c6a8e8e0bbd"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="536872925" DetailedCode="-2146893016 " DetailedSource="2"><Parameter Name="AgentTargetServer" Value="MemberServer.Contoso.com"/></ErrorInfo><RAStatus><RAReadDatasetDelta xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd" BytesTransferred="0" NumberOfFilesTransferred="0" NumberOfFilesFailed="0" DataCorruptionDetected="false"/></RAStatus></Status>
415BF1BD-04EF-486C-A8D0-0C6A8E8E0BBD WARNING Failed: Hr: = [ 0x80090328 ] : Encountered Failure: : lVal : hr
415BF1BD-04EF-486C-A8D0-0C6A8E8E0BBD WARNING Logging event for error: 33302, detailed: 0xa61590
415BF1BD-04EF-486C-A8D0-0C6A8E8E0BBD WARNING Failed: Hr: = [0x00008216] CCmdProcEvent::GetEventId: unexpected errorCode: detailed hr: 0x00a61540
Note:
The error codes
-2146893016
and
0x80090328
basically translates to
SEC_E_CERT_EXPIRED
# The received certificate has expired.
The error code 33302 is the service authentication failed.
DPM SERVER with Cert Expired
If the certificate has expired on the DPM server the you will see an error like this.
DPMRCurr.errlog
ExceptionPolicy.cs(169) WARNING InnerException of type System.IdentityModel.Tokens.SecurityTokenValidationException from Method = Build
02F8 094C 05/02 17:32:29.282 04 ExceptionPolicy.cs(174) WARNING Exception Message = The X.509 certificate CN=DPM2012.Contoso.com chain building failed. The certificate that was used has a trust chain that cannot be verified. Replace the certificate or change the certificateValidationMode. A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file.
02F8 094C 05/02 17:32:29.282 04 ExceptionPolicy.cs(174) WARNING Exception Stack = at System.IdentityModel.Selectors.X509CertificateChain.Build(X509Certificate2 certificate)
02F8 094C 05/02 17:32:29.282 04 cmdprocforcertificate.cpp(232) [0000000000B6FB90] 415BF1BD-04EF-486C-A8D0-0C6A8E8E0BBD WARNING Failed: Hr: = [0x80990941] pDpmCmdProcObject->SubmitResponse failed on server DPM2012.Contoso.com, hrOriginal = 0x80131501, No further retry
02F8 094C 05/02 17:32:29.282 04 cmdprocforcertificate.cpp(331) [0000000000B6FB90] 415BF1BD-04EF-486C-A8D0-0C6A8E8E0BBD WARNING CCommandProcessor:: SendOutboundCommandUsingCertificate failed for Server: DPM2012.Contoso.com
02F8 094C 05/02 17:32:29.282 04 cmdproc.cpp(2631) [0000000000B6FB90] 415BF1BD-04EF-486C-A8D0-0C6A8E8E0BBD WARNING Logging event for error: 33302, detailed: 0xa61590
02F8 094C 05/02 17:32:29.282 04 events.cpp(89) [0000000000A2FF90] 415BF1BD-04EF-486C-A8D0-0C6A8E8E0BBD WARNING Failed: Hr: = [0x00008216] CCmdProcEvent::GetEventId: unexpected errorCode: detailed hr: 0x00a61540
DPM DPMCPWrapperServiceCurr.errlog
CertificatesHelper.cs(498) NORMAL Certificate with subject: CN=DPM2012.Contoso.com and thumbprint: 02E436145567778DED5E95138343AE1F19163ED1 is not valid
0AD0 0C3C 05/02 18:07:28.110 09 CertificatesHelper.cs(503) WARNING Flags = NotTimeValid, Info = A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file.
0AD0 0C3C 05/02 18:07:28.110 09 CertificatesHelper.cs(166) WARNING The certificate with subject: CN=DPM2012.Contoso.com is not trusted
DPM CPWrarpper Error logs codes
For Client related errors (33300 – 33302) refer to dpmra*.errlog or msdpm*.errlog .
For service side errors (33303 – 33304) look up failures in dpmcpwrapperservice*.errlog
For PKI related issues, the Crypto API Event log is a very useful way to figure out what went wrong during certificate validation. This event log is available from Windows Vista onwards.
Configuration Step Errors
33231 : Certificate not found in Personal Store of “LocalMachine” StoreLocation.
33232 : Exception trying to locate a certificate.
33233 : Exception encountered trying to validate certificate.
33234 : The certificate is invalid.
33235 : Error trying to add a firewall rule.
33236 : Error trying to configure DpmCPWrapperService.
33237 : The generic fall back error.
33241 : No .NET 3.5 SP1 detected on the machine (seen only by SetDpmServer.exe).
Errors during business continuity
33300: Configuration errors in the WCF Client config file. (dpmra.exe.config or msdpm.exe.config).
33301: Client certificate is invalid.
33302: The service authentication failed.
33303: The client was not authorized by the service.
33304: The WCF Service is in a bad state. Some possible reasons can be:
a.) Service not running on the remote peer.
b.) Crash in the WCF Service.
c.) WCF unresponsive to client requests leading to Timeouts.
d.) Generic communication failures.
e.) Authentication failure of the client on the service side.
f.) Missing Registry keys
Conclusion: It is imperative that your Certificate infrastructure is extremely solid with a good connection to the CRL for both the DPM server and the member server. In addition there needs to be a stable link between the DPM server and the member server. Once the certs are in place they should be left alone and not need to be altered in any manner. Of course its best to be prepared with a contingency plan should things go awry. As mentioned earlier, its suggested to export your certificates for safe keeping should you have to recover from a missing certificate.
Appendix A
CAPI2 Event Logging
If you are facing repeated authentication failures, refer CAPI2 event viewer logs on both DPM and protected computer. This is not enabled by default. To enable it navigate to:
Event Viewer\Applications and Service Logs\Microsoft\CAPI2
Then right click on “Operational” and select “Enable Log”.
Once done reproduce the problem.
Example:
Going into the details of the properties we can see:
This tells us the CRL server cannot be reached.
4.) Make sure the DPM CPWrapper Service is started and set to “Automatic”. If it is not, then restart the service and test your Attach then OR your connectivity via cert usage.
Additional Resources
Microsoft Root Certificate Program : http://technet.microsoft.com/en-us/library/cc751157.aspx
How to use certificates to authenticate computers in workgroups or untrusted domains with Data Protection Manager : http://blogs.technet.com/b/dpm/archive/2012/04/23/how-to-use-certificates-to-authenticate-compu...
Shane Brasher | Senior Support Escalation Engineer
Get the latest System Center news on Facebook and Twitter :
App-V Team blog:
http://blogs.technet.com/appv/
ConfigMgr Support Team blog:
http://blogs.technet.com/configurationmgr/
DPM Team blog:
http://blogs.technet.com/dpm/
MED-V Team blog:
http://blogs.technet.com/medv/
Orchestrator Support Team blog:
http://blogs.technet.com/b/orchestrator/
Operations Manager Team blog:
http://blogs.technet.com/momteam/
SCVMM Team blog:
http://blogs.technet.com/scvmm
Server App-V Team blog:
http://blogs.technet.com/b/serverappv
Service Manager Team blog:
http://blogs.technet.com/b/servicemanager
System Center Essentials Team blog:
http://blogs.technet.com/b/systemcenteressentials
WSUS Support Team blog:
http://blogs.technet.com/sus/
The Forefront Server Protection blog:
http://blogs.technet.com/b/fss/
The Forefront Endpoint Security blog :
http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog :
http://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog:
http://blogs.technet.com/b/isablog/
The Forefront UAG blog:
http://blogs.technet.com/b/edgeaccessblog/