First published on TechNet on Apr 02, 2018
Hello Everyone, my name is Zoheb Shaikh and I'm a Premier Field Engineer with Microsoft India. I am back again with another blog and today I'll share with you something interesting that I came across recently which caused the Certificate Authority to go down, and how I was able to isolate the issue by using Process Monitor (Procmon). ( https://docs.microsoft.com/en-us/sysinternals/downloads/procmon ) Before I discuss about the issue, I would like to briefly share a bit of background on CDP & AIA extensions and their use. I could try to explain what the AIA and CDP are and the way to configure it, but here is a short article on it and how revocation works. https://docs.microsoft.com/en-us/windows-server/networking/core-network-guide/cncg/server-certs/co... https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/e... AIA and CDP extensions are very important for certificate validation. The Authority Information Access or AIA repository host CA Certificates. This location is "stamped" in the Authority Information Access extension of issued certificates. A client that is validating a certificate may not have every CA certificate in the chain. The client needs to build the entire chain to verify that the chain terminates in a self-signed certificate that is trusted (Trusted Root). CDP extensions host the CRLs that the CA publishes. The CRL Distribution Points extension is "stamped" in certificates. Client use this location to download CRLs that the CA Publishes. When a client is validating a certificate, it will build the chain to a Root CA. If the Root CA is trusted this means the certificate is acceptable for use. However, for applications that require revocation checking, the client must also validate that every certificate in the chain (with the exception of the Root) is not revoked. Coming back to the customer scenario, they had a 2 Tier CA Hierarchy with an Offline Root CA and an Enterprise Subordinate CA both running 2012 R2 and an IIS server hosting the CDP/AIA extensions of Root CA (As shown in the diagram below): 
Unable to renew or Enroll certificates getting the error | (The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE)
The first thing we did was to export a certificate in .cer format and run the command " certutil -verify -urlfetch " against the certificate. As a result, we got the error: Error retrieving URL: A connection with the server could not be established 0x80072efd (INet: 12029 ERROR_INTERNET_CANNOT_CONNECT) http://fabricam-ca1.corp.fabrikam.com/vd/Fabricam_Group-CA.crt We got this error for both CDP and AIA extensions. When we tried to manually browse these extensions in Internet Explorer, we were able to access them but from the command line (I.e. certutil -verify -urlfetch) it always failed. ROADBLOCK!! We ran the same command (certutil -verify -urlfetch) against public certificates and observed similar behavior. And again, we could successfully browse to their CDP & AIA extensions from Internet Explorer. Upon further checking, we found this behavior was occurring for about 20% of the users. We checked if there were any proxy settings in IE and found none . CAPI2 logging further confirmed that there were issues with Certificate Revocation checking for both Internal and Public CA's. Since we were in trouble we decided to collect a Procmon log with a simultaneous network trace, while again running "certutil -verify –urlfetch." We saw the following in PROCMON: 11:48:25.9643758 PM certutil.exe 2348 TCP Reconnect Fabricam-ca1.corp.fabricam.com: 51188->210.99.197.47:8080 SUCCESS Length: 0, seqnum: 0, connid: 0 We also saw multiple reconnects Operation > TCP Reconnect > Path > Fabricam-ca1.corp.fabricam.com:51188 -> 210.99.197.47:8080 Under Process Path > C:\Windows\system32\certutil.exe >> Command Line > certutil -verify -urlfetch subcacert1.cer In Network Monitor (Netmon), we observed the following: 676 12:31:35 AM 6/17/2015 9.7827898 0 certutil.exe 10.10.60.47 Some Public IP TCP TCP:Flags=CE....S., SrcPort=52443, DstPort=HTTP Alternate(8080), PayloadLen=0, Seq=1424697589, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192 {TCP:125, IPv4:3} 815 12:31:38 AM 6/17/2015 12.7970106 0 certutil.exe 10.10.60.47 Some Public IP TCP TCP:[ SynReTransmit #676]Flags=CE....S., SrcPort=52443, DstPort=HTTP Alternate(8080), PayloadLen=0, Seq=1424697589, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192 {TCP:125, IPv4:3} So the requests from the Sub CA were not getting a response from an external IP. Further analyzing the Procmon showed us the following: 11:48:22.9158915 PM certutil.exe 2348 RegQueryValue HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\InternetSettings\Connections\ WinHttpSettings SUCCESS Type: REG_BINARY, Length: 45, Data: 28 00 00 00 00 00 00 00 03 00 00 00 19 00 00 00 11:48:22.9159009 PM certutil.exe 2348 RegQueryValue HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\WinHttpSettings SUCCESS Type: REG_BINARY, Length: 45, Data: 28 00 00 00 00 00 00 00 03 00 00 00 19 00 00 00 11:48:22.9174322 PM certutil.exe 2348 RegQueryValue HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\WinHttpSettings SUCCESS Type: REG_BINARY, Length: 45, Data: 28 00 00 00 00 00 00 00 03 00 00 00 19 00 00 00 We got the above for the path C:\Windows\system32\certutil.exe >> Command line certutil -verify -urlfetch subcacert1.cer. We found that they had set Proxy settings before using Group Policy preferences, which got tattooed in the registry even though it was not reflected in Internet Explorer. Thus, we deleted the registry key HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\ Internet Settings\Connections\WinHttpSettings and then confirmed that the revocation check worked fine for external and internal websites. Lesson learned from this post? "WHEN IN DOUBT, USE PROCMON"
Hope this helps,
Zoheb