First published on TechNet on Mar 05, 2018
Hello all! Nathan Penn and Jason McClure here to cover some PKI basics, techniques to effectively manage certificate stores, and also provide a script we developed to deal with common certificate store issue we have encountered in several enterprise environments (certificate truncation due to too many installed certificate authorities).
PKI Basics
To get started we need to review some core concepts of how PKI works. As you browse secure sites on the Internet and/or within your organization, your computer leverages certificates to build trust with the remote site it is communicating with. Some of these certificates are local and installed on your computer, while some are installed on the remote site. If we were to browse to https://support.microsoft.com we would notice:
The lock lets us know that the communication between our computer and the remote site is encrypted. But why, and how do we establish that trust? When we typed https://support.microsoft.com,the site on the other end sent its certificate that looks like this:
Certificate Chain
We won't go into the process the owner of the site went through to get the certificate, as the process varies for certificates used inside an organization versus certificates used for sites exposed to the Internet. Regardless of the process used by the site to get the certificate, the Certificate Chain, also called the Certification Path, is what establishes the trust relationship between the computer and the remote site and is shown below.
As you can see, the certificate chain is a hierarchal collection of certificates that leads from the certificate the site is using (support.microsoft.com), back to a root of trust, the Trusted Root Certification Authority (CA). In the above example, DigiCert Baltimore Root is the Trusted Root CA. All certificates in between the site's certificate and the Trusted Root CA certificate, are Intermediate Certificate Authority certificates. To establish the trust relationship between a computer and the remote site, the computer must have the entirety of the certificate chain installed within what is referred to as the local Certificate Store. When this happens, a trust can be established and you get the lock icon shown above. But, if we are missing certs or they are in the incorrect location we start to see this error:
Certificate Store
The certificate store is separated into two primary components, a Computer store & a User store. The primary difference being that certificates loaded into the Computer store become global to all users on the computer, while certificates loaded into the User store are only accessible to the logged on user. To keep things simple, we will focus solely on the Computer store in this post. Leveraging the Certificates MMC (certmgr.msc), we have a convenient interface to quickly and visually identify the certificates currently loaded into the local Certificate Store. This tool also provides us the capability to efficiently review what certificates have been loaded, and if the certificates have been loaded into the correct location. This means we have the ability to view the certificates that have been loaded as Trusted Root CAs, Intermediate CAs, and/or both (hmmm… that doesn't sound right).
Identifying a Trusted Root CA from an Intermediate CA
Identifying a Root CA from an Intermediate CA is a fairly simple concept to understand once explained. Trusted Root CAs are the certificate authority that establishes the top level of the hierarchy of trust. By definition this means that any certificate that belongs to a Trusted Root CA is generated, or issued, by itself. Understanding this makes identifying a Trusted Root CA certificate exceptionally easy to identify as the "Issued To" and "Issued By" attributes will always match.
Alternatively, an Intermediate CA is a Certificate Authority that builds upon the trust of some other CA. This can be either, another Intermediate CA, or a Trusted Root CA. Understanding this makes identifying an Intermediate CA certificate just as easy as the "Issued To" and "Issued By" attributes must be different.
To sum up a Trusted Root CA is issued by itself, while an Intermediate CA is issued by something else. Simple stuff, right?
Managing the Certificate Store
We know about remote site certificates, the certificate chain they rely on, the local certificate store, and the difference between Root CAs and Intermediate CAs now. But what about managing it all? On individual systems that are not domain joined, managing certificates can be easily accomplished through the same local Certificates MMC shown previously. In addition to being able to view the certificates currently loaded, the console provides the capability to import new, and delete existing certificates that are located within.
On a domain joined systems it is recommended to manage PKI at the enterprise level (which may explain why we named one of the MMC Enterprise PKI). This is done through the Group Policy MMC (gpmc.msc), and we would typically make the changes to a single policy linked at the domain level. Using this approach, we can ensure that all systems in the domain have the same certificates loaded and in the appropriate store. It also provides the ability to add new certificates and remove unnecessary certificates as needed.
Too Many Certs
On several occasions both of us have gone into enterprise environments experiencing authentication oddities, and after a little analysis trace the issue to an Schannel event 36885.
This event is caused by the number of certificates loaded into the computer's Trusted Root Certificate Authorities (TRCA) and Intermediate Certificate Authorities (ICA) stores. The most important part of the above warning is the following: "Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated." Unfortunately, here is what we don't know: Where was the list truncated, which certificate authorities did it grab, which certificate authorities did it NOT grab, and do I have all the certs that will be needed to build any of the given certificate chains for the requests that will be made? At this point many of you are asking, "How many is too many?" The answer to this is it depends, as the limitation is based on the size of the store which is limited to 16 kilobytes and not the number of certificates. In December 2012, KB931125 was released and intended only for client SKUs. However, it was also offered for Server SKUs for a short time on Windows Update and WSUS. This package installed all TRCAs enrolled in the Microsoft Trusted Root Program (more than 330). While we offer a fix it tool for individual systems here (SSL/TLS communication problems after you install KB 931125 - Windows Server), this wasn't identified as an issue in several environments. On a small scale, customers that experience certificate bloat issues can leverage the Certificate MMC to deal with the issue on individual systems. Unfortunately, the ability to clear the certificate store on clients and servers on a targeted and massive scale with minimal effort does not exist. On a larger scale, customers would be required to leverage the Microsoft built-in "Certutil" application via a script. This technique requires the scripter to identify and code in the thumbprint of every certificate that is to be purged on each system (also very labor intensive).
Introducing CertPurge
Overview of Script
CertPurge will remove all locally installed certificates from the Trusted Root Certification Authorities, Intermediate Certification Authorities, and Third-Party Root Certification Authorities stores on the local machine. Only certificates that are being deployed to the machine from Group Policy will remain.
What it solves
The ability to clear the certificate store on clients and servers on a targeted and massive scale with minimal effort. This is needed to handle certificate bloat issues that can ultimately result in authentication issues. On a small scale, customers that experience certificate bloat issues can leverage the built-in certificate MMC to deal with the issue on a system by system basis as a manual process. On a larger scale, customers would be required to leverage the Microsoft built-in "Certutil" application via a script. This technique requires the scripter to identify and code in the thumbprint of every certificate that is to be purged on each system (also very labor intensive).
How it works
CertPurge scans the following registry locations ("HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates" & "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\EnterpriseCertificates") and builds an array for all entries found under the Trusted Root Certification Authorities, Intermediate Certification Authorities, and Third-Party Root Certification Authorities paths. CertPurge then leverages the array to delete every subkey.
Backout Mechanisms
Prior to performing any operations (i.e. building array, purging certificates), CertPurge generates a backup of the "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates" & "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\EnterpriseCertificates" paths in their entirety into a .reg file stored in the c:\windows\ directory. In the event that required certificates are purged, an administrator can import the backup files and restore all purged certificates. (NOTE: This is a manual process, so testing prior to implementation on a mass scale is highly recommended).
Why certificates pushed via GPO are not affected
Certificates pushed via GPO are stored in the "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\SystemCertificates" path. As CertPurge does not target this location, all certificates deployed via GPO are unaffected.
What to do if not all required certificates are being published via GPO
KB 293781 details the certificates that are required for the operating system to operate correctly. Removal of the certificates identified in the article may limit functionality of the operating system or may cause the computer to fail. Ensure at a minimum that these certificates are published via a GPO prior to implementing the CertPurge application\script. If a required certificate (either one from the KB, or one specific to the customer environment) is purged, that is not being deployed via GPO, the recommended approach is as follows
1. Restore certificates to an individual machine using the backup registry file,
2. Leveraging the Certificate MMC, export the required certificates to file,
3. Update the GPO that is deploying certificates by importing the required certificates,
4. Rerun CertPurge on machine identified in step 1 to re-purge all certificates,
5. Execute a GPUpdate on machine identified in step 1 to receive updated GPO certificate deployment,
6. TEST!!!
7. Did we mention Test?
CloseOut and Additional Resources
At this point, hopefully we all understand some of the basics, what a certificate chain is, the difference in a Root certificate and an Intermediate/Issuing certificate, and where those certificates should be located on our systems. Also, we now have a method for cleaning things up things in bulk should things get out of control and you need to re-baseline systems in mass. Let us know what you all think, and if there is another area you want us to expand on next.
Additional Resources:
Failed TLS connection between unified communications peers generates an Schannel warning (microsoft....SSL/TLS communication problems after you install KB 931125 - Windows Server
Configure Trusted Roots and Disallowed Certificates
Disclaimer The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.