Lost access to your Root CA in your 2-Tier PKI? Don’t worry, Use Cross Signing to Recover!
Hi there, this is Byron from the Microsoft Directory Service Support team. Today, I would like to share a new method to recover your PKI environment. Imaging how frustrating it would be if we lost our Root CA forever, such as a root certification authority VM was accidentally deleted, or the machine of the Root CA got into a no boot state with no valid way to recover it via backup! Also imagine that you did not backup the Root CA private key or database and have no way to recover. A traditional method would be rebuilding the whole PKI infrastructure from scratch. But there is another way which could save us a lot of time and effort (Good news to us! 😊) . How If it is a two-tier PKI, and the Intermediate CA Server is the one issuing certificates to the environment, and we still have access to the Intermediate CA Certificate with private key, we can build another Root CA, and “link” the Intermediate CA Server to the new Root CA. The steps are simple, just renew the Intermediate CA server with “Same Key pair” with this new Root CA. A high-level diagram below: In fact, this is like the Concept of “Cross Signing”; the newly issued Intermediate CA Certificate could be considered as a “Cross Certificate” as well. Cross Signing is typically used to make one PKI hierarchy chains up through another hierarchy, even when originally it was not configured that way. This technique is designed for a leaf certificate to build a different certificate verification chain, in case one of the chains fails, allowing certification verification to succeed. Cross signing is also used by public CA companies to start their business which we will discuss more in a later section. Here are detailed steps on how to do this: On the Intermediate CA Server. Open the Certification Authority Management Console. Right click on the CA Name node -> All Tasks -> Renew CA Certificate. Renew Intermediate CA server with “Same Key pair” to create the Certificate renewal request file: Submit the certificate request file (.req) on the newly Deployed Root CA Server, issue it, and go to: Issued Certificates node-> Right click on issued Certificate -> All Taks -> Export Binary Data… -> Save as .cer file. Copy this .cer file to the Intermediate CA. On the Intermediate CA Server. Install the newly issued Intermediate CA certificate issued by the new Root CA. The existing issued Leaf Certificates verification should continue to work and chain up to the new Root CA. We can use command Certutil -urlfetch -verify c:\certificate.cer > certificate.txt to export the certificate verification and chain build information. Previous Chain: Leaf certificate: 5700000006d907d38be599e05a000000000006 Issuer: CN=ContosoRootCA01,DC=contoso,DC=com (Previous corrupted Root CA) Current Chain: Leaf certificate: 5700000006d907d38be599e05a000000000006 Issuer: CN=ContosoRootCA02,DC=contoso,DC=com (New built Root CA) How and Why So exactly what is “Cross Signing”? Let’s see the following story (I found this story online): A long time ago (maybe not that long 😊), there was a Certification Authority company called LetsEncrypt. When they started their business, they generated their own 'ISRG' Root CA Certificate (ISRG Root X1). However, it takes time for the industry to accept new Root Certification Authorities. They could not wait that long to start their business, as this might have taken years. They deployed an Intermediate CAs first (LetsEncrypt X3 and LetsEncrypt X4) and used a popular public Certification Authority (DST Root X3) to sign those Intermediate CA Certificates. This allowed them to start their business immediately and issue Leaf Certificates to customers without waiting for the world to accept their own Root CA. So, the Certificate Chain back then was DST Root X3 --> LetsEncrypt X3 / LetsEncrypt X4 --> Leaf Certificate. After 5 years of hard work, the market accepted their Root CA (DST Root X3) and added it to all kinds of products’ Trusted Root. They signed their Intermediate CA certificates with their own Root CA. Now, another chain is available: ISRG Root X1 --> LetsEncrypt X3 / LetsEncrypt X4 --> Leaf Certificate. How come the same Intermediate CA can be chained up to different Root CAs? The magic trick is the Intermediate CA Certificate kept the “Same Asymmetric Key Pair” when it got signed by both DST Root X3 and ISRG Root X1. As you can see from the above screenshots, the issuer is different, and they are two different intermediate CA certificates, but the trick here is the Same Key Pair, even though they were signed by different Root CA’s, aka “Cross Signing”. Another key point here is not all Certificate Chains rely on the AIA path. Another common Certificate chain build method is Key match using AKI/SKI and PKCS#7 which means the server side sends both Leaf Certificate along with Intermediate CA to the client for verification, Client does not need to build chain to Intermediate using AIA. You can refer to this document for more information about this story: https://scotthelme.co.uk/cross-signing-alternate-trust-paths-how-they-work/ There is another “Cross Sign” method targeting the Root CA Certificate itself. The concept is a little bit different: Using one Root CA Certificate to sign another Root CA Certificate. What is the usage scenario? Why do we need this? Scenario 1 Here is a story: In a large corporation, deploying Root CA certificate to all devices might be challenging. It could be time-consuming and might take a lot of administrative effort. Therefore, during the time periods when large corporations renew their Root CAs, they must find a way for newly issued Certificates under the renewed Root CA start to work as soon as possible. The idea is to sign the new Root CA certificate using the old Root CA Certificate, so the chain could be: leaf Certificate --> Intermediate CA -> New Root CA (Signed by Old Root CA)--> Old Root CA. Because devices should trust Old Root CA already, the new Leaf Certificate works immediately after renewal without waiting for new Root CA Deployed to all devices. Scenario 2 Another scenario is once the New Root CA is deployed to the environment, the company wants to remove the old Root CA certificate from the Devices’ trusted Store for company policy reasons. How can the existing Certificate continue to work if they are not yet expired and issued by old Root CA Certificate? The solution is also Cross Certificate: Create a CA Certificate with old CA key pair but signed by the new Root CA Certificate. The chain build would be below: Existing Issued Leaf Certificate --> Intermediate CA --> Old CA Certificate (cross signed by new Root CA) --> New Root CA. In fact, Active Directory Certificate Service supports this and will generate Cross Certificates by default when renewing a Root CA with a new key. Remember that when we renew the Root CA, there will be two additional CRT files called XXXXX (0-1).crt and XXXXX(1-0).crt. These certificates(.crt) are Cross CA Certificate’s. (0-1) is the New Root CA Certificate signed by Old Root CA Certificate. (1-0) is the old Root CA Certificate signed by new Root CA Certificate. They are used for the above Scenario 1 and Scenario 2 As you can see, the Root CA Certificate has an AKI (Authority Key Identifier), which means it was signed by a CA and an SKI (Subject Key Identifier) that matches it. Of course, in Active Directory, we rarely see the deployment of the Cross Certificate, because for Windows devices, the Active Directory is sufficient to quickly deploy new Root CA Certificate to Windows domain members. Summary The common method to perform PKI disaster recovery when the CA private key and database cannot be restored involves rebuilding the entire PKI Hierarchy PKI from scratch, and replacing every single certificate used by applications and servers. While the common method mentioned above is still valid, the “Cross Signing” method illustrated in this blog offers an alternative quick method to recover our PKI Hierarchy. This could potentially save us from spending a lot of disaster recovery time & administrative effort 😊.
First Issuance manual, with automated renewals
Hey all Rob Greene again. Seems like I have been on this PKI kick lately, and today is not going to be any different. Occasionally, I will get a customer who must get certificates issued for things like Web sites, and they must have custom Subject Alternative Name (SAN) DNS values on the issued certificate. They hate that their web server admins must submit the request, and then as Certificate Authority Managers, they must look in the Certification Authorities database and review the Pending Requests table. Some background The first question that they ask is if there is a way to not require them (the CA Manager) to approve the request as this slows down the process of getting the certificate issued. From a technical perspective most of you would know that the answer is YES, you can configure the certificate template to automatically issue. However, this is NOT a good thing to do, and is seen as a Microsoft CA vulnerability; if you have a CA audit happen you might fail the audit because you allowed this. This specific vulnerability is considered the ESC1 vulnerability. ESC1 wants either an Enrollment Agent to cross sign the certificate request, a CA Manager to review the request, or both to validate the information in the Subject Alternative Name (SAN), when the certificate template’s Subject Name tab is set to “Supply in the request”, before allowing the certificate to be issued. This is recommended since the submitter is allowed to enter unvalidated information in the SAN, unlike when the certificate templates Subject Name tab is configured for “Build from this Active Directory information”. How to protect the template against ESC1? Given this background information, you should already be thinking to yourself, “Rob is not going to tell me to break anything security wise.”, and you would be correct. Rather, I am going to tell you that any certificate template that you have configured for “Supply in the request”, should at a minimum be configured to require CA Manager approval. Then for something like the certificate template that Network Device Enrollment Services (NDES) uses for its certificate, I would advise you to use the Enrollment Agent configuration. The Exchange Enrollment Agent (Offline request) template has the Enhanced Key Usage (EKU) of “Certificate Request Agent”, and this is the certificate NDES uses to sign the Certificate Service Request (CSR) that the SCEP client sends to the NDES Server. Of course, I am just giving some good examples of real-world certificate templates that you are going to be using within the environment. The last part is that the security permissions SHOULD be locked down as to who can enroll for each of these certificate templates. Specifically, enrollment for the two Certificate templates needed by the NDES services, it should be the CA Manager NDES Admin and the NDES Server. The Exchange Enrollment Agent (Offline request) and CEP Encryption certificate templates should be locked down as to who can request enrollments. Just like the custom Adatam Web Server certificate template, only IIS (Internet Information Services) Admins, and IIS Web Servers should have access to this template to be able to enroll with it. So what can you do for me here? Let’s first discuss the scenario that I am going to cover in the blog. Although I would love to cover all the possible uses that all you lovely internetz users might have for certificate enrollment, this blog will have to end at some point. The Scenario IIS Admins are tired of remembering to go to all their web servers and request new certificates for them before they expire. The CA Manager is also tired of the IIS Admins always sending emails nagging that they need their certificate enrollment requests issued. Then always having to go to the Certification Authority snap-in to approve/issue the same certificates over and over each year because of rules that web server certificates can only be valid for 398 days. See (https://thehackernews.com/2020/09/ssl-tls-certificate-validity-398.html) What is the solution? Side bar: If you want to modify the settings of any default certificate template, it is best to duplicate the original certificate template and then make changes to it. This is recommended so that you always have the default templates available in case you must create a new template in the future that is based off the original template. If the original template is modified two to three years down the line, you may not remember what was changed so that it could be changed back to the original template configuration. As far as naming goes, we would recommend that you put your company name or initials in the template name and keep the original template name after that. In this blog you will see the template used is called Adatam Web Server. This is a customized template based off the Web Server template. Keep in mind that this can be done for any certificate template that has the Subject Name tab configured for “Supply in the request”, this not limited to just web server certificates. I am in, so what’s next As you can imagine, there are several things that need to be configured to get this working: A Security group needs to be created for Users who are going to request the certificate via the Local Computer Certificate enrollment wizard based on the certificate template. A security group needs to be created for the computer accounts that need the certificate issued based on the security template. Certificate template needs to be created. Certificate template changes need to be made to support this enrollment method. Certificate template Permissions need to be configured. Group Policy setting for Computer Certificate Autoenrollment Security groups need to be created. The first two steps should be self-explanatory as to how to create groups and set up the group scopes. We are not going to cover group creation in this blog. I do want to point out one thing that happens often: Most Administrators seem to remember that if you add a user to a group, the user needs to log off and back on before the group can be added to their tokens. When adding a computer account to a new group it also needs to log off and back on before the group membership can be added to its token. Typically, the log off and back on for a computer will be a computer reboot; that needs to happen after the new group membership is added to its token. Certificate Template creation and changes The base template that you use to duplicate can of course be anything you would like, but in our example, we are just going to duplicate the good’ol faithful Web Server certificate template. Launch the Certificate Template snapin: CertTmpl.msc Find the certificate template named: Web Server Right click on the Web Server template and select Duplicate Template. The Compatibility tab, set the values as you wish. NOTE: Stay away from using anything higher than Windows Server 2012 R2/Windows 8.1 for your templates if you are using CEP and CES as these two web services have problems seeing templates with compatibility higher than this. The General tab, type in a descriptive name for the new certificate name you are configuring. The Request Handling tab, set the values that make sense for your template. NOTE: It might make sense to set the checkbox “Renew with the same key”. The Cryptography tab, you may want to modify the Minimum key size value and select the Provider Category and Providers that makes sense for the certificate templates use. NOTE: I would strongly suggest NOT setting Use alternate signature format on the template either. This is a proprietary format that many applications will not understand. This check box will only be available when you are using a Key Storage Provider (KSP) provider type. The Issuance Requirements tab is where the heavy lifting is happening. Check the box “CA certificate manager approval”. Select the radio option “Valid existing certificate” under the Require the following for reenrollment section toward the bottom of the dialog box. The Security tab is where the security groups that were created for the server admins and the servers themselves need to be configured on the template. You would add both security groups, and they only need the Allow Enroll permissions. Neither group should have the Autoenroll permission defined on it. When done setting the other options for the template, click the OK button. Setup the computer certificate autoenrollment group policy Next, we are not going to cover how to create a Group Policy Object, and things like GPO ordering, etc. We are going to discuss just the policy settings that are important for the discussion of this blog. Launch Group Policy Management Console: GPMC.msc Either create a new Group Policy or modify an existing one. Navigate to: Computer Configuration\Policies\Windows Settings\Security Settings\Local Policies\Public Key Policies. Double click on the policy setting “Certificate Services Client - Auto-Enrollment”. Enable the policy, and check the two boxes: Renew expired certificates, update pending certificates, and remove revoked certificates. Update certificates that use certificate templates. NOTE: To learn more about these two settings I would recommend going over to Vadims blog. He does a great job of explaining what each of these do: https://www.sysadmins.lv/blog-en/certificate-autoenrollment-in-windows-server-2016-part-3.aspx Click the OK button on the dialog box. Close the Group Policy Management Editor. Link the Group Policy object to what seems appropriate within your Active Directory structure. Trying to test this feature, but it seems like the renewal does not work. Of course, with any of these things, you should always test things before relying on the solution in a production environment. I do want to caution you about setting up a test environment and then setting up the certificate for a short issuance period as this tends not to work out well for most customers, and they eventually end up calling stating that autoenrollment is not working. There are few things to keep in mind about certificate automatic enrollment / automatic renewal process. The first thing to remember is that certificate auto enrollment / renewal only happens on the following triggers: At User logon, or computer boot, for the corresponding security context. Every 8 hours after user logon, or computer boot. This means that after the user or computer has been logged in or turned on for 8 hours then the auto enrollment code will happen again for the security context in question. To trigger auto-enrollment to run manually you have two options. Run GPUpdate /Force. Part of the function that this does is it causes an autoenrollment cycle to happen. This can be a bit much especially if you are applying a lot of user and computer group policy. This will trigger both computer and user autoenrollment to run. Use one of the two CertUtil command lines based on which security context you want enrollment to run against. For computer: CertUtil -Pulse For User: CertUtil -User -Pulse Certificate renewal WILL NOT happen until 90% of the Certificate lifetime has expired. Trying to use the template setting “Renewal period” does not change this fact. Auto enrollment / renewal attempt MUST happen while the certificate is still valid, and after the start of the 10% certificate lifetime left and the expiration time of the certificate. (MUST NOT BE EXPIRED). Like being unable to re-animate something that is already dead, you cannot renew a certificate that has already expired. The math that needs to be done is: (CertLifeTimeInDays * 24) * .10 The value from the above MUST be larger than 8 hours. Let's do some math Here is an example: Set up a certificate that is valid for two days for testing and we will run through the exercise here: Find out how many hours the certificate is valid for: 2*24 = 48 Hrs. Figure out what 10% of certificate lifetime in hours is: 48 *.10 = 4.8 Hrs. This shows us that for one of these certificates to be renewed, the process that runs only every 8 hours would have to run within this 4.8 Hrs. window. This is going to tell us that sometimes you would get lucky with a certificate renewal attempt being done, but more than likely MOST of the time you would fail to renew the certificate because it would expire before the auto enrollment cycle happens. Another example is a certificate that is valid for 4 days. Let us run through the exercise here. Find out how many hours the certificate is valid for: 4*24 = 96 Hrs. Figure out what 90% of certificate lifetime in hours is: 96 *.10 = 9.6 Hrs. This shows us that for at a 10% lifetime left it is still valid for over 8 hours. This certificate would always get at least one auto enrollment cycle to be run to renew the certificate. Assuming you don’t have anything else wrong, the renewal would happen successfully. This is the minimum lifetime that any certificate testing should be set to. Test, test, and more testing! Well, now you got your test lab, issuing short lived certificates and getting them automatically renewed, but… Now you are noticing that your application is not using the new certificate. Unfortunately, there are going to be some applications for which the automatic renewal might not work out well in your organization. However, the certificate could still be renewed manually by your IIS admins, and it would still not require the CA manager to be involved in the renewal process. The key thing is your IIS Admins would need to go through the manual certificate renewal process BEFORE the current certificate expires, as it is used to cross sign the renewal request. UPDATE 5/30/2024: After release of the blog it was brought to my attention of two different things to consider. First unknown to me (as I am not an IIS support engineer) apparently there is a setting within the IIS Management console to help with updating the site binding configuration so that this automatic renewal behavior would work. This feature is called Certificate Rebind and was new in IIS 8.5. https://learn.microsoft.com/en-us/iis/get-started/whats-new-in-iis-85/certificate-rebind-in-iis85 Second, we recently has a case were the customer had several websites running on an IIS Web Server. These websites all used unique certificates and were all generated manually using the same certificate template. When one of the certificates were ready to be renewed, they found that all website certificates were archived, and only one of the website certificates were automatically renewed. This would be by design behavior. The only thing that we could really recommend in this situation is to use one certificate for all websites and make sure that all the websites DNS names were added to the certificates Subject Alternative Name (SAN) field. Failure to do this will result in going through the entire manual issuance of the certificate all over again. And, going through the CA Manager approval process as well, since this would be a new certificate request. Hopefully, you found some good nuggets of information about how to lockdown certificate templates to protect against ESC1 vulnerabilities and what some of those less often used tabs do. And of course, how the certificate renewal process works, especially around shot lived certificates. Rob “I coulda been a Welder” Greene Extra credit for those who can figure out where the saying came from.NDES and the dreaded 2 & 10 Event ids stating “The parameter is incorrect"
Hey Guys Rob here again, today I am going to go over a set of typical Network Device Enrollment Service Event ID’s that you will inevitably encounter if you are maintaining an environment with NDES installed. These two events always seem to run together and can be seen in the Application Event log. Log Name: Application Source: Microsoft-Windows-NetworkDeviceEnrollmentService Date: DATE_TIME Event ID: 2 Level: Error User: NDES_APPLICATION_POOL_IDENTITY Computer: NDES_SERVER_COMPUTER_NAME Description: The Network Device Enrollment Service cannot be started (0x80070057). The parameter is incorrect. Log Name: Application Source: Microsoft-Windows-NetworkDeviceEnrollmentService Date: DATE_TIME Event ID: 10 Level: Error User: NDES_APPLICATION_POOL_IDENTITY Computer: NDES_SERVER_COMPUTER_NAME Description: The Network Device Enrollment Service cannot retrieve one of its required certificates (0x80070057). The parameter is incorrect. As you can see, although it is nice to see errors about a service or application, it does you no good if there is not enough information available to make something actionable about the event. Hopefully, this is where this blog will be helpful to all of you. The above errors happen for one of three common reasons. Access to the Private Keys for one or both Registration Authority (RA) certificates is not possible by the application pool identity account running the SCEP (Simple Certificate Enrolment Protocol) application pool. One or both RA certificates were NOT issued by the Certification Authority for which NDES is configured to forward Certificate Service Requests (CSR). The RA certificates are failing revocation checks. This means that either its certificates or one of the CA (certification authorities) certificates in the chain are failing revocation checks for some reason. If you try and access either the /CertSrv/MSCEP/MSCEP.dll or /CertSrv/MSCEP_Admin endpoints on the NDES Server you will also see an HTTP 500 error as well. Missing Private Key permissions Below are the steps for the first scenario to validate / add the application pool identity account. Private key Permissions: First, make sure you gave the Application Pool Identity account permission to the private keys on the newly issued certificates. Run: CertLM.msc Expand: Certificates - Local Computer\Personal\Certificates Click on the new certificate, and then right click on it, and select "All Tasks" Click on "Manage Private Keys" You should see the Permissions dialog box. Click the Add button, and type in the account running the SCEP Application Pool. Click the OK button. This account only requires "Allow" "Read" permissions. Once the permissions have been configured, click the OK button. Do this for all certificates that were recently renewed / issued. Open an elevated command prompt and type: IISReset Then, test accessing the website. If you are unsure what account is being used for the SCEP application pool, you can find this out by doing the following: Run: InetMgr.exe Navigate to: SERVERNAME\Application Pools Find the application pool named SCEP, and then look at the Identity column. NOTE: If the NDES role was configured with a non-domain service account and it is leveraging the Application PoolIdentity, please understand the ApplicatonPoolIdentity, and NetworkService are not the computer account. You will need to add NetworkService to the private key permissions for these two accounts. These accounts have very restricted rights on the system itself. Registration Authority certificates issued by wrong CA. The second scenario can only happen in a situation where you have more than one Certification Authority in the environment, where you have renewed the Registration Authority certificates, and one or both certificates were NOT issued by the Certification Authority that NDES is sending the certificate service requests to. The first thing we need to determine is what CA issued the two NDES certificates. Run: CertLM.msc Expand: Certificates - Local Computer\Personal\Certificates Look at the Issued By column for the current RA certificate. This will tell us the CA that issued the certificates. To find out the CA NDES is configured to use run: Regedit.exe Navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\MSCEP\CAInfo The registry value named Configuration shows you the CA computer \ CA Name that NDES is using. Validate that this is the CA that issued both RA certificates. If not, delete the certificates not issued by this CA and enroll again. NOTE: If you are using the MMC (Microsoft Management Console) to do the enrollment, you can specify the CA to use when you are filling out the information. You would click on the Certification Authority tab and select the CA to use. After procuring the NDES certificates from the correct CA, you must perform an IISESET from an elevated command prompt. One or both RA certificates are failing chaining or revocation checks. The third scenario is a bit trickier as most customers are not familiar with CAPI2 operational logging and how to interpret the data being provided. I am going to concentrate on looking at the NDES RA certificates to determine if they are failing a revocation check. By no means is this meant to be an exhaustive guide on how to use CAPI2 to troubleshoot chaining or revocation checking failures. The first two problems usually show themselves once the NDES has been in place for one or more years, and it failed just after replacing the existing NDES certificates. So, if everything was working before replacing the RA certificates, please review the two previous scenarios before jumping to an issue with certificate chaining or revocation checking. What is certificate chaining? Certificate chaining refers to the computer being able to take an end entity certificate and follow the chain all the way up to a Root CA certificate that is in the Trusted Root store of the computer. If the certificate cannot be chained to a Root CA certificate, then the certificate would not be considered a ‘Trusted’ certificate since the computer does not trust the root CA that issued the entire certificate chain. What is revocation checking? All certificates except Root CA certificates have a field on them called CRL Distribution Points (CDP). This lists different URLs that host a file known as a Certificate Revocation List (CRL). This file lists all issued certificates that, for several reasons, the CA Manager decided should no longer be trusted, and they wanted the world to know about this fact. Just like certificates, these CRL files have a finite lifetime restricting how long they can be used. Once the CRL’s Next update value has been reached it is no longer trusted, and the computer MUST download the newest CRL at that time. If it fails to download the CRL because the URL is not reachable, or the CRL has not been updated at the URL, then it will generate a revocation check failure. When the revocation check fails, the computer can no longer trust the certificate it was trying to use. In this case, it would mean that NDES would not trust the RA certificate and thus NDES would fail to start / run. Keep in mind that in a two-tier PKI hierarchy where you have an Offline Root CA and an Online Enterprise Issuing CA that issued the RA certificates, the following checks will happen: The RA certificate is going to be looked at to find the download locations of the Online Enterprise Issuing CA’s CRL. Then, we will validate that the RA certificate has NOT been revoked by the Online Enterprise Issuing CA Manager. The Online Enterprise Issuing CA’s CRL certificate is going to be looked at to find the download locations of the Offline Root CA’s CRL. Then, we will validate that the Online Enterprise Issuing CA certificate has NOT been revoked by the Offline Root CA Manager. As you can see, this can get complicated quickly depending on how many tiers the PKI hierarchy has within the environment. Enabling logging and data collection. First, we must enable CAPI2 Operational Event logging on the NDES server in question. Run: Eventvwr.msc Navigate to: Event Viewer\Applications and Services Logs\Microsoft\Windows\CAPI2\Operational Right click on Operational log, and we are going to do two things: Increase the log file size to at least 10000. A 1 MB log file is not going to be enough, and if the server is busy 10 MBs might not be enough, but it is a start. Enable the log. Click the OK button. A few commands need to be run in an elevated command prompt. IISReset This makes IIS (Internet Information Services) reload / reset and will cause the NDES application pool to try and load the certificates again once someone hits either the /CertSrv/MSCEP/MSCEP.dll or /CertSrv/MSCEP_Admin sites. b. CertUtil -SetReg Chain\ChainCacheResyncFileTime @Now This command tells CryptoAPI (CAPI) not to rely on the Crypto cache and instead attempt to access the real locations for AIA (Authority Information Access) and CDP locations. If these fail, it still allows the Crypto cache to be used so it will NOT cause an outage. It just helps with putting more error events in the log. c. CertUtil -URLCache * Delete This command tells CryptoAPI to delete everything in its FILE cache. This will NOT delete anything in its memory cache or per process Memory cache. 6. Lastly, try and access the /CertSrv/MSCEP_Admin page. You should see a HTTP 500 error, which is fine at this time, and you should also see the NetworkDeviceEnrollment events of 2 and 10 in the application event logs. Log Name: Application Source: Microsoft-Windows-NetworkDeviceEnrollmentService Date: DATE_TIME Event ID: 2 Level: Error User: NDES_APPLICATION_POOL_IDENTITY Computer: NDES_SERVER_COMPUTER_NAME Description: The Network Device Enrollment Service cannot be started (0x80070057). The parameter is incorrect. Log Name: Application Source: Microsoft-Windows-NetworkDeviceEnrollmentService Date: DATE_TIME Event ID: 10 Level: Error User: NDES_APPLICATION_POOL_IDENTITY Computer: NDES_SERVER_COMPUTER_NAME Description: The Network Device Enrollment Service cannot retrieve one of its required certificates (0x80070057). The parameter is incorrect. 7. Once the issue is reproduced, I suggest you return to the CAPI2 Operational Event log properties and disable it. We do not want other CryptoAPI calls happening on the server to push or overwrite the data in the event log. The CAPI2 event logs could have quite a few events in them. A good event ID based filter to start with is to only show the following events: 11,30,41-42,51-53,90 To find out what events are of interest for the Registration Authority (RA) certificates, you will want to do a Find in the event logs for either the thumbprint value of the certificate or the subject name of the certificate. Example of a common RA certificate subject name is: ADATAM-WEB01-MSCEP-RA The subject name is usually defined as the NDES servers name-MSCEP-RA. Once the certificate has been renewed at least you will want to validate the current subject name is of the Exchange Enrollment Agent (Offline request), and CEP Encryption based templates that are in use. CAPI2 Events of Interest It is well to understand a little bit about the different CAPI2 events that you are going to see in the event log that are related to chaining and revocation checking: Event ID 90 X509 Objects (X509Objects) - Shows all the certificates, CRL’s and OCSP (Online Certificate Status Protocol) Responses it was able to collect either via the certificate stores, or via CryptoAPI cache. This is a good event to review to see if the OS (operating systems) found all certificates in the chain or not. If you do not see a required certificate, then the chaining function will not succeed. This event also shows more detail about each certificate than the other events in the log. Event ID 11 Build Chain (CertGetCertificateChain) - Shows if the certificate chains to a valid root certification authority. In addition, it does revocation checks to see if all certificates in the chain succeed or fail their revocation check. Event ID 30 Verify Chain Policy (CertVerifyChainPolicy) - First thing with this event is to determine what Policy is getting verified. There are several types of policy checks that this event will check against (See CertVerifyChainPolicy link above for the list of policy checks). Given the policy it will either show success or failure. The policy check could pass and still show as a failure if Event ID 11 fails because of a revocation check failure you will see this same failure here. Event ID 41 Verify Revocation (CertVerifyRevocation) - Shows what CAPI2 knows about the status of the CryptoAPI cache in reference to revocation information. Event ID 42 Reject Revocation Information (CertRejectedRevocationInfo) - Shows that CryptoAPI cached data is being rejected as it is either stale or needs to go off system to get the latest CRL / OCSP response from the network. Event ID 53 Retrieve Object from Network (CryptRetrieveObjectByUrlWire) - Shows the status of attempting to access a specific AIA or CDP URI (Uniform Resource Identifier). It will give you the call status too. Example of troubleshooting with CAPI2 logging enabled First filter the collected CAPI2 event log with the following: 11,30,41-42,51-53,90 Click on Find and type something unique about the certificate. Either Subject Name or thumbprint value. We can see the first instance where the subject name is found, and it is shown as an error. When looking at these events you want to have the Detail, Friendly View when reviewing the entries. 4. Now when looking at the event, we will be interested in looking at multiple events in the logs to determine what is going on. Typically, in this type of problem, you want to look at the events in the following order: 90, 11, 30, and lastly 53. 5. We can see in event 11 that this is failing a revocation check. You will want to pay attention to the TrustStatus field in the Details section. The first TrustStatus is the overall TrustStatus. This tells you about the entire chain and specifically that one of the certificates in the trust path failed revocation. Below the overall TrustStatus, it will show each individual Chain Element (each certificate in the chain) in the certificate trust path and its TrustStatus. From looking at the above, we can determine that the ADATAM-WEB01-MSCEP-RA certificate is the one that is failing the revocation check. This means we need to look at the CA that issued this certificate and validate that its CRL is reachable and valid at the URIs (Uniform Resource Identifier) in question. If the PKI hierarchy has more CA’s, you may discover that the RA certificate is valid and that the Issuing CA’s certificate failed validation. If that is the case, it would mean that there is an issue with the Root CA’s CRLs (certificate revocation lists). 6. We are not going to look at the Event 30s as there are no policy checks that would be validated in the context of NDES RA certificates being valid or not. 7. Next we would typically jump right to Event 53s to see what might be going on with accessing the CRL / OCSP URLs. First thing to look at is the URL in the event. This tells us what path it is trying to access: The next part is we can see the HTTP Request and HTTP Response from the OCSP Server. It was an error of HTTP 503. This tells us there is an issue with the OCSP Server that must be addressed to resolve the NDES problem. 8. Another example is trying to access a Certificate Revocation List (CRL) file. It was able to successfully download the CRL file as evident by the following HTTP 200 (OK): 9. But after successfully downloading the CRL file we see Event 42 error. So, we can see that it is stating that the CRL at the HTTP URL path is no longer valid. Usually, when an HTTP 500 error is seen and is related to revocation checking, it is an unreachable or expired CRL. Most of the cases I have seen where this is the issue, it is that the Root CA’s CRL that has expired, and the customer has forgotten to boot the Root CA and publish the new CRL that gets created at service start.Designing and Implementing a PKI: Part II Implementation Phases and Certificate Authority Installation
First published on TechNet on Oct 13, 2009 The series:Designing and Implementing a PKI: Part I Design and Planning Designing and Implementing a PKI: Part II Implementation Phases and Certificate Authority InstallationDesigning and Implementing a PKI: Part III Certificate Templates Designing and Implementing a PKI: Part IV Configuring SSL for Web Enrollment and Enabling Key ArchivalDesigning and Implementing a PKI: Part V Disaster RecoveryChris here again.Renew Certificate Authority Certificates on Windows Server Core. No Problem!
Hi there! Rob and Jim are here from the Directory Services team. Today’s blog strives to clearly elucidate an administrative procedure that comes along more frequently with PKI Hierarchies being deployed to Windows Server Core operating systems. Installing the Certificate Services Role on Windows Server Core will not be covered in this blog, but this is good reference for this endeavor - https://learn.microsoft.com/en-us/powershell/module/adcsdeployment/install-adcscertificationauthority?view=windowsserver2022-ps In our scenario we already have an OFFLINE ROOT and an Enterprise Subordinate CA certificate that needs to be renewed. Both of these PKI roles are installed on the Windows Server Core operating system. To start the renewal process, validate if the customer has the following registry value in place so we know if / where the Certificate Signing Request (CSR) file is going to be written to. HKLM\System\CurrentControlSet\Services\CertSvc\Configuration\CA Name Value Name: RequestFileName Value Name: ParentCAMachine Value Name: ParentCAName These registry settings control where the CSR file and name will be located if they are specified. Make sure that the folder path already exists before moving forward. GENERATING A CSR WILL NOT create a missing folder. If the Root CA is an Enterprise Root CA (domain joined) the CSR creation will use the two Parent registry values to submit the certificate request to this Root CA. However, these values will not work and will probably not be present when utilizing an Offline (Standalone) Root CA. 2. In an elevated command prompt on the subordinate Issuing CA run the following command after deciding if reuse of the CA’s existing private key is in order or if a new private key should be generated: CertUtil -RenewCert [ReuseKeys] If you want to generate a new private key for the Subordinate CA, then type: If you want to generate a new private key for the Subordinate CA, then type: CertUtil -RenewCert If you want to use the existing private key for the Subordinate CA, then type: CertUtil -RenewCert ReuseKeys Additional information on CA certificate renewal options can be found here - Certification Authority Renewal - Win32 apps | Microsoft Learn Copy the resultant CSR .req File over to the Root CA. Now we can submit the request that we just copied to The Root CA which is also running on Windows Core OS. We are going to use the Certreq.exe command to submit this request to the Standalone Root CA. CertReq -Submit -Config "RootCAComputerDNSName\RootCAName" SubCACSRFIleName.req Example of the command line. Root CA Computer name: FourthCoffeeCA01.FourthCoffee.com Root CA Name: FourthCoffee Root CA 01 Sub CA CSR File Name: FourthCoffeeSubCARenewal01.req CertReq -Submit -Config "FourthCoffeeCA01.FourthCoffee.com\FourthCoffee Root CA 01" FourthCoffeeSubCARenewal01.req IMPORTANT: You should get a Request ID as part of the output. You will need to make a note of this for forthcoming steps. 5. If the CA Manager needs to approve the CSR, this can be accomplished via the certificate services management UI (cetsrv.msc) if possible as it is easier. However, since the Root is also running the Windows Core OS, we must run the following command: CertUtil -Resubmit RequestIDNumber Example of the command line: Request ID from step 4: Request ID = 3 CertUtil -Config "FourthCoffeeCA01.FourthCoffee.com\FourthCoffee Root CA 01" -resubmit 3 6. We next need to retrieve the CER/CRT file from the Root CA so that we can install the certificate on the Subordinate CA to complete the renewal. CertUtil -View -Restrict "RequestID=RequestIDNumber" -out RawCertificate > C:\FourthCoffeeSubCACert.cer 7. Assuming the Root CA's certificate has not been renewed, we just need to copy the resultant FourthCoffeeSubCACert.cer file back to the subordinate CA that is being renewed. 8. Back on the subordinate CA in an elevated command prompt we then need to install the subordinate CA's certificate. Using the following command: CertUtil -InstallCert CACertFileName Example: Certutil -InstallCert FourthCoffeeSubCACert.cer When this command is run the Certificate Service Service on the subordinate CA will start. We hope this blog will take some of the mystery and challenge out of interacting with Microsoft PKI running on Windows Server Core. Robert “what were you thinking” Greene and Jim “that’s for the birds” Tierney.
