[Today's post is contributed by Carol Bailey ]
A number of certificate deployment issues I see are down to timing issues. Like other distributed systems with many moving parts, Microsoft Certificate Services often requires more time and patience to do its job than administrators are prepared to give it. And yet this is a little odd, because SMS wasn't affectionately known as "Slow Moving Software" for nothing. Seasoned SMS admins know that packages can take hours to install on all their distribution points, and that an urgently needed application might install in the next few minutes or hours. But tasked with deploying certificates and we suddenly become signed up members of the "I want it now" society.
I think a good solution to dealing with these potential timing issues is planning well in advance, and having the confidence to know that you've configured everything correctly - and a dollop of experience also doesn't go amiss. When you don't have as much PKI experience under your belt as you do with SMS, what's the best strategy when it comes to deploying the native mode certificates?
My first tip is to never practice on the production network when you're using enterprise CAs - it's a false economy. It sounds obvious, but many people do this, thinking that they can easily uninstall the CA if necessary and when templates or certificates fail to show up they don't know whether it's down to misconfiguration or timing.
Aim to never uninstall an enterprise CA (and some settings can only be configured at installation time) because it writes to Active Directory. Your mileage may vary but I've had some bad experiences with computers trying to use an uninstalled CA that's hanging around in the background and had to clean up Active Directory manually. If you have to uninstall an enterprise CA, be sure to first carefully read KB 889250 ("How to decommission a Windows enterprise certification authority and how to remove all related objects from Windows Server 2003 and from Windows 2000 Server"). Even the title gives you a hint that this is not in the same category as uninstalling IIS or other server roles - and that's before you start to wade your way through all the procedures.
Trying to diagnose a PKI problem that's due to an invisible CA is a big time waster, so don't chance it, and don't gift it to another consultant or support person so that they unknowingly inherit this scenario. Instead, know exactly what settings and configuration your enterprise CA will need, then test, test, test on an isolated network until you can do it in your sleep with one hand behind your back and hopping up and down - and only then install for real on the production network.
After installing the enterprise CA(s), allow plenty of time for all computers in the forest to receive the associated certificates through group policy. Although they will be installed pretty quickly for your locally connected computers, there will usually be some computers across slow links, currently turned off, and currently off the network. These certificates are the foundation of your PKI solution and without them correctly installed on all the computers that need them, native mode will fail. For example, native mode clients do not check that their own certificate chains to a trusted root CA, which means that they can present it to their management point before the certificate chain is installed. However, if the same CA hierarchy is being used for the management point, the client also won't have the root CA to validate the management point's certificate (or the site server signing certificate), which means that communication will fail.
The same inherent delays apply when configuring certificate templates and deploying certificates. You're less likely to see these delays on your isolated network because everything is well connected with fewer computers - but even then, you might experience a few timing issues. A useful tip that was passed on to us was to use the "Certutil -pulse" command. I'm told that this works even when "gpupdate /force" doesn't and certificates (and new templates) that were dragging their feet suddenly turn up.
During the testing phase, when you're reconfiguring certificate templates that you've already issued on the CA, it's not necessary to delete them and add them again to the CA after the change. The changes will be picked up automatically, but the delete and add method does speed up the process through the new publication to Active Directory.
Deploying certificates through auto-enrollment is neat, but not instantaneous. Even on a test network where I've had Active Directory and the CA on a single server with a couple of computers, it's taken a while for the certificates to install even after a round of rebooting. How long is a while? Anything from five minutes to half an hour. And on a production network, it can be hours - which is why it's important to be confident that the configuration steps are correct before you move them onto the production network.
When auto-enrollment doesn't appear to work in the first few minutes on a production network, it's very natural to assume that something has gone wrong and start to fiddle - which is the worst thing you can do. When you know that the configuration is correct (because it worked on the test network), it's time to step away from the computer. I have a good example of this when I worked with a customer who was configuring a CA across a WAN link and couldn't get his client certificates to install through group policy. We went through the certificate template configuration and group policy configuration, rebooted, and waited. Nothing happened. We chatted some more and waited a bit longer - still nothing happened. I knew it was configured correctly because I had done it so many times before. Eventually I convinced him to leave it alone and ring me in the morning if the certificates still weren't installed. The following morning he didn't ring me, but I did get an email saying they had finally turned up.
Because auto-enrollment for client certificates can take some time, it's a good idea to deploy these first on your production network - before the server certificates. Having that extra time also helps to catch the few computers that were turned off because of vacations, or laptops that were off the intranet.
The step-by-step guides for the native mode certificates in the Configuration Manager library don't start with the client certificates. The order in these guides is the site server signing certificate first, followed by the Web server certificates, and then finally the client certificates. The main reason why I decided on this order was because the step-by-step is a very long document, and it made more sense to have the certificate deployment in the order in which I thought most people would have problems. The site server signing certificate with document signing capability and a unique string in the Subject was a puzzler even for experienced PKI admins, and without this certificate working, it's game over for native mode. So I put this first in case that was the only information that was needed. Deploying the server certificates was the next hardest, because the default Web server certificate template doesn't automatically populate the certificate Subject. And finally, deploying client certificates was the easiest because you can use certificate templates with very little configuration. I should also re-emphasize that the step-by-steps are specifically designed for a test network where the delays should be minimal. So I still think that this is the right order for step-by-step guides, but in production I recommend you deploy them in reverse order for timing reasons.
Braindump complete, I'll try to summarize some of these tips and tricks:
- Certificate deployment using an enterprise CA- like Configuration Manager - has inherent latencies that should be expected and planned for.
- Practice installing a CA and deploying certificates on a test network - never practice on the production network.
- Aim to never uninstall an enterprise CA.
- Allow plenty of time for the root CA certificate and any subordinate CA certificates to trickle down to computers in the forest.
- Don't be surprised if newly created certificate templates and certificates deployed through auto-enrollment don't work immediately - but try "Certutil -pulse" to kick it up a notch.
- Having successful testing experience under your belt will give you the confidence to wait rather than assume a problem.
- When deploying the certificates for native mode, start with the client certificates first, then the Web server certificates, and finally the site server signing certificate - the reverse of the order in the step-by-step guides.
For my final thought about Certificate Services and timing, I'll quote a Chinese proverb: Patience is power; with time and patience the mulberry becomes silk. I hadn't really thought of a PKI deployment as being silk before, but it's an interesting analogy.
This posting is provided "AS IS" with no warranties, and confers no rights.