My name is Ron Arestia, and I am a Security Researcher with Microsoft’s Detection and Response Team (DART). We respond to customer cybersecurity incidents to assist with containment and recovery from threat actors. In this blog post, we will be covering the basics of implementing an Active Directory Certificate Services (ADCS) offline root Certificate Authority (CA) and how to manage it securely. This is the first in a series of follow up posts to my original article reviewing Secure Configuration and Hardening of Active Directory Certificate Services.

In Part 1 of our series, we will focus on some high-level security discussions around your offline root certificate authority before you even begin installing the operating system.

Secure Source, Secure Start

Confidence in your PKI begins with a secure source

Are you familiar with the concept of a “secure source?” The term itself might seem a bit nebulous, so let’s unpack the concept contextually with general understanding and expand to Public Key Infrastructure (PKI).

In the larger context of “zero trust,” think of secure source as generally meaning a trusted publisher of some set of data such as software. Microsoft publishes the Windows operating system, and the binaries for those operating systems are available from locations such as the Volume Licensing portal or Visual Studio Enterprise. Each download is published with a corresponding SHA hash.

Why? If you use a tool like Get-FileHash in PowerShell against the downloaded binary, it will output a hash of the file that is unique to the data therein. The software publisher, Microsoft in this case, has gone through this same process prior to publishing to verify that the data in the binary has not changed. You can use any hashing tool to check your file integrity the same way:

This file is identical to the file published by Microsoft, and it can be considered a true source as published. If the source file was manipulated in transit or after download, the hashes would not be the same. Using more complex hashing algorithms, such as SHA256, provides greater confidence that the file has not been manipulated, and thus, you can trust that this is the operating system published by Microsoft. This is a secure source file.

Using this source ISO ensures that you are using a true copy of the Windows operating system as provided by Microsoft. This can be extended to any modern operating system available for download (e.g., Linux). Now that you have a secure source file, take a look at your hardware. Are you going to use a physical server to house your root certificate authority (CA)? Or are you going to install the root CA as a virtual machine (VM) on an existing hypervisor?

If you’re using a physical server, is the platform brand new from a trusted distributor, or are you piecing together a platform from spare parts? In the interest of secure source, I’d recommend you defer to the former, but if you’re confident in your spare parts inventory, you can use a previous platform with the caveat that you update all of the applicable firmware on either the new platform or the old. Why?

Physical servers by major distributors such as Cisco, Dell, and HP are a complex set of hardware platforms including power management, remote access controllers, disk controllers, and mainboards. Since the physical server isn’t currently in use, it’s the best time to update all of the firmware and prepare all of the most up-to-date software drivers for your installation to ensure common platform security issues are patched to current. A truly offline CA likely won’t suffer a catastrophic security compromise but taking pains to update your architecture prior to its implementation is the best strategy to ensure you aren’t scrambling to secure your platforms under duress in the future. As with the operating system, make sure you are comparing hashes for all of your firmware and software updates and documenting those hashes for future security or compliance audits.

If you are planning to implement an offline root CA on a virtualization platform, you are necessarily expanding your risk footprint unless you are installing the root CA virtual machine (VM) on a tier 0 hypervisor with strict access controls. Why?

As discussed in my first post in this series, and as we will discuss later, the private key of your root CA is the most important logical piece of information for your public key infrastructure (PKI). Protection of the private key is tantamount to every other operation for your PKI. If your private key lives on the local disk of your server, who has access to that hard drive? In a virtual environment, you’re sharing physical disks with other servers. Are your VM infrastructure engineers trusted? If a threat actor compromises the account of one of those engineers, they could easily exfiltrate the VMDK or VHDX of your offline root and have their way with the private key for your root CA. Consider it best practice, when virtualizing your root CA, that the virtual machine live on a dedicated tier 0 hypervisor with limited, controlled access. If you have a dedicated platform for your other tier 0 assets, your root CA can live therein with the proper controls in place.

If you don’t have a tier 0 hypervisor, you still have options. On a trusted workstation such as a secure/privileged access workstation (SAW or PAW) running Windows, you can install the Hyper-V role and setup a VM to run the offline root CA. In my experience, you still want to secure and segregate the disk, so installing a dedicated disk using a USB disk controller or even just a thumb drive to house your root CA installation is preferred. This also gives you the ability to lock the physical disk in a fire safe to control physical access to the private key.

It’s also possible to take a physical desktop or laptop slated for decommissioning giving it a new life as your root CA. This physical system can be locked in a secure location for safekeeping the same as the hard drive in the previous example albeit requiring more physical space which is why we usually see customers using a USB drive on a trusted system to run the root CA. The length of time required for this system to be setup, perform the required signing operations, and shutdown for safekeeping should be less than a day.

All of these ideas are tangential to the conversation of secure source, but it’s important to note that the secure source mentality should extend to the hardware you choose to use for your offline root CA. For instance, you would never use a laptop that was previously removed from inventory under suspicion of housing malware. Even if you completely wiped the disk, replaced the RAM, updated the UEFI/BIOS, there’s still a non-zero chance that the platform isn’t safe. Security of your offline root CA platform should be top-of-mind in every decision made.

Finally, your root CA installation and setup should follow a clean source/clean keyboard mentality all the way through to go live. The system should never be attached to a network in any way. (This introduces some complications in a virtualization environment even if you never connect a virtual adapter to the VM; the hypervisor itself is connected to some network, introducing some risk even on a sanitized tier 0 platform.) If you’re not leveraging a hardware security module (HSM) for the private key, the disk should be completely sequestered from any environment where any untrusted identity could access it or any of its ephemeral existence (i.e., backups, checkpoints, snapshots). And you want to ensure that you have dedicated but secure access to console to manipulate the operating system during setup and for future use. This should include a strategy to move certificate signing requests (CSRs), certificate revocation lists (CRLs), and authority information access (AIA) data such as the root and issuing CA certificates to and from the new root CA.

The Private Key (Why You Should Be Using HSM)

The most important logical piece of your PKI should not be on disk

I want to impress upon our customers the importance of the idea that the root certificate authority (CA) private key is the most important logical piece of information for your public key infrastructure (PKI). What does this mean?

Think of the private key for your root CA as the key to the front door of your home. If you own your home, you are the sole responsible party for the safety of that key. If you purchased a brand-new home, you’re likely the only person who has that key. You will be very prescriptive with whom you share that key. Your spouse or significant other, responsible members of your immediate family, maybe even close friends or neighbors will have a copy of that key to help you maintain or watch over your home while you’re away or for your safety if you live alone. If you purchase a pre-owned home, you will likely prioritize changing the locks of your new home, because you don’t know the people who might have a copy of that old key, and you will distribute that new key to individuals named previously. You should feel safe and secure in your own home.

Similar to the key to your home, the private key of your root CA should be safeguarded with the same zeal. If the private key of your root CA is in the hands of an individual or entity outside of your knowledge or control, the ability to sign certificates and certificate revocation lists (CRLs) is not exclusively under your control. If you’re not scrutinizing who is in your enterprise environment, anyone with a copy of that key could stand up a root CA, grab a copy of your root CA certificate, and use those two pieces of data to create certificates that look and act the same as certificates issued by your legitimate, known root CA. They can also generate CRLs that could be accepted as legitimate by your existing PKI sans certificates that you might have revoked for any reason. If every system in your enterprise trusts a single root CA, and that single root CA now exists as a rogue system in your environment, there’s nothing to prevent a threat actor from masquerading as a legitimate purveyor of certificates in your environment: server authentication certificates, user authentication/smart card certificates, and even code signing or trusted publisher certificates.

The private key is the single most important logical piece of data for your PKI. This data is stored by default on the system drive in your Windows Active Directory Certificate Services (ADCS) CA. A simple Internet search will reveal common locations for this key, so I will not publish that through this forum in deference to security.

Using certutil, however, you can backup the private key of your root CA, and it will exist on the local hard disk available to anyone with access to that system programmatically or physically through ownership of the disk or partition.

(This includes via access to a virtual disk under virtual infrastructure.)

Keeping the private key on the local hard disk of your root CA is akin to leaving your house key under your door mat. Is it safe there? Maybe, but there’s a non-zero chance that an untrusted person could simply lift the mat and gain access to your home. Keeping your root CA truly offline is arguably a better strategy and critical if you do not plan to implement a hardware security module (HSM). To continue with the house key analogy: it’s merely a clever way to obfuscate the key like one of those hide-a-key decorative rocks or placing the key in a more obscure location like under a sprinkler donut.

So why should you invest in a hardware security module (HSM)? The most common reason to not use HSM is cost, but several retail companies provide USB HSM devices for under US$1,000 which makes the availability much more financially viable for small businesses. For larger businesses, I would recommend enterprise-class hardware usually in the form of a tamper-resistant rackmount device with some sort of authorization to access the data thereon, ideally through the use of multifactor authentication (MFA). Cloud providers such as Microsoft are also providing HSM-as-a-service (HSMaaS) with comprehensive integration capabilities you should also investigate if you are a cloud-only enterprise or looking to reduce your on-premises datacenter footprint.

The HSM is a standalone platform dedicated to performing complex cryptographic operations such as random number generation, key generation, encryption and decryption, signing, and hashing. By offloading these operations to a dedicated platform, you are reducing the need to perform them on individual servers, possibly saving you processor time on individual servers and centralizing cryptographic operations on a secure platform that is physically and logically hardened against attackers. A driver and application programming interface (API) are used by the dependent system, such as your certificate or registration authorities, to perform key generation and access without exposing the key information. In the case of your offline root CA, this means that the generation of your private key and signing of certificate signing requests (CSRs) and certificate revocation lists (CRLs) with that key are not performed by the CA itself but through an interface to the HSM whereupon those operations are performed safely and the generated outputs passed to the CA for further operations. Since the private key never exists on the root CA in any form, the ability for a threat actor to gain access to this critical data is impossible.

Note: HSMs are cryptographic devices subject to export controls. It’s possible that your country has strict regulations for these devices. Please familiarize yourself with your state or national laws concerning these devices and make choices appropriate for your enterprise.

Without the private key of your root CA or any of your registration authorities (RA) or subordinate/issuing CAs, even the most advanced persistent threats would find it impractical to even try to masquerade as the same or perform any trusted cryptographic operations in your environment. They would move on to other high value targets in your environment not otherwise entangled with your PKI.

One last important note: implementing HSMs in your environment does not negate the need for your root CA to be truly offline nor does it abrogate any of your administrative responsibilities to best practice adherence. They are advanced tools designed to provide a high level of assurance for your organization’s cryptographic operations, and as with any tool, there are avenues for misuse, misconfiguration, and abuse without proper training and diligence.

Consult your Microsoft and HSM professionals for guidance on how to integrate HSM into your organization. Whether you’re starting fresh with PKI or you have a mature implementation, HSM should be considered universally for any security practitioner.

That’s going to conclude Part 1 of our series on securing your ADCS offline root CA. In Part 2 we will discuss guidelines for CRL and AIA publishing and common root CA deployment scenarios.