Blog Post

Core Infrastructure and Security Blog
4 MIN READ

Building the Totally Network Isolated Root Certification Authority

DanielMetzger's avatar
DanielMetzger
Icon for Microsoft rankMicrosoft
Feb 23, 2020

 

Hello everyone, my name is Daniel Metzger and I am a Senior Premier Field Engineer for Secure Infrastructure based in Switzerland. Lately I have done quite a few Public Key Infrastructure (PKI) migrations for customers mostly because their certification authorities (CAs) were running on Windows OS versions which were approaching end of support. Other customers had their root CA installed on a domain controller which needed to be replaced with the CA being migrated to another host as a prerequisite for demotion.

 

While talking customers through the migration process during scoping, in each of this cases we established the fact that the existing PKI did not match Microsoft's recommendations. Several customers just had a 1 tier PKI with the root CA and its private key being exposed to the LAN while others had a 2 tier PKI with a standalone root CA attached to the LAN, too. So each time the question was raised how to build a truly offline root CA following best practices.

 

The answer sometimes surprises them: Use a Windows 10 laptop and convert it to a secure Hyper-V host to run the offline root CA:

 

 

 

 

The solution proposed to customers meets the following standards:

  • The offline root CA is virtualized and runs on a dedicated, secured host system
  • The offline root CA is operated from a dedicated administrative workstation only
  • The private key of the root CA is protected in a hardware device

 

Prerequisites

 

The solution requires the following components:

  • A brand-new laptop which out of the box is never attached to any network by wire or wireless, running the latest version of Windows 10 Enterprise with BitLocker and the Hyper-V role enabled
  • A guest VM running the latest Windows Server Standard version acting as the offline root CA
  • Some trusted, access controlled, brand-new USB sticks to transport data to and from the root CA (a.k.a. certificate requests, issued certificates for subordinate CAs, CRLs)
  • A phone to activate both the host and the guest systems

 

Both the installation sources for Windows 10 Enterprise and Windows Server must be completely trusted, following the clean source principle:

 

https://docs.microsoft.com/en-us/windows-server/identity/securing-privileged-access/securing-privileged-access-reference-material#clean-source-principle
"Active Directory administrative tier model" -> Clean source principle

 

I would strongly recommend downloading a fresh copy of the installation media from Microsoft at least twice, better three times, using different download hosts and connections. Then compare the hash values of the downloads. If they are all the same you can safely assume that all resulting downloads have not been altered while in transport thus, they are trustworthy. Store the installation media files in a safe and secured location – just some network share somewhere with some space left will not do.

 

Installation steps

 

Most of the steps to install an offline root CA are covered here:

 

https://gallery.technet.microsoft.com/Windows-Server-2016-Active-165e88d1
"Windows Server 2016 Active Directory Certificate Services Lab Build"

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/hh831574(v%3Dws.11)
"Certification Authority Guidance"

https://social.technet.microsoft.com/wiki/contents/articles/2900.offline-root-certification-authority-ca.aspx
"Offline Root Certification Authority (CA)"

 

The Windows 10 Enterprise host must be prepared and set up in the most secure way. Since it has and will be never attached to a network the installation needs to be done locally like from a DVD with the clean source installation media. We do not install any security updates or additional software like antivirus, etc., since the host and its contents never get exposed. OS activation is done by phone.

The hardware has to support the latest Windows 10 security features like Credential Guard, Device Guard and BitLocker with TPM 2.0. You can apply local versions of the Security Compliance Toolkit (SCT) for Windows 10 but since the device never touches any network this is more of a nice to have than a real must.

 

Next would be to copy the installation media for Windows Server to the host and to build a Version 2 VM hosting the offline root CA.

 

 

 

 

Do not connect the VM to any virtual switch and disable Checkpoints as they are not recommended for virtualized CA hosts (reversing the state of a CA actually can create a disaster situation) and will prevent access to the virtual disk from the physical host as required later on.

 

 

 

 

OS activation is again done by phone.

 

To transport files to and from the offline root CA, my customers shut down the guest OS and attach the virtual disk to the hosts file system.

 

 

 

 

 

 

 

 

Then they eject the virtual disk so the VM is able to start next time it is needed. Any removable storage used for this purpose is tested, requires BitLocker for write access and is access protected with a PIN or similar. And of course, it has never been in use for other tasks.

 

To access the host for the offline root CA, a minimal number of local accounts need to be created. Only these accounts are permitted to log on and to operate the guest VM. The same applies to the VM itself.

 

I recommend initiating a backup of the CA configuration and database each time a new certificate or revocation list is created. The private key is backed up once each time a new root CA certificate is issued and stored on a secured removable storage device as are the CA backups. The physical CA host and backup storage devices are then stored in a safe and protected location with very restricted access. This could be a safe.

Updated Feb 24, 2020
Version 2.0

6 Comments

  • Jero's avatar
    Jero
    Copper Contributor

    Hi Daniel,
    You've mentioned that customers had their root CA installed on a domain controller which needed to be replaced with the CA being migrated to another host as a prerequisite for demotion. What's the best way to keep the DC and move the CA role to a new server?

  • jorngrotnes's avatar
    jorngrotnes
    Copper Contributor

    Daniel, isn't the answer of the first comment that the hidden root device needs a hardware security module, and having that is indeed the case for a modern Win10 workstation/PC?

    As I understand, we could build an Azure based virtual environment and purchase PAWs to manage it, while keeping the original root offline (the physical Win10 PC), right? I would like to suggest this cloud based PKI setup to a client that is in the process of expanding their server environment into the cloud, where the case for a on-prem PKI is less attractive. If there is a white-paper or something on recommendations for Azure PKI (you have already pointed me in the right direction above) it'd make me happy to get any pointer.

     

  • You would need to communicate with the Root CA host somehow to manage it (CRLs, Issuing CA certificates, etc.) as you would not have physical access. If you run the Root CA directly as an Azure VM or with nested virtualization on a Hyper-V host in azure, you must have a way to touch it. Even when it would be located in its own Vnet, you have to get to it from another system, whether this would be an Azure Bastion (built in the same Vnet) or from another VM or physical machine using one of the management ports. The weakness here is the host you are coming from if it is not a clean sourced Privileged Access Workstation (PAW-CSM) which we build for customers today using Azure AD join, Conditional Access, AutoPilot, Intune and Microsoft Defender for Endpoint. If it's just a common workstation or management server you are coming from, it would not be trusted which makes the connection to the Root CA not trustworthy, so you had to assume breach of the Root CA. It would also mean that the Root CA never was truly offline.

     

    If you place an Issuing CA in Azure but not the Root CA, it would be a different story. Still you need to treat it as Tier 0 entity and place it in the dedicated Identity subscription of a proper Azure Landing Zone (deployed from one of our templates). PAWs would also be required, but your Root CA would stay offline as described in the article. 

  • jxfrey's avatar
    jxfrey
    Copper Contributor

    DanielMetzger what would your advice be if we'd like to install a new PKI and would like to do this entirely in Azure. There are ways in completely securiing and isolating Root CA on azure through network, rbac, backup/recovery, monitoring and alerting... however, the use of HSM would not be possible. what is your take on this? Ultimate goal would still be to have 2-tier PKI with an offline root CA. Thank you in advance.

  • The article deals with a root CA built on a laptop. How to incorporate HSM is out of scope for this article, but you might approach your HSM vendor.

  • Due_Zeh's avatar
    Due_Zeh
    Copper Contributor

    How do you fullfill the last requirement in a virtualized enviroment:

    • The private key of the root CA is protected in a hardware device

    Do you use a HSM to secure the private Keys and if it's the case, how do you configure it?