Azure Key Vault Replication: Why Paired Regions Alone Don’t Guarantee Business Continuity
As customers modernize toward multi‑region architectures in Azure, one question comes up repeatedly: “If my region goes down, will Azure Key Vault continue to work without disruption?” The short answer: it depends on what you mean by “work.” Azure Key Vault provides strong durability and availability guarantees, but those guarantees are often misunderstood—especially when customers assume paired‑region replication equals full disaster recovery. In reality, Azure Key Vault replication is designed for survivability, not uninterrupted write access or customer‑controlled failover. This post explains: How Azure Key Vault replication actually works (per Microsoft Learn) Why paired‑region failover does not equal business continuity Two reference architectures that implement true multi‑region Key Vault availability, with Terraform How Azure Key Vault Replication Works (Per Microsoft Learn) Azure Key Vault includes multiple layers of Microsoft‑managed redundancy. In‑Region and Zone Resiliency Vault contents are replicated within the region. In regions that support availability zones, Key Vault is zone‑resilient by default. This protects against localized hardware or zone failures. Paired‑Region Replication If a Key Vault is deployed in a region with an Azure‑defined paired region, its contents are asynchronously replicated to that paired region. This replication is automatic and cannot be configured, observed, or tested by customers. Microsoft‑Managed Regional Failover If Microsoft declares a full regional outage, requests are automatically routed to the paired region. After failover, the vault operates in read‑only mode: ✅ Read secrets, keys, and certificates ✅ Perform cryptographic operations ❌ Create, update, rotate, or delete secrets, keys, or certificates This is a critical distinction. Paired‑region replication preserves access — not operational continuity. Why Paired‑Region Replication Is Not Business Continuity From a reliability and DR perspective, several limitations matter: Failover is Microsoft‑initiated, not customer‑controlled No write operations during regional failover No secret rotation or certificate renewal No way to test DR Accidental deletions replicate No point‑in‑time recovery without backups Microsoft Learn explicitly states that critical workloads may require custom multi‑region strategies beyond built‑in replication. For many customers, this means Azure Key Vault becomes a single‑region dependency in an otherwise multi‑region application design. The Multi‑Region Key Vault Pattern The two GitHub repositories below implement a common architectural shift: Multiple independent Key Vaults deployed in separate regions, with customer‑controlled replication and failover. Instead of relying on invisible platform replication, the vaults become first‑class, region‑scoped resources, aligned with application failover. Solution 1: Private, Locked‑Down Multi‑Region Key Vault Replication Repository: 👉 https://github.com/jclem2000/KeyVault-MultiRegion-Replication-Private Architecture Highlights Independent Key Vault per region Private Endpoints only No public network exposure Terraform‑based deployment Controlled replication using Event Based synchronization What This Enables ✅ Full read/write access during regional outages ✅ Continued secret rotation and certificate renewal ✅ Customer‑defined failover and RTO ✅ DR testing and validation ✅ Strong alignment with zero‑trust and regulated environments Trade‑offs Higher operational complexity Requires automation and application awareness of multiple vaults Solution 2: Low‑Cost Public Multi‑Region Key Vault Replication Repository: 👉 https://github.com/jclem2000/KeyVault-MultiRegion-Replication-Public Architecture Highlights Independent Key Vault per region Public endpoints Minimal networking dependencies Terraform‑based Controlled replication using Event Based synchronization Optimized for simplicity and cost What This Enables ✅ Full read/write availability in any region ✅ Clear and testable DR posture ✅ Lower cost than private endpoint designs ✅ Suitable for many non‑regulated workloads Trade‑offs Public exposure (mitigated via firewall rules, RBAC, and conditional access) Not appropriate for all compliance requirements Requires automation and application awareness of multiple vaults Azure Native Replication vs Customer‑Managed Multi‑Region Vaults Capability Azure Paired Region Multi‑Region Vaults Read access during outage ✅ ✅ Write access during outage ❌ ✅ Secret rotation during outage ❌ ✅ Customer‑controlled failover ❌ ✅ DR testing ❌ ✅ Isolation from accidental deletion ❌ ✅ Predictable RTO ❌ ✅ Azure Key Vault’s native replication optimizes for platform durability. The multi‑region pattern optimizes for application continuity. When to Use Each Approach Paired‑Region Replication Is Often Enough When: Secrets are mostly static Read‑only access during outages is acceptable RTO is flexible You prefer Microsoft‑managed recovery Multi‑Region Vaults Are Recommended When: Secrets or certificates rotate frequently Applications must remain writable during outages Deterministic failover is required DR testing is mandatory Regulatory or operational isolation is needed Closing Thoughts Azure Key Vault behaves exactly as documented on Microsoft Learn—but it’s important to be clear about what those guarantees mean. Paired‑region replication protects your data, not your ability to operate. If your application is designed to survive a regional outage, Key Vault must follow the same multi‑region design principles as the application itself. The reference architectures above show how to extend Azure’s native durability model into true operational resilience, without waiting for a platform‑level failover decision.
First Issuance manual, with automated renewals
Hey all Rob Greene again. Seems like I have been on this PKI kick lately, and today is not going to be any different. Occasionally, I will get a customer who must get certificates issued for things like Web sites, and they must have custom Subject Alternative Name (SAN) DNS values on the issued certificate. They hate that their web server admins must submit the request, and then as Certificate Authority Managers, they must look in the Certification Authorities database and review the Pending Requests table. Some background The first question that they ask is if there is a way to not require them (the CA Manager) to approve the request as this slows down the process of getting the certificate issued. From a technical perspective most of you would know that the answer is YES, you can configure the certificate template to automatically issue. However, this is NOT a good thing to do, and is seen as a Microsoft CA vulnerability; if you have a CA audit happen you might fail the audit because you allowed this. This specific vulnerability is considered the ESC1 vulnerability. ESC1 wants either an Enrollment Agent to cross sign the certificate request, a CA Manager to review the request, or both to validate the information in the Subject Alternative Name (SAN), when the certificate template’s Subject Name tab is set to “Supply in the request”, before allowing the certificate to be issued. This is recommended since the submitter is allowed to enter unvalidated information in the SAN, unlike when the certificate templates Subject Name tab is configured for “Build from this Active Directory information”. How to protect the template against ESC1? Given this background information, you should already be thinking to yourself, “Rob is not going to tell me to break anything security wise.”, and you would be correct. Rather, I am going to tell you that any certificate template that you have configured for “Supply in the request”, should at a minimum be configured to require CA Manager approval. Then for something like the certificate template that Network Device Enrollment Services (NDES) uses for its certificate, I would advise you to use the Enrollment Agent configuration. The Exchange Enrollment Agent (Offline request) template has the Enhanced Key Usage (EKU) of “Certificate Request Agent”, and this is the certificate NDES uses to sign the Certificate Service Request (CSR) that the SCEP client sends to the NDES Server. Of course, I am just giving some good examples of real-world certificate templates that you are going to be using within the environment. The last part is that the security permissions SHOULD be locked down as to who can enroll for each of these certificate templates. Specifically, enrollment for the two Certificate templates needed by the NDES services, it should be the CA Manager NDES Admin and the NDES Server. The Exchange Enrollment Agent (Offline request) and CEP Encryption certificate templates should be locked down as to who can request enrollments. Just like the custom Adatam Web Server certificate template, only IIS (Internet Information Services) Admins, and IIS Web Servers should have access to this template to be able to enroll with it. So what can you do for me here? Let’s first discuss the scenario that I am going to cover in the blog. Although I would love to cover all the possible uses that all you lovely internetz users might have for certificate enrollment, this blog will have to end at some point. The Scenario IIS Admins are tired of remembering to go to all their web servers and request new certificates for them before they expire. The CA Manager is also tired of the IIS Admins always sending emails nagging that they need their certificate enrollment requests issued. Then always having to go to the Certification Authority snap-in to approve/issue the same certificates over and over each year because of rules that web server certificates can only be valid for 398 days. See (https://thehackernews.com/2020/09/ssl-tls-certificate-validity-398.html) What is the solution? Side bar: If you want to modify the settings of any default certificate template, it is best to duplicate the original certificate template and then make changes to it. This is recommended so that you always have the default templates available in case you must create a new template in the future that is based off the original template. If the original template is modified two to three years down the line, you may not remember what was changed so that it could be changed back to the original template configuration. As far as naming goes, we would recommend that you put your company name or initials in the template name and keep the original template name after that. In this blog you will see the template used is called Adatam Web Server. This is a customized template based off the Web Server template. Keep in mind that this can be done for any certificate template that has the Subject Name tab configured for “Supply in the request”, this not limited to just web server certificates. I am in, so what’s next As you can imagine, there are several things that need to be configured to get this working: A Security group needs to be created for Users who are going to request the certificate via the Local Computer Certificate enrollment wizard based on the certificate template. A security group needs to be created for the computer accounts that need the certificate issued based on the security template. Certificate template needs to be created. Certificate template changes need to be made to support this enrollment method. Certificate template Permissions need to be configured. Group Policy setting for Computer Certificate Autoenrollment Security groups need to be created. The first two steps should be self-explanatory as to how to create groups and set up the group scopes. We are not going to cover group creation in this blog. I do want to point out one thing that happens often: Most Administrators seem to remember that if you add a user to a group, the user needs to log off and back on before the group can be added to their tokens. When adding a computer account to a new group it also needs to log off and back on before the group membership can be added to its token. Typically, the log off and back on for a computer will be a computer reboot; that needs to happen after the new group membership is added to its token. Certificate Template creation and changes The base template that you use to duplicate can of course be anything you would like, but in our example, we are just going to duplicate the good’ol faithful Web Server certificate template. Launch the Certificate Template snapin: CertTmpl.msc Find the certificate template named: Web Server Right click on the Web Server template and select Duplicate Template. The Compatibility tab, set the values as you wish. NOTE: Stay away from using anything higher than Windows Server 2012 R2/Windows 8.1 for your templates if you are using CEP and CES as these two web services have problems seeing templates with compatibility higher than this. The General tab, type in a descriptive name for the new certificate name you are configuring. The Request Handling tab, set the values that make sense for your template. NOTE: It might make sense to set the checkbox “Renew with the same key”. The Cryptography tab, you may want to modify the Minimum key size value and select the Provider Category and Providers that makes sense for the certificate templates use. NOTE: I would strongly suggest NOT setting Use alternate signature format on the template either. This is a proprietary format that many applications will not understand. This check box will only be available when you are using a Key Storage Provider (KSP) provider type. The Issuance Requirements tab is where the heavy lifting is happening. Check the box “CA certificate manager approval”. Select the radio option “Valid existing certificate” under the Require the following for reenrollment section toward the bottom of the dialog box. The Security tab is where the security groups that were created for the server admins and the servers themselves need to be configured on the template. You would add both security groups, and they only need the Allow Enroll permissions. Neither group should have the Autoenroll permission defined on it. When done setting the other options for the template, click the OK button. Setup the computer certificate autoenrollment group policy Next, we are not going to cover how to create a Group Policy Object, and things like GPO ordering, etc. We are going to discuss just the policy settings that are important for the discussion of this blog. Launch Group Policy Management Console: GPMC.msc Either create a new Group Policy or modify an existing one. Navigate to: Computer Configuration\Policies\Windows Settings\Security Settings\Local Policies\Public Key Policies. Double click on the policy setting “Certificate Services Client - Auto-Enrollment”. Enable the policy, and check the two boxes: Renew expired certificates, update pending certificates, and remove revoked certificates. Update certificates that use certificate templates. NOTE: To learn more about these two settings I would recommend going over to Vadims blog. He does a great job of explaining what each of these do: https://www.sysadmins.lv/blog-en/certificate-autoenrollment-in-windows-server-2016-part-3.aspx Click the OK button on the dialog box. Close the Group Policy Management Editor. Link the Group Policy object to what seems appropriate within your Active Directory structure. Trying to test this feature, but it seems like the renewal does not work. Of course, with any of these things, you should always test things before relying on the solution in a production environment. I do want to caution you about setting up a test environment and then setting up the certificate for a short issuance period as this tends not to work out well for most customers, and they eventually end up calling stating that autoenrollment is not working. There are few things to keep in mind about certificate automatic enrollment / automatic renewal process. The first thing to remember is that certificate auto enrollment / renewal only happens on the following triggers: At User logon, or computer boot, for the corresponding security context. Every 8 hours after user logon, or computer boot. This means that after the user or computer has been logged in or turned on for 8 hours then the auto enrollment code will happen again for the security context in question. To trigger auto-enrollment to run manually you have two options. Run GPUpdate /Force. Part of the function that this does is it causes an autoenrollment cycle to happen. This can be a bit much especially if you are applying a lot of user and computer group policy. This will trigger both computer and user autoenrollment to run. Use one of the two CertUtil command lines based on which security context you want enrollment to run against. For computer: CertUtil -Pulse For User: CertUtil -User -Pulse Certificate renewal WILL NOT happen until 90% of the Certificate lifetime has expired. Trying to use the template setting “Renewal period” does not change this fact. Auto enrollment / renewal attempt MUST happen while the certificate is still valid, and after the start of the 10% certificate lifetime left and the expiration time of the certificate. (MUST NOT BE EXPIRED). Like being unable to re-animate something that is already dead, you cannot renew a certificate that has already expired. The math that needs to be done is: (CertLifeTimeInDays * 24) * .10 The value from the above MUST be larger than 8 hours. Let's do some math Here is an example: Set up a certificate that is valid for two days for testing and we will run through the exercise here: Find out how many hours the certificate is valid for: 2*24 = 48 Hrs. Figure out what 10% of certificate lifetime in hours is: 48 *.10 = 4.8 Hrs. This shows us that for one of these certificates to be renewed, the process that runs only every 8 hours would have to run within this 4.8 Hrs. window. This is going to tell us that sometimes you would get lucky with a certificate renewal attempt being done, but more than likely MOST of the time you would fail to renew the certificate because it would expire before the auto enrollment cycle happens. Another example is a certificate that is valid for 4 days. Let us run through the exercise here. Find out how many hours the certificate is valid for: 4*24 = 96 Hrs. Figure out what 90% of certificate lifetime in hours is: 96 *.10 = 9.6 Hrs. This shows us that for at a 10% lifetime left it is still valid for over 8 hours. This certificate would always get at least one auto enrollment cycle to be run to renew the certificate. Assuming you don’t have anything else wrong, the renewal would happen successfully. This is the minimum lifetime that any certificate testing should be set to. Test, test, and more testing! Well, now you got your test lab, issuing short lived certificates and getting them automatically renewed, but… Now you are noticing that your application is not using the new certificate. Unfortunately, there are going to be some applications for which the automatic renewal might not work out well in your organization. However, the certificate could still be renewed manually by your IIS admins, and it would still not require the CA manager to be involved in the renewal process. The key thing is your IIS Admins would need to go through the manual certificate renewal process BEFORE the current certificate expires, as it is used to cross sign the renewal request. UPDATE 5/30/2024: After release of the blog it was brought to my attention of two different things to consider. First unknown to me (as I am not an IIS support engineer) apparently there is a setting within the IIS Management console to help with updating the site binding configuration so that this automatic renewal behavior would work. This feature is called Certificate Rebind and was new in IIS 8.5. https://learn.microsoft.com/en-us/iis/get-started/whats-new-in-iis-85/certificate-rebind-in-iis85 Second, we recently has a case were the customer had several websites running on an IIS Web Server. These websites all used unique certificates and were all generated manually using the same certificate template. When one of the certificates were ready to be renewed, they found that all website certificates were archived, and only one of the website certificates were automatically renewed. This would be by design behavior. The only thing that we could really recommend in this situation is to use one certificate for all websites and make sure that all the websites DNS names were added to the certificates Subject Alternative Name (SAN) field. Failure to do this will result in going through the entire manual issuance of the certificate all over again. And, going through the CA Manager approval process as well, since this would be a new certificate request. Hopefully, you found some good nuggets of information about how to lockdown certificate templates to protect against ESC1 vulnerabilities and what some of those less often used tabs do. And of course, how the certificate renewal process works, especially around shot lived certificates. Rob “I coulda been a Welder” Greene Extra credit for those who can figure out where the saying came from.
