Troubleshoot Hyper-V guarded fabrics and shielded VMs
Use this FAQ when a guarded fabric can't be deployed, a host can't attest, a shielded virtual machine (VM) can't be provisioned or started, or an operation such as migration, servicing, or recovery fails. It accompanies the Microsoft Learn module: https://learn.microsoft.com/en-us/training/modules/guarded-fabric-shielded-virtual-machines/ Warning Don't use TPM clearing, an unrelated new or local KP, disabled Secure Boot or CI, a weaker attestation mode, certificate-validation bypasses, or copied owner private keys as generic troubleshooting fixes. Don't delete protected VM state or checkpoint files to make an error disappear. These actions can weaken the trust boundary or permanently remove access to encrypted data. In this FAQ Why can't I install or prepare the platform? Why does HGS deployment or service operation fail? Why can't hosts or owners reach the correct HGS service? Why do certificate enrollment, import, or private-key operations fail? What do HGS client attestation results mean? Why does a TPM-trusted host fail a specific security check? Why do code integrity policy changes break the host? Why do host-key or legacy AD-based attestation fail? Why does attestation pass but a VM still can't obtain its keys? Why do template preparation or image validation fail? Why do shielding-data creation and VM provisioning fail? Why can't the owner access the guest or confirm its protection? Why do live migration, cluster placement, or failover fail? Why do servicing, key rollover, or retirement cause an outage? Why does withdrawing trust not immediately contain an incident? What fails when HGS or a site is unavailable? Why do backups or disaster recovery fail? Why does Windows guest BitLocker recovery fail? How do I recover from TPM failures or an offline CA? Why are diagnostics inconclusive or misleading? Why can't I install or prepare the platform? Resolve platform failures before investigating HGS policy. A feature name appearing in a script doesn't establish that the target system supports it. Failure mode or symptom Likely causes Resolution overview HostGuardian isn't available or the host can't become guarded. The Hyper-V host runs an unsuitable Windows Server edition or release. Use a supported Datacenter edition for guarded Hyper-V hosts. Don't confuse this requirement with HGS servers, whose role is available in Standard and Datacenter. Check the Windows Server 2025 edition inventory. The HGS role is installed on a Hyper-V host, but attestation support is missing. HostGuardianServiceRole was confused with the host-side HostGuardian feature. Install the correct host features on the guarded host and deploy HGS on its separately administered servers. Installing the HGS role doesn't enroll the local Hyper-V host or shield its VMs. A required cmdlet, module, or wizard is missing. Missing RSAT/provisioning tools, wrong machine or PowerShell environment, unsupported parameter, or incomplete feature installation. Check Get-WindowsFeature , Get-Command , module source/version, and installed help on the machine that performs that stage. Install the documented tools there. Don't move owner secrets to a host just because the host has more tools. A graphical shielding-data or template wizard won't run on Server Core. The workflow assumes a local Desktop Experience interface. Use the supported PowerShell workflow or a trusted Desktop Experience management system with the required tools. Server Core support for HGS and Hyper-V doesn't imply every graphical preparation tool is available locally. Features or protection settings are configured, but don't take effect. A required restart hasn't occurred, Group Policy hasn't applied, or a policy is configured but not running. Complete the approved maintenance restart and check effective policy and running security services. Distinguish Win32_DeviceGuard configured values from running values. Drain workloads first where necessary. TPM-trusted prerequisites or virtualization startup fail. Missing/disabled TPM 2.0, SLAT, IOMMU, hardware virtualization, UEFI boot, or Secure Boot; incompatible firmware or drivers. Verify actual firmware settings and supported hardware, not only specifications. TPM mode requires UEFI 2.3.1 or later and the relevant protections to function. Replace unsuitable hardware or use a separately approved architecture; don't silently lower assurance. Enabling Host Guardian support or HVCI causes a stop error or device failure. A driver or device is incompatible with virtualization-based code integrity. Keep the host out of placement, preserve the crash evidence securely, and use the supported rollback or recovery path. Obtain compatible OEM firmware/drivers and test the security configuration before returning the host to service. A copied example fails with a particular guest, device, nested-virtualization, or management combination. Support was inferred from ordinary Hyper-V support or an older example. Verify the exact build, guest OS, protection policy, device feature, and operation. Use an explicitly supported combination. A successful unshielded-VM test isn't support evidence for the shielded equivalent. References Guarded-host prerequisites: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-guarded-host-prerequisites Windows Server edition comparison: https://learn.microsoft.com/windows-server/get-started/editions-comparison?view=windows-server-2025&preserve-view=true Guarded-host troubleshooting: https://learn.microsoft.com/troubleshoot/windows-server/virtualization/troubleshoot-guarded-hosts Get-HgsTrace: https://learn.microsoft.com/powershell/module/hgsdiagnostics/get-hgstrace?view=windowsserver2025-ps&preserve-view=true Test-HgsClientConfiguration: https://learn.microsoft.com/powershell/module/hgsclient/test-hgsclientconfiguration?view=windowsserver2025-ps&preserve-view=true Why does HGS deployment or service operation fail? The HGS cluster supplies attestation and key protection. It is separate from the Hyper-V cluster that provides workload availability. Failure mode or symptom Likely causes Resolution overview The role installs, but no working HGS forest or service exists. Role installation was mistaken for forest creation and HGS initialization. Complete the documented Install-HgsServer and Initialize-HgsServer stages, with the correct identity, certificates, and restart boundaries. Inspect Get-HgsServer ; a successful role installation alone isn't completion. New-forest setup or domain-controller promotion fails. Incorrect starting domain membership, DNS/network configuration, permissions, name conflicts, or an incomplete previous attempt. Review deployment and AD DS events, correct the actual prerequisite, and follow the supported installation/recovery path. Don't experiment by deleting AD or cluster objects. Preserve DSRM recovery credentials separately. HGS can't be deployed correctly in an existing forest. Attempted deployment in a child domain, inadequate privileges, or unsuitable bastion-forest design. Deploy in the existing forest's root domain using the documented prerequisites. If the forest isn't an isolated, highly privileged security boundary, redesign rather than accepting fabric-administrator control over HGS. An additional HGS node won't join or initialize. The new node can't resolve the existing forest, credentials can't join/promote it, a reboot is outstanding, or the existing-node address is wrong. Restore domain connectivity and authorized enrollment, then initialize against a reachable existing HGS node. Verify domain, cluster, and HGS membership rather than assuming one implies the others. HGS fails after losing a node or during maintenance. Insufficient healthy nodes, quorum or cluster-network failure, insufficient remaining capacity, or a shared failure domain. Restore cluster communications and healthy capacity using the approved cluster recovery plan. Service one node at a time. Don't force quorum or remove cluster state without a recovery decision and evidence. Requests intermittently fail through a healthy-looking service name. One HGS node has stale policy, broken TLS, unusable keys, a failed application pool, or an HSM access problem. Correlate failures with the responding node. Exercise attestation and key release on every node while preserving service-name validation. Repair or safely remove the failing node from service; one successful clustered request isn't sufficient. The KPS application pool can't use its service identity. HGS node computer accounts can't retrieve the gMSA password, AD replication is unhealthy, or service identity configuration changed. Repair the specific gMSA retrieval permissions and AD/service configuration. Separately check private-key permissions. Don't replace the gMSA with a broad administrator account. KPS returns HTTP 500 with a required-privilege or security-audit-source error. The HGS gMSA lacks Generate security audits ( SeAuditPrivilege ), often after security-policy hardening. Restore the documented user right through the controlling policy on every HGS node. Preserve required default principals as well as the gMSA: assigning this right replaces, rather than appends to, the existing list. A JEA reviewer can't connect or can't run a requested command. Wrong microsoft.windows.hgs endpoint, incorrect role membership, WinRM authentication/listener failure, or a command outside the delegated role. Repair the endpoint and authenticated remoting path. Use reviewer access for inspection and the authorized HGS administrator role for changes. A reviewer being unable to edit policy is expected, not grounds for granting Domain Admins. The entire fabric can't start after a power or site outage. Virtualized HGS, DNS, identity, or backup-access services depend on shielded VMs that need the unavailable HGS. Restore an independently bootable recovery path. Place essential recovery services and credentials outside that circular dependency. A production architecture must be able to start its security dependencies before protected workloads. HGS becomes slow or times out during a restart wave. Simultaneous attestation/key-release demand, HSM latency, resource exhaustion, or too much capacity removed for maintenance. Measure the constrained resource, restore capacity, and stagger starts or maintenance. Validate latency and key access under representative load. Longer timeouts alone don't repair unavailable keys or failed attestation. References HGS troubleshooting: https://learn.microsoft.com/troubleshoot/windows-server/virtualization/troubleshoot-guarded-fabric-hgs Managing HGS: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs HGS deployment prerequisites: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-prepare-for-hgs Why can't hosts or owners reach the correct HGS service? Check transport and service identity before changing host trust. The host's Attestation and KeyProtection URLs and the owner's metadata URL serve different operations. Failure mode or symptom Likely causes Resolution overview The HGS service name doesn't resolve or resolves to the wrong nodes. Missing/stale DNS records, incorrect conditional forwarders, wrong DNS suffix, or use of a node name instead of the distributed service name. Correct authoritative records and forwarding, then verify resolution from each affected host and owner-management system. Don't use an IP-address URL to bypass a TLS-name failure. TCP 443 or required deployment/domain traffic is blocked. Routing, firewall, VLAN, network ACL, or segmentation changes. Restore only the required documented flows for HGS, AD DS, cluster operation, and management. Opening HTTPS alone doesn't satisfy forest creation, replication, gMSA, or cluster dependencies. TCP succeeds but HTTPS authentication fails. Expired/not-yet-valid TLS certificate, service-name/SAN mismatch, wrong binding, missing private key, or an untrusted chain. Correct the endpoint certificate and binding on the affected nodes, validate time, and distribute the verified public trust chain to clients. Retain name and chain validation. TLS identity is separate from HGS guardian authorization. TLS works on one computer but not another. CA trust was deployed only in the HGS forest, an intermediate was put in the root store, or client policy hasn't applied. Install verified roots and intermediates in their correct local computer stores using the appropriate domain or owner policy. HGS-forest Group Policy doesn't configure hosts in another forest or independent owner systems. Chain validation is slow or fails despite the root being installed. Required intermediate discovery, CRL, or AIA endpoints are unavailable; certificate dates or trust policy are wrong. Repair the relevant PKI publication and network paths and inspect chain-validation errors. Don't globally disable revocation checks. HGS service-key revocation semantics aren't a reason to ignore TLS trust failures. A service URL returns an unexpected page, 404, or the wrong service. Incorrect protocol/path, wrong HGS instance, or proxy/routing configuration. Compare client settings with the intended Get-HgsServer output and use the correct /Attestation and /KeyProtection paths. Confirm the intended organization and guardian keys before accepting metadata. Requests fail only through an inspection, load-balancing, or WAN path. The intermediary changes protected traffic, routes to unhealthy nodes, or introduces an unsupported TLS/proxy configuration. Compare with an approved direct service path, preserve errors, and correct the intermediary design. VMM specifically documents provisioning failures when packets are modified in transit. Don't weaken message validation to accommodate rewriting. HGS breaks after HTTPS hardening. The required HTTP IIS binding was removed, or TLS settings became incompatible with the client stack. Restore the supported HGS binding configuration; retain HTTP binding as documented while using approved network controls for access restrictions. Investigate protocol compatibility separately. Don't re-enable obsolete TLS versions as the default fix. Health certificates appear expired or not yet valid; Kerberos also fails. Time drift among hosts, HGS nodes, domain controllers, or the management system. Restore trusted time synchronization first. If HGS's internally managed attestation signer remains invalid, follow its documented renewal-task procedure. That certificate isn't the administrator-provided KPS signing certificate. Hosts contact an old service or lose fallback after a configuration edit. Stale endpoints or a Set-HgsClientConfiguration call omitted the fallback parameters. Inspect all four endpoint values and restore the approved primary/fallback pairs. Omitting fallback parameters clears that configuration. Recheck both services' policies and keys; URL configuration doesn't synchronize them. References HGS certificates and HTTPS: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-obtain-certs Set-HgsServer: https://learn.microsoft.com/powershell/module/hgsserver/set-hgsserver?view=windowsserver2025-ps&preserve-view=true HGS troubleshooting: https://learn.microsoft.com/troubleshoot/windows-server/virtualization/troubleshoot-guarded-fabric-hgs Fallback configuration: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-branch-office Why do certificate enrollment, import, or private-key operations fail? Identify the certificate's purpose before repairing it: HGS signing, HGS encryption, HTTPS, host-key identity, owner identity, template signing, and guest endpoint identity aren't interchangeable. Failure mode or symptom Likely causes Resolution overview A standalone CA request stays pending or no certificate file is returned. Administrative approval is required; submission was mistaken for issuance. Record the request ID, have the CA administrator authenticate and approve the request, and retrieve the issued certificate. Don't repeatedly generate new keys while waiting for approval. Certificate enrollment fails with a template or CA configuration error. An enterprise-template procedure was applied to a standalone CA, the ComputerName\CAName value is wrong, or enrollment access is unavailable. Use the correct CA and enrollment workflow. Standalone CAs don't use certificate templates; put the required properties in the request and have the CA administrator review the issued result. HGS rejects a signing/encryption certificate or crypto operations fail. Legacy CSP rather than KSP, unsuitable algorithm/key size, missing usages, or a vendor provider mismatch. Obtain the documented HGS certificate profile: KSP, RSA of at least 2048 bits, required signing/data-encipherment usages and Server Authentication EKU. Verify the issued certificate rather than assuming the request was honored. Certificate acceptance succeeds but the private key is absent. Only a public .cer was imported, acceptance occurred on a different computer from the request, or the original key was lost. Accept against the original request/key or restore an approved PFX/provider-backed key. A public certificate can't recreate its private key. Reissuing with a different key is a rollover, not recovery of the original. PFX import or export fails. Wrong password, damaged file, inaccessible path/store, missing private key, or intentionally nonexportable/HSM-backed key. Verify the intended artifact and authorization. Use the original protected PFX/password or the provider's supported provisioning and recovery method. Don't try to make a nonexportable key exportable as an incident shortcut. A certificate shows HasPrivateKey , but HGS can't use it. The gMSA lacks access, the provider can't open the key, or the key is available only under another identity. Verify actual cryptographic use under the HGS service identity and repair the minimum private-key/provider permissions. Key association, file access, and successful service use are separate checks. A new HGS node has certificate objects but can't release keys. Thumbprint-referenced private keys weren't installed or authorized on that node. Provision the original private-key access on every node. PFX-provided HGS keys can replicate; thumbprint references don't replicate their private keys. Test each node before admitting traffic. HSM-backed operations fail or become intermittent. HSM outage, network/session failure, missing KSP/driver, inaccessible key set, authorization change, or HSM capacity exhaustion. Use the vendor-supported health and recovery procedures, restore the correct key set, and verify node and gMSA access. Don't create unrelated replacement keys or export keys into an unapproved software store. HTTPS fails after adding a node or renewing certificates. TLS certificates/bindings were assumed to replicate with HGS service keys. Configure the approved TLS identity and private-key access on each node. Test service-name and chain validation from hosts. TLS deployment and HGS signing/encryption-key replication are different processes. A service or host-key certificate expires, but authorization continues. Expiry was incorrectly treated as key withdrawal. This can be expected: HGS uses its signing/encryption certificates as key containers, and host-key attestation registers the public key rather than enforcing certificate expiry. Explicitly retire the relevant service key or host registration. TLS expiry still affects transport. Importing guardian metadata fails even though the service still uses those keys. Owner-side certificate checks reject an expired or untrusted guardian certificate. Authenticate the fabric and keys independently, repair trust or obtain appropriate updated metadata/certificates through an approved process. Don't normalize untrusted-root/expiry bypasses. Import-time validation and HGS's use of registered keys have different semantics. Old VMs fail after a certificate was renewed with the same subject. Renewal generated a new key pair; the subject stayed the same but cryptographic identity changed. Recover the original uncompromised key if available, or perform owner-authorized existing-KP updates as a planned rollover. Inventory retained backups and PDKs before retiring the old key. References Obtain certificates for HGS: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-obtain-certs Certificate and gMSA troubleshooting: https://learn.microsoft.com/troubleshoot/windows-server/virtualization/troubleshoot-guarded-fabric-hgs HGS key management: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs Create a host key: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-create-host-key What do HGS client attestation results mean? Inspect AttestationStatus , AttestationSubStatus , the service contacted, and the event timeline together. Don't equate a previously successful result with a fresh test of the current configuration. Result Interpretation Resolution overview NotConfigured The client isn't configured for HGS-backed attestation/key protection; it can be in local mode. Configure the approved HGS endpoints and retest. Local vTPM operation isn't proof of guarded-fabric protection. Unknown No usable attestation result is available, commonly because attestation hasn't yet been attempted. Check configuration, initiate a fresh supported client test, and collect diagnostics. Don't treat the absence of a rejection as acceptance. TransientError A networking, service, or other temporary failure prevented completion. Correlate DNS, TLS, IIS, HGS-node health, and timing. Retry after the underlying condition is repaired rather than repeatedly re-enrolling the host. Expired A previously issued health certificate is no longer valid. Check clocks and service reachability, then obtain fresh attestation. Investigate renewal failures if the condition recurs. UnauthorizedHost The presented identity isn't admitted under the active mode. Compare the actual TPM identity, host public key, or legacy AD group identity with the applicable registration. An AD-group membership repair is relevant only to AD-based attestation. TpmError The host couldn't complete TPM-related attestation work. Review TPM, firmware, and measured-boot evidence; check TPM readiness and OEM updates. Don't clear or replace the TPM before assessing protected-state and recovery consequences. InsecureHostConfiguration One or more enabled host security requirements aren't satisfied. Follow the actual substatus and verify effective host state. Correct unintended drift or approve a justified policy/evidence change; don't disable every failed check. Passed but the VM still fails Host admission succeeded; VM guardian authorization, provisioning, guest encryption, and migration authentication weren't thereby proven. Move to the failing later layer. Inspect KPS and the VM's existing KP for key-release errors, or provisioning/guest/migration evidence for later failures. References Attestation status and substatus reference: https://learn.microsoft.com/troubleshoot/windows-server/virtualization/troubleshoot-guarded-hosts Why does a TPM-trusted host fail a specific security check? The following substatuses describe the physical Hyper-V host, not the guest vTPM. Confirm the exact spelling and policy behavior on the installed build. Failure mode or substatus Likely causes Resolution overview TPM registration reports a missing or untrusted endorsement-key certificate. Export wasn't elevated, the OEM didn't provision the certificate as expected, or HGS lacks the TPM manufacturer's trust chain. Export the identifier with appropriate privileges, investigate missing certificates with the OEM, and install verified manufacturer roots/intermediates in TrustedTPM_RootCA and TrustedTPM_IntermediateCA on every HGS node. The HGS TLS CA isn't the TPM manufacturer CA. A replacement host or TPM isn't recognized. Enrollment describes the old physical TPM, even if the hostname is unchanged. Verify the new hardware's provenance and security configuration, enroll its actual identity, and withdraw the retired identity from all applicable HGS services. Don't reuse another host's identifier file. SecureBoot Secure Boot isn't enabled or the required boot trust configuration isn't in use. Enable Secure Boot with the Microsoft Secure Boot template. Investigate incompatible or revoked boot components; don't repair by leaving Secure Boot disabled. SecureBootSettings Current measurements don't match approved baselines, including after firmware, DBX, boot-authority, or debug-setting changes. Explain the measurement change. Roll back unintended drift or capture and approve a new baseline from a verified configuration. Use a baseline appropriate to that hardware/firmware class. VirtualSecureMode VBS isn't running, despite possible policy configuration. Check firmware virtualization, platform protections, effective policy, hypervisor startup, driver compatibility, and required restart. Verify the running protected environment before reattestation. HypervisorEnforcedCodeIntegrityPolicy Hypervisor-enforced code integrity isn't active or isn't enforced as required. Correct HVCI configuration and incompatible components, activate the intended policy, and verify runtime state. An audit-only policy or a registry setting alone doesn't prove enforcement. CodeIntegrityPolicy The active CI policy is missing, stale, or not registered with HGS in the expected form. Compare the deployed policy and registered approved unsigned binary. Activate the intended enforced policy and register its matching measurement through change control. See the next section for policy-deployment failures. Iommu IOMMU/DMA protection is unavailable, disabled, or not required by the host's VBS configuration. Enable supported hardware and the required VBS DMA protection; replace unsuitable hardware if necessary. Disabling the HGS requirement changes the threat model. PagefileEncryption Host paging-file encryption isn't enabled as required. Apply the supported encrypted-paging configuration and any required restart, then reattest. This check concerns host paging data, not guest BitLocker volumes. FullBoot The host resumed from sleep or hibernation rather than completing the required full boot. Arrange a controlled full restart with workloads drained. Verify the restart path instead of repeatedly resuming the host. HibernationEnabled Host hibernation configuration doesn't meet admission policy. Apply the approved no-hibernation configuration and restart. Don't relax the policy merely to preserve a sleep/resume operational practice. DumpsEnabled Dumps are enabled while policy requires them disabled. Disable the disallowed dumps. If operational requirements justify encrypted dumps instead, use an explicitly approved HGS dump-policy design rather than an ad hoc exception. DumpEncryption Dumps are permitted but not encrypted as required. Disable dumps or configure supported dump encryption according to policy. Protect diagnostic collection as potential tenant-sensitive material. DumpEncryptionKey The host encrypts dumps with a key that HGS doesn't recognize. Use an approved dump-encryption key and register its correct hash/policy with HGS. Preserve the independent private-key recovery authority needed to read an authorized dump. TcgLogVerification Measured-boot log retrieval/verification fails because of firmware, TPM, hardware, or an outdated PXE network boot program. Collect TPM/boot evidence, review boot order, update approved firmware and PXE components, and involve the OEM where necessary. Don't register an unverifiable log as an accepted baseline. BitLocker An enabled HGS host-BitLocker policy isn't satisfied. Verify and restore the required protection of the host OS volume. This policy is disabled by default and is independent of guest encryption. Don't change it without the HGS policy owner's decision. All artifacts seem present, but attestation still fails or unexpectedly uses an older version. Incompatible identity/baseline/CI artifact versions, a version setting retained during upgrade, or matching legacy artifacts that permit fallback. Inventory actual versions and installed command interfaces. Establish a complete approved version-2 path, then retire unapproved version-1 artifacts. Don't invent a version parameter or downgrade to bypass endorsement validation. Failure follows TPM clearing, security reset, or replacement even though the server still boots. TPM-protected or VBS identity material changed; previously sealed state or offline caches aren't usable. Recover host security configuration and re-establish online attestation/key release under the approved identity. Use independent recovery material for affected encrypted host data. A TPM reset isn't a harmless inventory operation. References Guarded-host security checks: https://learn.microsoft.com/troubleshoot/windows-server/virtualization/troubleshoot-guarded-hosts TPM artifact capture and versions: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-tpm-trusted-attestation-capturing-hardware Trusted TPM roots: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-install-trusted-tpm-root-certificates HGS attestation policies: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs Why do code integrity policy changes break the host? App Control for Business code integrity controls what can run. HGS checks whether the host is using an approved policy; it doesn't certify that every allowed program is safe. Failure mode or symptom Likely causes Resolution overview An approved driver, backup agent, or management tool is blocked. Missing allow rules, an overly narrow rule, or an update invalidated a hash rule. Examine CodeIntegrity events and verify the publisher/package. Update only the necessary approved rules, validate in audit on a reference system, and deploy/register the enforced policy through change control. No audit events appear, but enforcement later breaks workloads. The audit policy wasn't active, or the test didn't exercise the relevant workload. Verify the policy is active after deployment/restart, then exercise backup, management, drivers, servicing, and recovery. An empty log without representative activity isn't evidence of compatibility. HGS rejects a host after a signed CI policy is registered. The signed wrapper or a different binary was supplied instead of the matching unsigned measured policy. Register the correct unsigned form corresponding to the deployed signed policy. Don't remove signed-policy protection from the host merely to simplify registration. CI deployment reports a missing provider or policy-format error. The single-policy update provider isn't available, or the tool, file format, and target build don't match. Verify the installed provider/tool and supported policy format. Validate both local enforcement and HGS measurement; support for CiTool or multiple policies alone doesn't prove that a particular HGS combination works. A policy file was updated, but the host still presents the old policy. Deployment targeted the wrong location, activation failed, a restart is required, or signed update rules rejected the change. Inspect the active policy and deployment events. Use the supported activation/update procedure and required restart. Don't repeatedly register different HGS artifacts to conceal an inactive host policy. A signed policy prevents boot or can't be replaced. Required boot components were blocked, the authorized update signer is unavailable, or signed-policy recovery wasn't prepared. Use the approved signed-policy recovery/update process with retained signing authority and recovery access. Preserve evidence and restore a known-good policy; don't arbitrarily delete EFI or protected policy files. Attestation passes with an unexpectedly permissive software configuration. A broad publisher rule, audit policy, obsolete policy, or other unintended admission artifact remains accepted. Review enforcement and policy scope, remove unapproved artifacts after impact analysis, and test rejection of a representative unapproved configuration. A Passed result can expose a policy-design failure rather than prove strong protection. References TPM and CI preparation: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-tpm-trusted-attestation-capturing-hardware Deploy App Control policies: https://learn.microsoft.com/windows/security/application-security/application-control/app-control-for-business/deployment/deploy-appcontrol-policies-with-script HGS policy management: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs Why do host-key or legacy AD-based attestation fail? Host-key possession, AD group membership, and TPM-backed measured health are different admission mechanisms. HGS has one active attestation mode. Failure mode or symptom Likely causes Resolution overview A host key was registered, but the host is still rejected. HGS is using another mode, the client presents another key, or the required version isn't active. Compare HGS mode/version with the host's actual public key and registration. Stage a service-wide mode change only after all affected clients are ready; registration doesn't switch the mode. Host-key attestation fails after rebuild, rotation, or inventory scripting. The local key was regenerated, lost, or replaced without enrolling the new identity. Identify the intended key and its provenance. Enroll an approved replacement on every applicable HGS service and withdraw the old registration. Don't run key-generation commands as read-only inventory. A custom host-key certificate can't be used. Missing/unusable private key, wrong provider permissions, or an unsuitable certificate profile. Verify the documented host-key profile and private-key usability. Custom certificates require the appropriate RSA key, Client Authentication EKU, and Digital Signature usage; don't substitute the KPS or TLS certificate blindly. A host with unapproved drivers still passes host-key attestation. Key possession was mistaken for measured-health validation. This is a design limitation, not a driver-detection bug. Use TPM-trusted attestation when measured boot and CI assurance are required. Don't describe a host-key success as evidence of a clean software state. Removing one hostname doesn't reliably deauthorize a host, or an expired certificate still works. Another registration refers to the same key, the key is shared/copied, or certificate expiry was mistaken for withdrawal. Identify and withdraw the actual public-key authorization everywhere. Assess every host holding that private key and move to distinct protected identities. Validate the installed removal interface; don't guess a cmdlet or policy name. Many hosts fail after switching attestation modes. Required artifacts weren't staged, mode changes haven't reached all nodes, or clients are incompatible. Inspect mode on each node and the client population. Complete the approved transition or controlled rollback, then collect fresh results. Don't delete the old enrollment before the approved replacement path is proven. A legacy AD-authorized host is rejected after group or domain changes. Wrong registered group SID, stale computer-account group membership, failed required trust, or DNS/Kerberos/time failure. Repair the documented AD-based trust and group path, refresh machine membership as required, and retest. Recreating a group with the same name doesn't preserve its SID. This isn't a fix for TPM or host-key mode. A workgroup host is rejected solely because it isn't in the HGS domain, or AD mode is assumed removed. Domain requirements or deprecation were misinterpreted. TPM and host-key attestation don't require joining the HGS forest. Diagnose actual admission evidence. AD-based attestation is deprecated, not automatically removed; maintain only required legacy use and plan a supported migration. References Host-key preparation: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-create-host-key Attestation-mode transitions: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-upgrade-to-2019 HGS management: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs Why does attestation pass but a VM still can't obtain its keys? Investigate KPS, the VM's existing KP, and guardian identities. A host may be eligible to run some owners' VMs while correctly being denied access to others. Failure mode or symptom Likely causes Resolution overview One VM fails while other shielded VMs start on the same host. Its KP doesn't authorize the available fabric keys, an old key was retired, or the VM's protected state is inconsistent. Preserve the current KP and compare its guardian identities with KPS keys and the recent change record. The owner authorizes any KP update; HGS administrators restore approved service-key usability. A VM fails only at a recovery site or new destination. The destination passes attestation but its fabric guardian isn't authorized by that VM. Have the owner grant the approved recovery guardian access to the existing KP, preserving its secret. Configure future PDKs separately. Storage access and host admission don't authorize the destination. KPS reports an unwrap/key-protection failure for many VMs. Wrong KPS endpoint, unavailable signing/encryption keys, disabled keys, invalid health evidence, or a service/private-key failure. Correlate KPS and client events, validate fresh attestation, and check the complete required key set on every node. Don't weaken attestation or copy owner keys to hosts. Updating a PDK or making HGS keys primary doesn't repair an existing VM. Those operations don't rewrite the deployed VM's KP. Use the owner-authorized grant/revoke workflow on the existing KP. Treat the PDK as a future-provisioning artifact and retain the original PDK for its recovery role. Owner guardian creation or KP changes fail with missing-key/access errors. Only public metadata is present, the wrong certificate store or identity is used, or owner private-key access is missing. Restore and authorize the matching owner keys on the owner-controlled management system. Validate actual use, not just HasPrivateKey . Fabric-administrator membership isn't owner authority. A KP export can't be parsed or fails after a file transfer. Binary data was formatted as text, truncated, overwritten, or confused with another VM's KP. Retrieve an authentic lossless byte export and preserve its VM/version association. Keep protected originals and authenticated handoffs. Don't reconstruct bytes from a console display. A returned KP is stale or conflicts with the VM's current state. The VM started during the exchange, another operator changed it, or the wrong export/return pair was used. Stop the exchange and reconcile current state under owner control. Keep the VM off for the complete approved update and use distinct artifact names. Don't apply a stale result simply because the command accepts bytes. Grant succeeds but required revocation or export fails. An incomplete workflow left a partial or old staged return file. Treat the whole change as failed. Don't transfer a grant-only result when revocation was required. Preserve the original and create a new fully verified return artifact only after every required operation succeeds. A VM becomes inaccessible after attaching a new or local KP. The unrelated KP doesn't preserve the secret associated with existing vTPM/protected state, or authorization was weakened. Stop further changes. Use an approved matching rollback KP/consistent backup or owner recovery path. Last-known-good rollback is appropriate only if it doesn't restore withdrawn trust. A new PDK KP isn't a repair for an existing VM. Guardian names match, but authorization still fails. Friendly names or certificate subjects were mistaken for cryptographic identity; metadata came from another fabric or predates rollover. Compare authenticated signing/encryption key identities and the actual KP content. Obtain approved metadata independently of its filename or display name and authorize the intended keys. Owner keys are lost while some VMs continue to run. Current KPS authorization still works, but owner update/recovery authority is unavailable. Preserve running availability and remaining recovery options while retrieving independent owner-key backups. Plan owner-authorized recovery/reprotection if possible. Continued operation isn't proof that future KP updates or disaster recovery remain possible. A KP exchange fails while reading or writing its staging files. The directory exists only on the other computer, a file occupies the directory path, permissions are wrong, an output name is already in use, or another operator changed the staged artifact. Prepare and restrict the host-side and owner-side directories independently. Use unused filenames, preserve original exports, coordinate concurrent access, and stop on every failed read/write. Don't overwrite a retained KP or send an unverified partial file. Also see the module's existing-KP exchange procedure. References HGS key rollover: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs Grant KP access: https://learn.microsoft.com/powershell/module/hgsclient/grant-hgskeyprotectoraccess?view=windowsserver2025-ps&preserve-view=true Revoke KP access: https://learn.microsoft.com/powershell/module/hgsclient/revoke-hgskeyprotectoraccess?view=windowsserver2025-ps&preserve-view=true Why do template preparation or image validation fail? The publisher prepares and signs the template; the owner decides whether to trust it. Signature integrity isn't a substitute for approval of the image's contents. Failure mode or symptom Likely causes Resolution overview Protect-TemplateDisk rejects the disk or the Windows template can't boot correctly. Wrong VM generation, MBR rather than GPT, dynamic guest disk, unsuitable partitions, or wrong Windows-volume filesystem. Prepare the supported generation 2 Windows layout: GPT, a basic guest disk, correct EFI/boot partitions, and an NTFS Windows volume. A dynamically expanding VHDX is not a Windows dynamic disk. Don't format the EFI system partition as NTFS. Specialization fails across every VM from a template. The source wasn't generalized, Windows updates/setup are incomplete, or a required guest component is missing. Repair and validate the retained servicing source, complete supported generalization, and create a fresh protected copy. Don't repair the sole signed template in place. Template signing fails. Missing or inaccessible publisher private key, wrong certificate/profile, unsuitable preparation tools, or an inaccessible disk. Verify the publisher's approved signing identity, key usability, tools, and disk access. Keep publisher private keys in the publisher-controlled preparation environment. Template protection is interrupted or runs out of space. Insufficient storage, file locks, I/O failure, or process interruption during an in-place transformation. Preserve evidence and the failed copy, correct the dependency, and restart from the known-good unprotected servicing source. Don't distribute a partially transformed disk with an old catalog. A protected template fails when booted as an ordinary VM. A secure-provisioning template was used as a normal boot disk. Use the supported shielded provisioning path. For servicing, return to the unprotected source. Booting the protected template as an ordinary VM isn't a supported validation method. The PDK and VSC aren't applicable to each other. Wrong template identity, publisher, version rule, catalog, or PDK type. Compare the owner's approved template identity and exact version semantics. Supply the correct signed artifacts or obtain approval for a new version. Don't broaden acceptance merely to eliminate the mismatch. Applicability passes, but actual provisioning rejects the disk. The supplied VHDX changed after signing or doesn't correspond to the catalog. Quarantine the disk, investigate substitution or corruption, and obtain the authentic protected image. Retest actual disk validation, not just the PDK/VSC relationship. A signed image or later accepted version contains unapproved software. Publisher compromise, weak source-image review, or overly broad future-version trust. Contain affected deployments, investigate the publishing authority, and rebuild from approved sources. Tighten version/publisher approval and rotate affected secrets. A valid signature doesn't establish that the software is benign. References Windows shielded template preparation: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-create-a-shielded-vm-template Owner shielding-data preparation: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-tenant-creates-shielding-data Why do shielding-data creation and VM provisioning fail? Keep PDK creation, template validation, VM configuration, and guest specialization separate. A visible VM object or returned helper object isn't proof of successful provisioning. Failure mode or symptom Likely causes Resolution overview PDK creation fails or required guardian/qualifier inputs are missing. Missing owner keys/tools, wrong guardian selection, malformed input, inaccessible files, or a new-template/existing-VM parameter-set mismatch. Use the correct workflow on the trusted owner system, validate all inputs, and select the intended owner, fabrics, and image qualifiers. Preserve the original encrypted package after creation. Provisioning can't read the PDK, VSC, template, or specialization file. Wrong path, file permissions, incomplete transfer, corrupt artifact, or inaccessible library/share. Restore authenticated intact artifacts and minimum required read/write access for the provisioning identity. Check storage locally and remotely. Don't grant broad access to plaintext owner files. New-ShieldedVM isn't found or behaves differently from the example. Missing/unapproved GuardedFabricTools version, command shadowing, dependency mismatch, or unsupported target-build behavior. Verify command source and the reviewed package version, including nested modules and installed interfaces. The module's pinned 1.1.1 example isn't evidence of universal Windows Server 2025 compatibility. Don't bypass execution policy or import an unreviewed replacement. A generation 2 VM exists but isn't actually shielded or fully provisioned. Only New-VM , vTPM enablement, or state-encryption settings were applied; the owner's policy and initialization were omitted. Use a validated complete orchestration that applies the PDK-derived policy/KP and secure provisioning. Inspect resulting firmware/security state and provisioning status. Don't invent a Set-VMSecurity -ShieldingRequested parameter. VM creation or template copy fails before specialization. Insufficient disk space or memory, invalid path, conflicting VM name, missing virtual switch, permissions, or storage failure. Correct the specific capacity/configuration issue and reconcile partial artifacts before retrying. The example CPU/memory values aren't capacity recommendations for every workload. Specialization hangs or Windows setup reports an input error. Invalid unattend XML, missing placeholder, invalid product key, unsupported setup setting, or script failure. Validate the owner's answer file and supplied values against the target image. Reproduce with nonsecret test data in an isolated owner-approved lab. Correct the PDK for a controlled new attempt; don't downgrade a production VM's policy for console access. Computer naming, networking, or enrollment fails during setup. FSK tokens don't match owner-approved placeholders, address/DNS data is wrong, duplicate names/IPs exist, or domain-join credentials/connectivity fail. Reconcile exact placeholders and the intended network/identity services. Keep credentials inside owner-controlled encrypted inputs, not fabric-readable specialization values. A guest setup script or endpoint certificate isn't installed. A file referenced by the answer file wasn't included in the PDK, the path is wrong, or required secret access isn't available. Have the owner correct package contents, paths, and setup logic. Verify the resulting guest endpoint identity. Don't expose private certificates to the fabric to work around packaging errors. A helper returns without an unambiguous success result. An asynchronous job is still running, failed, or wasn't correctly tracked. Inspect Get-ShieldedVMProvisioningStatus and its error details until the outcome is clear. A missing job/status object, running VM, or object returned by a helper isn't a success criterion. A retry fails because the VM already exists or has partial protected state. Creation/initialization was rerun blindly against an interrupted deployment. Isolate the failed instance, preserve logs and artifacts, and use the workflow's supported cleanup/retry procedure under change control. Don't overwrite a working VM, its disks, or its KP. Existing-VM conversion fails or leaves encryption incomplete. A new-template recipe was reused; generation, disk layout, guest, helper, or conversion prerequisites aren't satisfied. Take a recoverable backup and use a supported conversion path with owner approval and downtime. Verify actual volume encryption. Adding a vTPM alone neither encrypts existing disks nor removes old plaintext copies. Newly provisioned VMs share credentials or unintended identities. A reusable PDK/answer file hard-codes per-VM secrets or names. Stop further deployment, repair the owner-controlled identity/credential lifecycle, and rotate affected credentials. Reissue approved packages. Successful provisioning can still represent a serious configuration failure. References Create shielding data: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-tenant-creates-shielding-data Create a shielded VM with PowerShell: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-create-a-shielded-vm-using-powershell ShieldedVMProvisioning reference: https://learn.microsoft.com/powershell/module/shieldedvmprovisioning/?view=windowsserver2025-ps&preserve-view=true GuardedFabricTools package: https://www.powershellgallery.com/packages/GuardedFabricTools/1.1.1 Why can't the owner access the guest or confirm its protection? Once provisioning and key release succeed, distinguish network access, guest authentication, guest boot, and encryption state. The fabric operator and the guest administrator aren't the same security identity. Failure mode or symptom Likely causes Resolution overview The VM runs but RDP, WinRM, or SSH is unreachable. Wrong virtual switch/VLAN, address, DNS, route, guest firewall, disabled listener, or incomplete guest setup. Fabric administrators check the external path; the owner checks guest services and firewall/configuration. Restore the intended management endpoint without changing the KP or exposing guest secrets. The endpoint responds but authentication fails. Wrong, expired, locked, or unauthorized guest credentials; domain/time failure; incompatible authentication configuration. Repair guest/domain identity under owner authority and use a supported authentication method. Fabric membership isn't a guest credential bypass. RDP/WinRM certificate or SSH host-key identity doesn't match expectations. Wrong endpoint, stale deployment identity, certificate renewal/configuration failure, or possible interception. Stop trusting the session until the owner verifies the endpoint through an independent channel. Repair the correct identity and trust configuration; don't routinely dismiss warnings or accept replacement keys blindly. Basic console, debugging, disk inspection, or a security-weakening edit is blocked. The VM's shielded policy intentionally restricts fabric-administrator access. Treat the block as expected protection. Use owner-authenticated guest management or a documented owner recovery path. Don't replace the policy with encryption-supported mode as a routine operational fix. Enhanced Session Mode or PowerShell Direct isn't available. Unsupported host/guest combination, missing guest configuration, owner restrictions, or absent guest credentials. Check version-specific support and owner policy. Where supported, these paths still require guest authentication; their existence doesn't grant an administrator a way around owner authorization. Windows starts only to a BitLocker recovery prompt. Guest boot measurements or vTPM/protected state changed, a recovery condition occurred, or restored state doesn't match. Preserve the recovery point and use the owner's valid OS-volume recovery mechanism. Diagnose guest changes separately from host attestation. Don't reset the vTPM to bypass recovery. A vTPM is enabled, but the OS volume isn't protected. Provisioning didn't complete, BitLocker wasn't enabled, encryption is incomplete, or protection is suspended. The owner inspects volume status, encryption percentage, and protection status inside the guest, then repairs the supported encryption configuration and escrow. Host-side security properties don't prove guest encryption. A data disk is unencrypted or can't unlock after restart/restore. OS-volume shielding was assumed to cover data volumes, or data-volume protectors/normal-unlock dependencies weren't preserved. Establish and verify an owner-controlled encryption, unlock, and recovery policy for every data volume. Escrow the matching recovery material. The OS disk's shielded recovery key isn't a data-disk key. An application is compromised or traffic is readable despite shielding. Guest malware, vulnerable applications, stolen owner credentials, or unencrypted application traffic lie outside the host-isolation guarantee. Apply guest/application incident response, patching, authorization, and network encryption. Host attestation doesn't attest guest application behavior, and encrypted migration doesn't encrypt ordinary workload traffic. An encryption-supported or ordinary VM doesn't meet the malicious-admin requirement. VM policy was confused with host eligibility, BitLocker, or vTPM enablement. Reassess the required protection and use an approved shielding/reprotection workflow where supported. Verify existing copies and exposed secrets. Enrolling a host doesn't upgrade every VM's policy. References Guarded-fabric protection boundaries: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-and-shielded-vms Shielding-data guest configuration: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-tenant-creates-shielding-data Version-specific management improvements: https://learn.microsoft.com/windows-server/get-started/whats-new-in-windows-server-2019#troubleshooting-improvements Why do live migration, cluster placement, or failover fail? Check both ordinary Hyper-V dependencies and the destination's guarded-fabric eligibility. Neither source-host success nor a healthy HGS cluster proves that a particular destination can run the VM. Failure mode or symptom Likely causes Resolution overview The destination rejects a shielded VM. It fails attestation, can't reach an authorized KPS, lacks the required features, or its guardian isn't in the VM's KP. Validate the destination independently and repair the exact admission/key-release issue. Don't copy owner or HGS private keys onto destination hosts; generic local-vTPM certificate-copy advice isn't an HGS repair. Migration breaks after upgrading to Windows Server 2025. Credential Guard prevents the CredSSP delegation path previously used by live migration. Configure supported Kerberos constrained delegation or resource-based constrained delegation with the AD administrator. Keep Credential Guard enabled. Changing VirtualMachineMigrationAuthenticationType alone doesn't configure AD delegation. Kerberos migration fails with access/delegation errors. Missing/duplicate SPNs, wrong service/account scope, delegation restrictions, DNS/time problems, or an unsuitable initiating identity. Correct the specific Kerberos and constrained-delegation design for migration and, when needed, storage services. Test from the actual initiating identity. Don't enable unconstrained delegation as a generic repair. Migration fails only from a remote management session. The authentication path requires delegation that the remote workflow doesn't supply. Compare local and remote initiation under the approved design and repair supported delegation. Don't assume host attestation supplies credentials for the management second hop. Migration reports processor or VM configuration incompatibility. CPU-feature differences, incompatible firmware/settings, or a VM configuration version newer than the destination supports. Choose a compatible destination or a supported planned compatibility setting. Treat configuration-version upgrades as potentially irreversible for return migration. Don't assume a compatibility switch handles every CPU/vendor difference. Transfer stalls, times out, or stops after starting. Migration-network failure, inadequate bandwidth, packet loss, firewall restrictions, overloaded storage, or concurrency/capacity limits. Identify and repair the failing path or resource, reduce contention, and test at representative load. Preserve state/migration encryption rather than disabling it for throughput. VM storage isn't available at the destination. Missing shared-storage/CSV access, incorrect SMB/storage permissions, absent storage-move parameters, or an unsupported disk/device arrangement. Restore supported destination storage access or use the correct storage-migration workflow. Don't copy an active VHDX and treat it as a consistent VM transfer. Migration succeeds but the workload loses network access. Destination virtual switches, VLANs, network mappings, or guest addressing differ. Correct destination networking and verify owner access and application health. A completed compute move doesn't prove network continuity. Cluster failover can't start the VM even though HGS is healthy. The VM role isn't configured correctly, no eligible node/capacity exists, cluster quorum is lost, or shared resources are unavailable. Repair the workload cluster and placement policy. Registering a clustered VM role adds availability management, not shielding. The Hyper-V cluster and HGS cluster need independent health checks. Resume/import fails with saved-state incompatibility. The saved state is incompatible with the destination build/CPU or the restore set is damaged. Preserve a consistent backup and assess a supported owner-approved cold-start path. Discarding saved memory can lose unsaved work; don't delete vTPM, KP, protected configuration, or disk-chain files as a generic fix. Replica failover or failback fails. Destination host/key authorization wasn't maintained, storage/network mapping differs, or a previously trusted site was withdrawn. Revalidate the selected replica's KP, destination attestation, keys, current policies, and application consistency in isolation. Test failback as a separate operation; replication success alone isn't proof of recoverability. Drain/migration fails while a backup or checkpoint operation is active. A VM is in an incompatible transitional state, a backup lock is held, or checkpoint processing is stuck. Coordinate with the backup product and inspect VMMS/cluster/backup events. Complete or recover the supported operation before retrying. Don't manually delete checkpoint chains or restart shared services without assessing other VMs. Also see the module's destination-eligibility and protected-state guidance. References Credential Guard and Windows Server 2025 migration: https://learn.microsoft.com/windows/security/identity-protection/credential-guard/considerations-known-issues#live-migration-with-hyper-v-breaks-when-upgrading-to-windows-server-2025 Hyper-V migration setup: https://learn.microsoft.com/windows-server/virtualization/hyper-v/deploy/set-up-hosts-for-live-migration-without-failover-clustering Why do servicing, key rollover, or retirement cause an outage? Security evidence and cryptographic dependencies survive beyond the change window. Test both the current workload population and retained recovery points. Failure mode or symptom Likely causes Resolution overview An approved firmware, driver, OS, or Secure Boot update causes rejection. The intended change altered measured evidence or CI rules without coordinated approval/registration. Verify the actual update, stage matching approved artifacts, reattest, and start a protected test workload. Don't authorize unexplained measurements just because the change window was approved. Older hosts fail after HGS policy activation. New checks were activated before all hosts supplied the required evidence. Follow the module's cumulative-update order: guarded hosts first, then HGS nodes, then any separately required policy activation. For OS-version upgrades, use the specific compatibility matrix rather than assuming the same order. OS-version upgrade breaks mixed-version operation. Unsupported host/HGS combinations, old attestation versions, or configuration-version changes. Validate the exact source, target, and intermediate combinations. Stage the supported upgrade sequence and rollback boundaries. An older Windows Server upgrade article isn't blanket approval for every later release combination. New VMs work after key rollover, but older VMs or backups fail. Old KPs/PDKs authorize retired keys, and primary-key selection didn't update them. Restore an approved uncompromised recovery path or have owners update existing KPs. Inventory checkpoints, replicas, backups, and future-provisioning packages before disabling old keys. Some requests fail immediately after new keys become primary. The new key's private half or permissions weren't provisioned on every node. Restore consistent key access on all nodes before completing the transition. Test actual key release; public metadata publication alone doesn't prove usable private keys. An obsolete or weaker host configuration still passes after rollout. Temporary old baselines, CI policies, host registrations, or legacy versions were never retired. End the approved overlap window, withdraw obsolete artifacts with dependency review, and test fresh rejection. Don't keep a permanent weaker fallback simply to avoid documenting exceptions. Retirement destroys service availability or recovery capability. The last usable HGS key/node, required domain service, owner-key copy, or backup dependency was removed too early. Stop retirement, recover approved material where possible, and complete dependency-aware decommissioning. Clear-HgsServer and Uninstall-HgsServer have different domain/cluster consequences and aren't general reset commands. References HGS patching, key management, and decommissioning: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs Version-specific upgrade guidance: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-upgrade-to-2019 Why does withdrawing trust not immediately contain an incident? Trust-policy changes affect future authorization decisions. They don't erase already released secrets, stop every running workload, or rewrite historical backups. Failure mode or symptom Likely causes Resolution overview A deauthorized host continues running a shielded VM. The VM already obtained keys; existing health evidence or eligible cached material may remain relevant. Isolate the host and its management/network access under the incident plan, block placement, preserve evidence, and recover on clean infrastructure. Verify future rejection separately from immediate containment. A withdrawn host is still admitted elsewhere. Only primary HGS was updated, another registration/version remains, or a shared host key is still trusted. Inventory every applicable HGS service and actual identity, including fallback and legacy artifacts. Withdraw the correct identity everywhere and test fresh attestation. Don't remove a shared baseline to target one host. HGS or its signing/encryption keys are compromised. An attacker can alter admission or use the fabric's key-protection authority. Rebuild from clean administrative authority, review policy, establish clean keys, and have owners withdraw compromised guardians from existing KPs and future PDKs. Don't blindly restore unreviewed HGS state. An owner guardian key is compromised. The attacker has owner-level authority affecting that ownership group. Assess all dependent workloads and copies. Establish clean owner recovery/reprotection authority through a validated process and rotate exposed guest secrets. Rotating only HGS keys doesn't repair owner-key compromise. A publisher/template-signing key is compromised. Malicious images can be signed as an approved publisher. Stop new provisioning from the affected trust, preserve evidence, rebuild approved images, and revise owner-approved catalogs/version rules and publishing keys. Investigate already deployed images rather than merely re-signing them. Plaintext answer files, PFX passwords, owner keys, or decrypted recovery keys are exposed. Insecure staging, shared folders, transcripts, support bundles, or backup access. Restrict and preserve evidence through incident response, assess which secrets were disclosed, rotate/reprotect as required, and remove unauthorized copies under policy. Deleting one working file doesn't undo exposure. Rotating keys doesn't eliminate the exposure. Plaintext, old KPs, old keys, credentials, or backups were already copied. Assess the full exposure window and affected copies. Reprotect data and rotate guest/application secrets where required. Wrapping-key rotation isn't retroactive erasure. Shielding doesn't prevent deletion, outage, or hostile administration of the whole design. Fabric administrators retain availability control, or HGS/owner/publisher roles aren't genuinely separated. Protect backup retention and independent recovery, enforce administrative separation, and treat collapsed authority as a design failure. Shielding isn't an availability guarantee or protection from every trusted authority. Also see the module's compromise-response unit. References HGS separation of duties and key management: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs Protection boundaries: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-and-shielded-vms What fails when HGS or a site is unavailable? Distinguish a running VM from a cold start. A currently running workload doesn't demonstrate that it can restart, migrate, or recover after another failure. Failure mode or symptom Likely causes Resolution overview Running VMs survive an HGS outage but cold starts fail. Running VMs already have usable material; new starts lack an available authorized release path or eligible cache. Restore primary/fallback HGS or use previously established eligible offline operation. Avoid unnecessary VM/host restarts until dependencies are understood. Don't replace the KP. Fallback URLs are configured, but the recovery HGS rejects the host or VM. Enrollment, policies, keys, TLS trust, or KP authorization weren't maintained independently. Repair the missing approved dependency on the recovery service and host. Distinct fabric keys require owner authorization for that guardian; endpoint configuration doesn't copy policy or keys. Fallback doesn't override a primary security rejection. Availability fallback was mistaken for a way around failed trust checks. Correct the actual security condition. Don't design recovery HGS to accept hosts specifically rejected by primary policy. Fallback is for an unreachable/unresponsive service, not a weaker admission route. Offline caching is enabled, but the VM won't start offline. The host/VM never obtained eligible cacheable material, the build isn't supported, or the VM was only guest-restarted. While HGS is available, perform the required full VM shutdown and start under the approved caching policy, then test an isolated outage. Enabling the service flag alone doesn't populate every VM's cache. Offline startup worked before servicing but fails afterward. CI, Secure Boot, VBS identity, or other security-state changes invalidated the cache. Restore online attestation and obtain new eligible material after the approved change. Don't attempt to carry an old cache across a changed security identity. A cache on one host doesn't help another host or recovery site. Offline material is host-, VM-, and security-state-specific rather than portable recovery authority. Provide the destination with its own valid online authorization and any approved cache eligibility. Full-site recovery requires independent keys and VM backups, not copies of a lost host's cache. Operators expect a host restart to work offline in every case. Cache eligibility was inferred from a prior VM-only start rather than the exact reboot/security-state scenario. Test the intended restart and servicing cases. Maintain online recovery capacity for cases that invalidate eligibility; don't promise offline operation solely from a configuration flag. Disabling caching or withdrawing trust is expected to erase cached secrets immediately. Service policy was confused with deletion of material already released to another system. Apply explicit containment and assess cached/running-state exposure. Use caching only after approving its availability-versus-revocation tradeoff. References Fallback HGS and offline operation: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-branch-office Why do backups or disaster recovery fail? Recover the authorization, keys, and protected state needed by the selected recovery point. A recent VHDX isn't necessarily a recoverable shielded VM. Failure mode or symptom Likely causes Resolution overview VM backups exist, but recovery operators can't access them. Backup identity, DNS, MFA/secret access, repository decryption keys, or storage depends on the failed fabric. Restore independent recovery authority and infrastructure first. Keep tested access to encrypted off-site/offline or immutable copies outside routine fabric control. HGS state import fails. Wrong export password, damaged/incomplete file, uninitialized destination, incompatible build, or unresolved import errors. Verify the protected export, initialize a supported clean HGS destination, and inspect each reported error. Reconcile duplicates deliberately; don't assume a command returning some objects means full restoration. HGS import succeeds but key release still fails. The export contains public references, not the thumbprint-referenced/HSM private keys. Restore the original private keys through their independent recovery method and authorize every node. HGS state import can't reconstruct missing nonexported keys. HGS restores without working DNS, TLS, AD identity, or JEA. Those dependencies were assumed to be part of the HGS state export. Recreate the service identity, domain/cluster configuration, administration, and TLS endpoints as required. Restore legacy AD host-group membership separately; an HGS export preserves group SIDs, not membership. Review policy and then test client attestation and actual VM key release. The restored HGS service admits retired hosts or uses compromised keys. An old or unreviewed export restored historical trust. Review all imported identities, versions, policies, and keys before exposing the service. Remove withdrawn trust under the incident/recovery plan. A usable backup isn't automatically an approved security configuration. Rebuilding HGS with the old certificate names doesn't unlock existing VMs. The new certificates contain different keys. Restore the original uncompromised key material or use surviving owner authority to authorize clean replacement keys in existing KPs. Names can't recreate cryptographic continuity. A disk-only restore can't start the protected VM. VM configuration, KP, vTPM/protected state, or checkpoint-chain dependencies weren't backed up. Restore a supported consistent whole-VM set. If that set is unavailable, assess the separate owner-authorized guest-volume recovery path. Attaching the VHDX to a new VM isn't equivalent to restoring it. A restored VM has BitLocker or state-consistency errors. Disks, configuration, KP, vTPM state, or differencing-disk parents come from different points in time. Recover a consistent known-good set without editing the originals. Preserve disk-chain relationships and use supported backup tooling. Don't mix state files to see which combination boots. A pre-rollover backup can't start or authorizes a compromised guardian. Its KP still depends on historical keys and permissions. Inspect in isolation. Use an approved retained uncompromised key or clean owner-authorized recovery/reprotection. Never re-enable a compromised guardian in production solely for convenience. Backup succeeds but application/item-level restore fails. Crash consistency was mistaken for application consistency, or the product lacks the required shielded-VM/guest-aware support. Verify product/version/operation support and use owner-authorized guest credentials where required. Test application recovery and data integrity, not only successful VM boot. A checkpoint or replica doesn't provide an independent recovery copy. It shares storage, administrative control, chain dependencies, or unavailable keys with production. Maintain separately protected backups and test full restoration. Checkpoints and replication can support recovery operations but don't replace independent retention or recoverable keys. Recovery exceeds RTO or loses more data than the RPO allows. Security dependencies are older than VM backups, retrieval was never rehearsed, or the recovery site lacks capacity/network mappings. Define and test RPO/RTO for HGS, owner keys, KPs, original PDKs, VM state, and applications together. Refresh the recovery inventory after policy/key changes and rehearse dependency-ordered recovery. Recommended order: recover clean authority and backup access; restore power, networks, DNS, time, and required identity services; rebuild/restore HGS and keys; establish eligible hosts/storage; restore consistent VMs; then have owners validate guest and application health. Also see the module's artifact-custody and recovery units. References HGS backup and recovery: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs Why does Windows guest BitLocker recovery fail? This is an owner-controlled guest-volume operation, not HGS service restoration. Work on an isolated consistent copy and preserve the original recovery point. Failure mode or symptom Likely causes Resolution overview Save-ShieldedVMRecoveryKey can't find the expected material. Wrong disk, damaged/inaccessible data, or a VM outside the documented eligibility. Select the Windows guest's OS disk and verify provenance. The documented mechanism requires a Windows shielded VM deployed with a PDK created on Windows Server version 1709 or later. Use another valid owner-escrowed protector if this path doesn't apply. Extraction works, but Unprotect-ShieldedVMRecoveryKey fails. Wrong/or newly generated PDK, missing matching owner private keys, or private-key access failure. Retrieve the original deployment PDK and usable matching owner keys on the trusted recovery computer. A replacement file with the same name or an HGS backup isn't equivalent. The recovered file isn't accepted as a typed recovery password. The output is a decrypted .bek key file, not necessarily a numerical recovery-password string. Use the supported BitLocker recovery procedure that accepts the key file for the matching OS volume. Protect the decrypted file as plaintext recovery authority. The OS recovery key doesn't unlock an encrypted data volume. The material is specific to the OS volume; data volumes have independent protectors. Retrieve that data volume's owner-escrowed recovery material. Don't overwrite or remove its protectors while searching for a working key. A correct key doesn't restore a healthy workload. The filesystem/application is damaged, the disk chain is incomplete, or the recovery point is inconsistent. Recover the best consistent backup and use supported volume/application repair on copies. Decryption establishes access, not data integrity or application consistency. Original owner authority and all valid recovery protectors are unavailable. Irrecoverable key loss or destruction of the only consistent recovery artifacts. Exhaust independently retained legitimate recovery sources and assess whether another authorized path remains usable. If none exists, data can be permanently unrecoverable. Fabric-administrator access, new certificates, or a new vTPM don't provide a bypass. After recovery, restore the intended encryption and authorization, validate owner access, and remove temporary decrypted material according to the approved handling policy. References Save-ShieldedVMRecoveryKey: https://learn.microsoft.com/powershell/module/shieldedvmdatafile/save-shieldedvmrecoverykey?view=windowsserver2025-ps&preserve-view=true Unprotect-ShieldedVMRecoveryKey: https://learn.microsoft.com/powershell/module/shieldedvmdatafile/unprotect-shieldedvmrecoverykey?view=windowsserver2025-ps&preserve-view=true How do I recover from TPM failures or an offline CA? Use these recovery sequences when a Hyper-V host or HGS server reports TPM errors, a server stops at BitLocker recovery after a hardware change, or an unavailable certification authority (CA) blocks certificate operations. The host TPM, HGS guardian keys, and CA signing keys have different recovery dependencies. Replacing one doesn't recreate another. How do I recover when a Hyper-V host's TPM fails? Symptoms can include TpmError , a missing or not-ready TPM, or a host BitLocker recovery prompt. Firmware configuration, driver problems, and TPM lockout can resemble hardware failure. Confirm the cause with the server OEM rather than treating every attestation failure as a failed chip. The host's physical TPM is separate from each guest's vTPM. Protect running workloads and collect evidence. Stop new placement on the affected host. Preserve running VMs where safe; don't restart them merely to test the fault. On the affected host, inspect Get-Tpm , Get-HgsClientConfiguration , and TPM, BitLocker, and HostGuardian events. Record recent firmware, motherboard, boot, and policy changes. Retrieve recovery material before hardware changes. Obtain the host's escrowed BitLocker recovery password or other valid recovery protector, plus recovery material for any separately encrypted host data volumes. Preserve accessible, consistent VM backups and their configuration, KPs, vTPM/protected state, disks, and checkpoint relationships. A guest's BitLocker recovery key doesn't unlock the host's encrypted volume. Recover workload availability on an eligible host. If supported migration can still complete, move workloads to a healthy guarded host that passes attestation and can obtain keys authorized by each VM's existing KP. Otherwise, use a supported cluster-failover or consistent whole-VM restore workflow. Don't assume live migration can succeed after the TPM fault. Physical TPM failure doesn't, by itself, permanently bind an intact shielded VM to the failed host. Repair or replace the failed platform component. Follow the OEM's supported hardware and firmware procedure. Use the host recovery protector if BitLocker blocks access. After repair, restore the required TPM 2.0, UEFI, Secure Boot, VBS, and enforced CI configuration. Complete the supported BitLocker post-recovery process so protection uses the intended TPM; verify protection is active and recovery material is escrowed before returning the host to production. Don't leave BitLocker suspended. Register a replacement TPM as a new identity. If the TPM was replaced, capture its actual platform identifier with Get-PlatformIdentifier and have the HGS administrator register it on every applicable primary and fallback HGS deployment. Verify the replacement TPM's endorsement-certificate chain. Reuse existing approved baselines and CI policies only when they still match; approve new artifacts only for an explained configuration change. An old identifier export isn't a backup of the original TPM's private keys. Withdraw the retired host identity without deleting shared baselines or weakening attestation. Prove recovery before resuming placement. Restore the correct primary and fallback endpoints if the host was rebuilt, then obtain fresh attestation with Test-HgsClientConfiguration . Test a controlled shielded-VM cold start and have the owner verify guest access, encryption, and application health. Don't rely on the failed host's old offline cache. Where caching is approved, establish new eligibility with the required full VM shutdown and start while HGS is available. Recovery limit: If the only VM copy is on an encrypted host volume and neither a valid host-volume recovery protector nor an independent backup is available, HGS can't unlock that host volume for you. If only a guest disk survives, assess the separate owner-authorized guest-volume recovery path; creating a new vTPM or KP isn't equivalent to recovering the original VM. References Guarded-host and TPM troubleshooting: https://learn.microsoft.com/troubleshoot/windows-server/virtualization/troubleshoot-guarded-hosts TPM identity registration and baseline capture: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-tpm-trusted-attestation-capturing-hardware BitLocker recovery and post-recovery tasks: https://learn.microsoft.com/windows/security/operating-system-security/data-protection/bitlocker/recovery-process Offline key-material caching requirements: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-branch-office How do I recover when an HGS server's TPM fails? First establish whether the error concerns the HGS server itself or a Hyper-V host trying to attest. HGS operating in TPM-attestation mode validates the Hyper-V hosts' TPM evidence; that setting doesn't automatically store HGS guardian keys in the HGS server's TPM. This module creates HGS certificates with the Microsoft Software Key Storage Provider. A local TPM failure can prevent the server's BitLocker-protected OS from unlocking or affect other TPM-protected credentials, but it doesn't automatically destroy those software-backed HGS keys. Determine the outage scope. Check individual HGS nodes, not just the service name. Keep healthy nodes serving requests and use the approved maintenance procedure to keep an unusable node out of the service path. Preserve quorum, DNS, and remaining capacity. Avoid unnecessary guarded-host or VM restarts while the available key-release paths are reduced. Secure the available recovery material. Retrieve the affected HGS server's own BitLocker recovery protector before resetting or replacing its TPM. From a healthy HGS node, preserve an up-to-date Export-HgsServerState backup and its password through the approved custody process. Also locate the original HGS private-key backups or HSM recovery capability and the separately maintained TLS material. If no node is available, use the independently retained backups. Repair the node or replace it in the surviving cluster. Recover OS access and repair the hardware through the OEM-supported process, then restore the server's BitLocker protection. If the node can't be repaired, follow the documented procedure to add a replacement HGS node to the surviving deployment. In this module, HGS nodes are domain controllers: coordinate domain-controller replacement and AD health rather than blindly rolling back a server disk image. Losing one node doesn't require rebuilding the entire HGS forest or changing every VM's guardian. Use HGS disaster recovery if the entire cluster is lost. Reestablish the required clean identity, DNS, time, and administration dependencies; install and initialize a replacement HGS deployment; then use Import-HgsServerState with the approved export and its password. Review restored host identities, baselines, CI policies, and withdrawals before admitting workloads. Prefer the documented HGS rebuild-and-state-restore procedure over assuming a whole-server image reconstructs the distributed service. Restore usable keys on every node. PFX-based keys can be restored from the HGS export and replicated by HGS. Thumbprint-referenced and HSM-backed private keys require their separate restoration or provider-access procedure and gMSA permissions on each node. Restore TLS certificates and bindings separately; they aren't supplied by the HGS state export. Preserve the original, uncompromised fabric key identities when possible. Neither a new certificate with the old subject nor a replacement TPM recreates a lost private key. Validate the complete service and refresh backups. Confirm AD, gMSA, DNS, TLS, and HGS health, including actual private-key use on every node. Run HGS diagnostics, obtain fresh host attestation, and test key release with a controlled shielded-VM cold start. Have the owner verify the workload, then restore the intended node redundancy and take a new protected HGS export. Recovery limit: If all usable copies of the required HGS private keys are lost, use another fabric guardian already authorized by the VM's KP, or have the owner use surviving owner keys to authorize replacement fabric keys in the existing KP. Preserve its secret and keep the VM off throughout that exchange. A new PDK doesn't repair deployed KPs. Without an authorized key path, owner authority, or valid guest recovery material, some VM data can be permanently unrecoverable. References HGS prerequisites and deployment model: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-prepare-for-hgs Single-node and complete-cluster HGS recovery: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs#backing-up-and-restoring-hgs HGS private-key storage, replication, and replacement: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs#key-management BitLocker recovery for the HGS server: https://learn.microsoft.com/windows/security/operating-system-security/data-protection/bitlocker/recovery-process How do I maintain or recover the fabric when the CA is offline? An unavailable certificate-issuance service, an intentionally offline root CA, and an unreachable certificate-validation endpoint aren't the same failure. Existing certificates don't become invalid merely because the issuing service is stopped. HGS doesn't need a newly issued certificate for every VM start, and its signing/encryption key-container certificates have different expiry and revocation semantics from TLS certificates. HTTPS and other certificate consumers can still fail when certificate validity, chain building, or required revocation checks fail. Identify which CA and operation are affected. Determine whether the CA is an offline root used during controlled signing operations or an issuing CA needed for pending requests and renewals. This module uses a standalone CA; don't assume enterprise certificate templates or autoenrollment apply. If existing HGS operations still work, preserve availability while arranging the required CA operation rather than replacing working guardians. Separate issuance from certificate validation. From the affected host or management network, check certificate dates, trusted roots and intermediates, service-name matching, and time. Inspect the exact authority information access (AIA) locations and certificate revocation list (CRL) distribution points (CDPs) embedded in the affected certificate chain. A running CA doesn't prove those locations work, and a stopped CA doesn't make separately hosted publication endpoints unavailable. Restore the publication path when chain or revocation retrieval fails. Repair DNS, network access, web or directory publication, and the transfer of the correct CA certificates and signed CRLs. An existing valid CRL can be served without keeping the CA's signing service online. If a required CRL has expired, the responsible CA must publish a new signed CRL; repair parent-CA publication dependencies as needed. For an offline root, use the approved controlled startup, signing, and transfer process, then return it offline. Keep any required base and delta CRLs consistent. Don't disable revocation checking or change clocks to conceal expiry. Repair an unexpectedly unavailable issuing CA. Have the CA administrator inspect the AD CS service ( CertSvc ), events, storage, database/log access, and signing-key or HSM availability. Correct the identified dependency under the CA's recovery procedure. Restoring an HTTP CDP alone doesn't restore issuance, and repeatedly restarting the service won't recreate a missing signing key or repair a damaged database. Restore the original CA if its server is lost. Use the supported AD CS restoration/migration procedure with the original CA identity, CA certificate and private keys, database and logs, registry configuration, and CAPolicy.inf where used. Restore HSM access and any applicable AD dependencies separately. Match the standalone/enterprise and root/subordinate CA types. Prevent the old and replacement servers from operating independently as the same CA. Keep the AIA/CDP locations referenced by existing certificates available; changing extensions on the new server affects only newly issued certificates. Complete pending issuance and required renewals. Retain each outstanding request and its matching private key, obtain the approved issued certificate, and accept it on the system holding that key. Renew expired TLS or CA certificates through the appropriate PKI procedure, distribute the required public trust chain, and update TLS bindings on each HGS node. Restoring the CA or replacing a TLS certificate doesn't, by itself, require changing HGS guardian keys. If fabric signing/encryption key pairs really change, use the owner-authorized existing-KP workflow rather than assuming a same-name certificate preserves access. Verify service continuity before closing the outage. Test permitted enrollment/renewal, chain building, and revocation retrieval from representative client networks using both existing certificates and an approved test certificate. Confirm fresh HGS attestation and a controlled VM key-release test. Monitor certificate and CRL expiry and verify publication before the next planned offline period. A deliberately offline root can remain offline between its scheduled operations while its publication endpoints stay available. Recovery limit: If the CA signing key is irretrievably lost, a replacement CA with the same name can't issue certificates or fresh CRLs under the original key. Plan a controlled trust-chain transition and affected-certificate replacement. A CA backup doesn't normally contain the HGS private keys generated on HGS nodes; preserve and recover those keys independently. References CA backup and restoration/migration: https://learn.microsoft.com/windows-server/identity/ad-cs/migrate-certification-authority CRL publication, offline-root operation, and CDP recovery: https://learn.microsoft.com/windows-server/identity/ad-cs/operate-certificate-revocation-crl-cdp PKI design, AIA, and CDP requirements: https://learn.microsoft.com/windows-server/identity/ad-cs/pki-design-considerations HGS signing/encryption certificate expiry and revocation behavior: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs#renewing-or-replacing-keys Why are diagnostics inconclusive or misleading? Missing or stale evidence can delay recovery and hide trust failures. Collect enough information to identify the layer without disclosing the secrets that layer protects. Failure mode or symptom Likely causes Resolution overview A requested event channel is absent or has no events. Wrong computer/channel/build, disabled logging, incorrect time window, insufficient access, or no attempt occurred. Discover installed HostGuardian, CodeIntegrity, VMMS, worker, and cluster channels under the correct authority. Correlate timestamps and generate an approved diagnostic attempt. Don't suppress errors and label an empty result healthy. Monitoring is green while cold starts or key release fail. Monitoring tests only TCP, one node, a running VM, or a previously cached result. Monitor the full authorized path: fresh attestation, key usability on each node, controlled cold starts, fallback readiness, and recoverability. Keep synthetic tests isolated and non-destructive. Events across systems appear contradictory. Clock drift, stale status, different responding HGS nodes, mixed modes/versions, or unrelated attempts. Align the timeline, operation/VM identity, endpoint, node, and change record. Retain the original output alongside the interpretation. Logs disappeared after compromise or rollback. Evidence existed only on an affected host, retention was too short, or restoration replaced local logs. Forward and retain security events outside routine fabric control and preserve evidence before remediation. Missing evidence doesn't prove that no compromise occurred. A diagnostic bundle contains keys or tenant secrets. Transcripts captured PFX passwords, recovery protectors, answer files, private keys, or sensitive configuration. Restrict the bundle immediately, apply the incident-handling process, and produce a sanitized minimum necessary copy for support. Don't paste decrypted .bek files or private-key material into tickets. No listed cause explains the failure. A product defect, hardware fault, vendor integration, or unmodeled combination requires deeper investigation. Capture a minimal reproduction on a supported isolated system, correlate version-specific release notes, and escalate with evidence. Don't fabricate an error-code interpretation or progressively disable protections until the symptom disappears. References HGS diagnostics: https://learn.microsoft.com/powershell/module/hgsdiagnostics/get-hgstrace?view=windowsserver2025-ps&preserve-view=true HGS event monitoring: https://learn.microsoft.com/windows-server/security/guarded-fabric-shielded-vm/guarded-fabric-manage-hgs Guarded-host troubleshooting: https://learn.microsoft.com/troubleshoot/windows-server/virtualization/troubleshoot-guarded-hostsActive Directory Domain Services modules on Microsoft Learn
The following is a list of Active Directory Domain Services modules on Microsoft Learn sorted from introductory to advanced. Is there an Active Directory topic that isn't listed here that you'd like to see covered? (AD CS modules are coming). If you know all this, remember we have the free 45 minute practical Active Directory administration test that gets you a validated Microsoft credential. You can take the test here https://aka.ms/ADDSAppliedSkillTest. Introductory 1. Introduction to AD DS URL: https://learn.microsoft.com/en-us/training/modules/introduction-to-ad-ds/ Assessment: BEGINNER This module establishes the essential AD DS vocabulary and mental model. It introduces directory services, users, groups, computers, forests, domains, sites, domain controllers, and organizational units, then shows how directory objects and their properties are managed. It should come first because nearly every later module assumes familiarity with these structures and their relationships. Intermediate 2. Create and manage Active Directory objects URL: https://learn.microsoft.com/en-us/training/modules/create-manage-active-directory-objects/ Assessment: INTERMEDIATE This module turns the foundational concepts into routine administrative skills. It covers users, groups, computers, organizational units, object properties, object creation and configuration, bulk user-account management, and basic domain controller maintenance. The module is the natural bridge from understanding AD DS to operating it. 3. Manage Active Directory Domain Services using PowerShell cmdlets URL: https://learn.microsoft.com/en-us/training/modules/manage-active-directory-domain-services-use-powershell-cmdlets/ Assessment: INTERMEDIATE This module applies PowerShell to the object-management tasks learned previously. It introduces cmdlets for creating and maintaining users, groups, group memberships, computers, organizational units, and other directory objects, giving learners a repeatable and scalable alternative to graphical administration. Prior PowerShell familiarity is expected. 4. Deploy and manage Active Directory Domain Services domain controllers URL: https://learn.microsoft.com/en-us/training/modules/deploy-manage-active-directory-domain-services-domain-controllers/ Assessment: INTERMEDIATE This module moves from directory objects to the servers that host the directory. It reviews forests and domains, explains domain controller topology, covers domain controller deployment and movement between sites, and introduces operations master roles. It provides the deployment context needed before studying domain controller maintenance and role placement in greater depth. 5. Manage AD DS domain controllers and FSMO roles URL: https://learn.microsoft.com/en-us/training/modules/manage-active-directory-domain-services-flexible-single-master-operation-roles/ Assessment: INTERMEDIATE This module deepens domain controller administration through deployment, maintenance, backup and recovery considerations, global catalog placement, Flexible Single Master Operations (FSMO) role placement and management, and an introduction to schema management. It builds on the preceding deployment module and prepares learners for later architecture, recovery, and design topics. 6. Implement Group Policy Objects URL: https://learn.microsoft.com/en-us/training/modules/implement-group-policy-objects/ Assessment: INTERMEDIATE This module introduces domain-based Group Policy Objects (GPOs), including scope, inheritance, creation, configuration, storage, administrative templates, and the Central Store. It should precede the other Group Policy modules because it explains both the logical processing model and the underlying storage components on which later policy and troubleshooting work depends. 7. Create and configure Group Policy Objects in Active Directory URL: https://learn.microsoft.com/en-us/training/modules/create-configure-group-policy-objects-active-directory/ Assessment: INTERMEDIATE This module reinforces GPO definition, scope, inheritance, and domain-based configuration, then extends those skills into domain password policy and fine-grained password policy. Although some content overlaps the preceding module, its identity-policy focus makes it a useful second step after learners understand GPO storage, templates, and general processing. 8. Manage security in Active Directory URL: https://learn.microsoft.com/en-us/training/modules/manage-security-active-directory/ Assessment: INTERMEDIATE This module addresses practical directory hardening through user rights, access restrictions, delegated permissions, the Protected Users group, Windows Defender Credential Guard, NTLM blocking, and identification of problematic accounts. It connects identity administration and Group Policy knowledge to least-privilege operations and modern credential protection. 9. Implement and manage Active Directory Certificate Services URL: https://learn.microsoft.com/en-us/training/modules/implement-manage-active-directory-certificate-services/ Assessment: INTERMEDIATE This module introduces public key infrastructure and Active Directory Certificate Services (AD CS), including certification authority types, AD CS design and implementation, certificate enrollment, revocation, and trust. It belongs after core directory security because certificates extend AD-based authentication and authorization into a broader trust infrastructure. Advanced 10. Understand Active Directory Group Policy security settings URL: https://learn.microsoft.com/en-us/training/modules/understand-active-directory-security-policies/ Assessment: ADVANCED This module provides an in-depth treatment of security policy design and operation at scale. It covers password, lockout, and Kerberos account policies; fine-grained password policies; user rights; security options; auditing; secured groups, services, registry keys, files, and logs; network and application policies; and Windows Server 2025 OSConfig baselines. Its assumptions about GPO processing, security identifiers, access tokens, ACLs, Kerberos, and NTLM make it the bridge from practical GPO administration to protocol-level authentication hardening. 11. Active Directory Domain Services authentication and Kerberos hardening URL: https://learn.microsoft.com/en-us/training/modules/active-directory-authentication-kerberos/ Assessment: ADVANCED This module examines Windows authentication end to end, explaining how SSPI, Negotiate, NTLM, and Kerberos interact and tracing Kerberos from Ticket Granting Ticket issuance through service-ticket presentation and authorization. It covers diagnosis of Service Principal Name, delegation, Privilege Attribute Certificate, encryption-type, and NTLM fallback problems; staged migration from NTLM to Kerberos; RC4 auditing and remediation for Windows Server 2025; and planning for PKINIT agility, Kerberos encryption policy, SMB NTLM blocking, and password-change hardening. It caps the security stage because it requires learners to combine policy, identity, logging, and authentication-protocol knowledge. 12. Understand how Active Directory Domain Services uses DNS URL: https://learn.microsoft.com/en-us/training/modules/understand-active-directory-domain-name-system/ Assessment: ADVANCED This module explains and diagnoses the DNS mechanisms on which AD DS depends. It traces service discovery and domain controller location through SRV records, LDAP ping, site selection, and caching, and examines AD-integrated zones, application directory partitions, secure dynamic updates, registration, replication convergence, baselining, and failure recovery. This knowledge is essential before designing or troubleshooting site topology and replication. 13. Active Directory sites, topology, and replication URL: https://learn.microsoft.com/en-us/training/modules/active-directory-site-replication/ Assessment: ADVANCED This module develops a detailed model of sites, subnets, site links, costs, bridgeheads, connection objects, the Knowledge Consistency Checker (KCC), and the Inter-Site Topology Generator (ISTG). It also covers DC Locator behavior, topology anti-patterns, update sequence numbers, replication vectors, object metadata, replication failure diagnosis, and the Windows Server 2025 replication priority boost. The module supplies the replication foundation needed for the deeper storage and design modules that follow. 14. Understand the Active Directory Domain Services database and SYSVOL URL: https://learn.microsoft.com/en-us/training/modules/understand-active-directory-database/ Assessment: ADVANCED This module explains how a domain controller stores and replicates directory data and SYSVOL through separate mechanisms. It distinguishes the logical directory from the local ESE database and directory partitions, traces transactional writes and logical AD DS replication, explains SYSVOL access and DFS Replication, follows a GPO across its directory and file-system components, and defines safe backup, recovery, virtualization, and version boundaries. It is especially useful for diagnosing whether a failure lies in directory data, SYSVOL, replication, or client access. 15. Understand the Active Directory Domain Services Schema URL: https://learn.microsoft.com/en-us/training/modules/understand-active-directory-schema/ Assessment: ADVANCED This module examines the forest-wide schema that defines directory object classes, attributes, syntax, inheritance, links, identifiers, search behavior, security metadata, and global catalog content. It covers safe inspection, schema versions, governance, extension design and deployment, indexing and replication effects, access control, and schema-related troubleshooting. The topic is advanced because schema changes are forest-wide, effectively permanent, and demand disciplined testing and change control. 16. Design a single-domain Active Directory forest URL: https://learn.microsoft.com/en-us/training/modules/design-single-domain-active-directory-forest/ Assessment: ADVANCED This module combines the preceding operational knowledge into a resilient single-domain forest design. It covers durable DNS and UPN namespaces, AD-integrated DNS, sites, subnets, site links, replication convergence and failure behavior, and placement of domain controllers, DNS servers, global catalogs, FSMO roles, and the authoritative time source. Learners also validate designs against objective evidence and failure scenarios. 17. Design a multi-domain or multi-forest Active Directory environment URL: https://learn.microsoft.com/en-us/training/modules/design-multi-domain-forest-trust/ Assessment: ADVANCED This module extends single-domain design to scenarios requiring security isolation, legal separation, administrative autonomy, mergers, or replication boundaries. It addresses domain trees and partitions, namespace coexistence, delegation, GPO scope, schema governance, trust direction and security, cross-forest name resolution, global catalog and authentication paths, advanced replication and capacity, and read-only domain controller branch designs. It should follow single-domain design because additional domains and forests introduce complexity that must be justified. 18. Manage advanced features of AD DS URL: https://learn.microsoft.com/en-us/training/modules/manage-advanced-features-of-ad-ds/ Assessment: ADVANCED This module brings together several specialized administration scenarios: creating trust relationships, implementing Enhanced Security Administrative Environment (ESAE) forests, monitoring and troubleshooting replication, and creating custom AD DS partitions. These tasks rely on a mature understanding of forest boundaries, trusts, security, partitions, and replication, so they are best approached after both architecture modules. 19. Deploy and manage Azure IaaS Active Directory domain controllers in Azure URL: https://learn.microsoft.com/en-us/training/modules/deploy-manage-azure-iaas-active-directory-domain-controllers-azure/ Assessment: ADVANCED This module applies established AD DS administration to Azure infrastructure. It compares directory and identity service options, prepares Azure virtual networking for domain controllers, deploys and configures AD DS on Azure virtual machines, installs a replica domain controller, and creates a new forest on an Azure virtual network. The broad prerequisites in on-premises AD DS, Azure IaaS, networking, resiliency, security, PowerShell, automation, and monitoring make this advanced in the proposed path. 20. Active Directory Domain Services migration URL: https://learn.microsoft.com/en-us/training/modules/active-directory-domain-services-migration/ Assessment: ADVANCED This module evaluates how to move an existing AD DS environment to Windows Server 2025. It compares in-place forest upgrade with migration to a new forest, then outlines the process for each approach. Although concise, migration is placed late because selecting and executing a safe strategy requires solid knowledge of domain controllers, operations roles, DNS, replication, security, architecture, recovery, and change management. 21. Troubleshoot Active Directory URL: https://learn.microsoft.com/en-us/training/modules/troubleshoot-active-directory/ Assessment: ADVANCED This module provides broad recovery and troubleshooting coverage for AD DS failures and degraded performance. It addresses restoring deleted objects with the AD Recycle Bin, recovering the AD DS database and SYSVOL, troubleshooting replication, and diagnosing hybrid authentication issues. The potential impact of recovery operations and the need to correlate several directory subsystems make this an advanced operational module despite its relatively compact scope. 22. Troubleshoot Active Directory Domain Services replication URL: https://learn.microsoft.com/en-us/training/modules/troubleshoot-active-directory-replication/ Assessment: ADVANCED This module develops a disciplined process for diagnosing replication failures with Repadmin, DCDiag, PowerShell, and event logs. It teaches learners to distinguish true failures from expected latency or transient topology states, isolate DNS, network, authentication, topology, and domain controller health dependencies, recognize data-consistency risks, apply a targeted correction or escalate safely, and verify convergence and the original business symptom. This diagnostic discipline should precede advanced recovery so learners can distinguish a correctable replication fault from a broader consistency failure that warrants restoration. 23. Advanced Active Directory back up and recovery URL: https://learn.microsoft.com/en-us/training/modules/active-directory-backup-recovery/ Assessment: ADVANCED This module treats AD DS backup and recovery as Tier 0 architecture and incident response rather than a single restore operation. It covers backup design around recovery time and recovery point objectives, topology, retention, and trust boundaries; evidence-based backup validation; selection of object, domain controller, domain, or forest recovery scope; recovery of objects, attributes, hierarchies, DNS, and SYSVOL; domain and forest sequencing involving FSMO roles, RID state, global catalogs, and trusts; safe virtualized domain controller recovery; and comprehensive validation before reconnection. It is the final capstone because safe recovery requires the learner to integrate nearly every preceding topic while making high-impact decisions under controlled change and security processes.Advanced Windows Firewall
If you administer Windows, you’re already aware of Windows Firewall. You might use it to allow an application, open a port, or block unwanted inbound traffic. But those familiar tasks only scratch the surface of what it can do. The Microsoft Learn module Understand advanced Windows Firewall takes you beyond basic rules and introduces Windows Firewall as a powerful platform for host-based segmentation, authenticated access, traffic protection, operational evidence, and incident response. In the module you'll learn about the following: Rule creation and management View effective, enabled rules in the ActiveStore . Inspect the port, address, application, and service filters associated with a rule. Create precisely scoped inbound rules for services such as HTTPS, WinRM, Remote Desktop, and WMI. Enable, disable, modify, and remove existing rules with PowerShell. Define a traffic contract before creating a rule. Correctly distinguish local and remote ports and addresses. Scope rules by protocol, port, address, application, service, profile, user, and computer. Restrict rules to stable executable paths, Windows services, or packaged application identities. Limit access to management subnets, jump hosts, privileged workstations, application tiers, and collectors. Create consistent rule names and groups for ownership and automation. Firewall profiles Apply different policies to Domain, Private, and Public networks. Enable the firewall and block unmatched inbound traffic on every profile. Restrict administrative exceptions to the profiles that require them. Inspect active network profiles and their default actions. Design policies that remain secure during DNS, routing, domain-controller, or network-adapter failures. Test how network failure states affect profile selection. Host segmentation Build a traffic matrix describing permitted communication between device and application tiers. Implement default-deny inbound segmentation. Block unnecessary workstation-to-workstation communication. Preserve approved management, monitoring, application, recovery, and domain-member traffic. Reduce lateral movement through controlled management paths. Deliver firewall policy centrally through Group Policy. Disable local firewall-rule merging. Disable local connection-security-rule merging. Stage enforcement through logging, discovery, pilots, and role-based deployment. Define success criteria and maintain a tested rollback path. IPsec and identity-based access Design IPsec connection security rules for peer authentication. Provide packet integrity, replay protection, and optional encryption. Select Kerberos or certificate-based authentication for different trust scenarios. Protect legacy plaintext applications without changing the application. Coordinate secure firewall rules with compatible connection security rules. Use request authentication during deployment before enforcing required authentication. Correctly define IPsec endpoints and traffic selectors. Require authenticated traffic before allowing access. Authorize traffic by Active Directory user-group membership. Require both an authorized user and an authorized managed computer. Combine identity with network, service, application, and profile restrictions. Create narrowly scoped authenticated bypass rules. Design governed identity exceptions. Validate both successful and denied authorization scenarios. Outbound traffic control Understand how stateful inspection permits response traffic. Avoid unnecessary inbound rules for dynamic client ports. Identify the dependencies required before introducing outbound default-deny. Restrict administrative tools, service accounts, high-risk applications, and servers to approved destinations. Account for dynamic cloud services, proxies, certificate endpoints, and content delivery networks. Introduce outbound restrictions gradually through discovery, narrow allow rules, pilots, and monitoring. Logging and evidence Enable logging for dropped packets and successful connections. Configure the log location and maximum size for every profile. Verify effective logging settings. Interpret firewall log fields such as action, protocol, address, port, interface, and direction. Use observed traffic to discover application dependencies. Distinguish observed traffic from authorized traffic. Use dropped-packet records to confirm that traffic reached the host firewall. Use successful-connection records to confirm firewall admission. Forward firewall evidence to protected central storage. Correlate firewall data with process, authentication, application, and network telemetry. Troubleshooting Follow a structured diagnostic sequence from the application listener through firewall and IPsec state. Inspect the merged runtime policy rather than only an individual policy source. Trace an effective rule back to Group Policy or another originating store. Identify conflicting or overriding block rules. Inspect active IPsec rules and main-mode and quick-mode security associations. Diagnose authentication, trust, time, name-resolution, selector, and cryptographic mismatches. Capture and interpret IPsec negotiation traffic on UDP ports 500 and 4500. Differentiate firewall admission failures from application or identity failures. Make controlled policy changes without disabling the firewall or creating unrestricted exceptions. Windows Firewall might already be a familiar part of your Windows environment. This module will help you appreciate just how much security and diagnostic functionality is built into it—and how to apply that functionality with greater precision. Start learning: Understand advanced Windows FirewallWindows Server 2025 Hyper-V Workgroup Cluster with Certificate-Based Authentication
In this guide, we will walk through creating a 2-node or 4-node Hyper-V failover cluster where the nodes are not domain-joined, using mutual certificate-based authentication instead of NTLM or shared local accounts. Here we are going to leverage X.509 certificates for node-to-node authentication. If you don't use certificates, you can do this with NTLM, but we're avoiding that as NTLM is supported, but the general recommendation is that you deprecate it where you can. We can't use Kerberos because our nodes won't be domain joined. It's a lot easier to do Windows Server Clusters if everything is domain joined, but that's not what we're doing here because there are scenarios where people want each cluster node to be a standalone (probably why you are reading this article). Prerequisites and Environment Preparation Before diving into configuration, ensure the following prerequisites and baseline setup: Server OS and Roles: All cluster nodes must be running Windows Server 2025 (same edition and patch level). Install the latest updates and drivers on each node. Each node should have the Hyper-V role and Failover Clustering feature available (we will install these via PowerShell shortly). Workgroup configuration: Nodes must be in a workgroup. The nodes should be in the same workgroup name. All nodes should share a common DNS suffix so that they can resolve each other’s FQDNs. For example, if your chosen suffix is mylocal.net, ensure each server’s FQDN is NodeName.mylocal.net. Name Resolution: Provide a way for nodes to resolve each other’s names (and the cluster name). If you have no internal DNS server, use the hosts file on each node to map hostnames to IPs. At minimum, add entries for each node’s name (short and FQDN) and the planned cluster name (e.g. Cluster1 and Cluster1.mylocal.net) pointing to the cluster’s management IP address. Network configuration: Ensure a reliable, low-latency network links all nodes. Ideally use at least two networks or VLANs: one for management/cluster communication and one dedicated for Live Migration traffic. This improves performance and security (live migration traffic can be isolated). If using a single network, ensure it is a trusted, private network since live migration data is not encrypted by default. Assign static IPs (or DHCP reservations) on the management network for each node and decide on an unused static IP for the cluster itself. Verify that necessary firewall rules for clustering are enabled on each node (Windows will add these when the Failover Clustering feature is installed, but if your network is classified Public, you may need to enable them or set the network location to Private). Time synchronization: Consistent time is important for certificate trust. Configure NTP on each server (e.g. pointing to a reliable internet time source or a local NTP server) so that system clocks are in sync. Shared storage: Prepare the shared storage that all nodes will use for Hyper-V. This can be an iSCSI target or an SMB 3.0 share accessible to all nodes. For iSCSI or SAN storage, connect each node to the iSCSI target (e.g. using the Microsoft iSCSI Initiator) and present the same LUN(s) to all nodes. Do not bring the disks online or format them on individual servers – leave them raw for the cluster to manage. For an SMB 3 file share, ensure the share is configured for continuous availability. Note: A file share witness for quorum is not supported in a workgroup cluster, so plan to use a disk witness or cloud witness instead. Administrative access: You will need Administrator access to each server. While we will avoid using identical local user accounts for cluster authentication, you should still have a way to log into each node (e.g. the built-in local Administrator account on each machine). If using Remote Desktop or PowerShell Remoting for setup, ensure you can authenticate to each server (we will configure certificate-based WinRM for secure remote PowerShell). The cluster creation process can be done by running commands locally on each node to avoid passing NTLM credentials. Obtaining and Configuring Certificates for Cluster Authentication The core of our setup is the use of mutual certificate-based authentication between cluster nodes. Each node will need an X.509 certificate that the others trust. We will outline how to use an internal Active Directory Certificate Services (AD CS) enterprise CA to issue these certificates, and mention alternatives for test environments. We are using AD CS even though the nodes aren't domain joined. Just because the nodes aren't members of the domain doesn't mean you can't use an Enterprise CA to issue certificates, you just have to ensure the nodes are configured to trust the CA's certs manually. Certificate Requirements and Template Configuration For clustering (and related features like Hyper-V live migration) to authenticate using certificates, the certificates must meet specific requirements: Key Usage: The certificate should support digital signature and key encipherment (these are typically enabled by default for SSL certificates). Enhanced Key Usage (EKU): It must include both Client Authentication and Server Authentication EKUs. Having both allows the certificate to be presented by a node as a client (when initiating a connection to another node) and as a server (when accepting a connection). For example, in the certificate’s properties you should see Client Authentication (1.3.6.1.5.5.7.3.2) and Server Authentication (1.3.6.1.5.5.7.3.1) listed under “Enhanced Key Usage”. Subject Name and SAN: The certificate’s subject or Subject Alternative Name should include the node’s DNS name. It is recommended that the Subject Common Name (CN) be set to the server’s fully qualified DNS name (e.g. Node1.mylocal.net). Also include the short hostname (e.g. Node1) in the Subject Alternative Name (SAN) extension (DNS entries). If you have already chosen a cluster name (e.g. Cluster1), include the cluster’s DNS name in the SAN as well. This ensures that any node’s certificate can be used to authenticate connections addressed to the cluster’s name or the node’s name. (Including the cluster name in all node certificates is optional but can facilitate management access via the cluster name over HTTPS, since whichever node responds will present a certificate that matches the cluster name in SAN.) Trust: All cluster nodes must trust the issuer of the certificates. If using an internal enterprise CA, this means each node should have the CA’s root certificate in its Trusted Root Certification Authorities store. If you are using a standalone or third-party CA, similarly ensure the root (and any intermediate CA) is imported into each node’s Trusted Root store. Next, on your enterprise CA, create a certificate template for the cluster node certificates (or use an appropriate existing template): Template basis: A good starting point is the built-in “Computer” or “Web Server” template. Duplicate the template so you can modify settings without affecting defaults. General Settings: Give the new template a descriptive name (e.g. “Workgroup Cluster Node”). Set the validity period (e.g. 1 or 2 years – plan a manageable renewal schedule since these certs will need renewal in the future). Compatibility: Ensure it’s set for at least Windows Server 2016 or higher for both Certification Authority and Certificate Recipient to support modern cryptography. Subject Name: Since our servers are not domain-joined (and thus cannot auto-enroll with their AD computer name), configure the template to allow subject name supply in the request. In the template’s Subject Name tab, choose “Supply in request” (this allows us to specify the SAN and CN when we request the cert on each node). Alternatively, use the SAN field in the request – modern certificate requests will typically put the FQDN in the SAN. Extensions: In the Extensions tab, edit Key Usage to ensure it includes Digital Signature and Key Encipherment (these should already be selected by default for Computer templates). Then edit Extended Key Usage and make sure Client Authentication and Server Authentication are present. If using a duplicated Web Server template, add Client Authentication EKU; if using Computer template, both EKUs should already be there. Also enable private key export if your policy requires (though generally private keys should not be exported; here each node will have its own cert so export is not necessary except for backup purposes). Security: Allow the account that will be requesting the certificate to enroll. Since the nodes are not in AD, you might generate the CSR on each node and then submit it via an admin account. One approach is to use a domain-joined management PC or the CA server itself to submit the CSR, so ensure domain users (or a specific user) have Enroll permission on the template. Publish the template: On the CA, publish the new template so it is available for issuing. Obtaining Certificates from the Enterprise CA Now for each cluster node, request a certificate from the CA using the new template. To do this, on each node, create an INF file describing the certificate request. For example, Node1.inf might specify the Subject as CN=Node1.mylocal.net and include SANs for Node1.mylocal.net, Node1, Cluster1.mylocal.net, Cluster1. Also specify in the INF that you want Client and Server Auth EKUs (or since the template has them by default, it might not be needed to list them explicitly). Then run: certreq -new Node1.inf Node1.req This generates a CSR file (Node1.req). Transfer this request to a machine where you can reach the CA (or use the CA web enrollment). Submit the request to your CA, specifying the custom template. For example: certreq -submit -attrib "CertificateTemplate:Workgroup Cluster Node" Node1.req Node1.cer (Or use the Certification Authority MMC to approve the pending request.) This yields Node1.cer. Finally, import the issued certificate on Node1: certreq -accept Node1.cer This will automatically place the certificate in the Local Machine Personal store with the private key. Using Certificates MMC (if the CA web portal is available): On each node, open Certificates (Local Computer) MMC and under Personal > Certificates, initiate New Certificate Request. Use the Active Directory Enrollment Policy if the node can reach the CA’s web enrollment (even if not domain-joined, you can often authenticate with a domain user account for enrollment). Select the custom template and supply the DNS names. Complete the enrollment to obtain the certificate in the Personal store. On a domain-joined helper system: Alternatively, use a domain-joined machine to request on behalf of the node (using the “Enroll on behalf” feature with an Enrollment Agent certificate, or simply request and then export/import). This is more complex and usually not needed unless policy restricts direct enrollment. After obtaining each certificate, verify on the node that it appears in Certificates (Local Computer) > Personal > Certificates. The Issued To should be the node’s FQDN, and on the Details tab you should see the required EKUs and SAN entries. Also import the CA’s Root CA certificate into Trusted Root Certification Authorities on each node (the certreq -accept step may do this automatically if the chain is provided; if not, manually import the CA root). A quick check using the Certificates MMC or PowerShell can confirm trust. For example, to check via PowerShell: Get-ChildItem Cert:\LocalMachine\My | Where-Object {$_.Subject -like "*Node1*"} | Select-Object Subject, EnhancedKeyUsageList, NotAfter Make sure the EnhancedKeyUsageList shows both Client and Server Authentication and that NotAfter (expiry) is a reasonable date. Also ensure no errors about untrusted issuer – the Certificate status should show “This certificate is OK”. Option: Self-Signed Certificates for Testing For a lab or proof-of-concept (where an enterprise CA is not available), you can use self-signed certificates. The key is to create a self-signed cert that includes the proper names and EKUs, and then trust that cert across all nodes. Use PowerShell New-SelfSignedCertificate with appropriate parameters. For example, on Node1: $cert = New-SelfSignedCertificate -DnsName "Node1.mylocal.net", "Node1", "Cluster1.mylocal.net", "Cluster1" ` -CertStoreLocation Cert:\LocalMachine\My ` -KeyUsage DigitalSignature, KeyEncipherment ` -TextExtension @("2.5.29.37={text}1.3.6.1.5.5.7.3.1;1.3.6.1.5.5.7.3.2") This creates a certificate for Node1 with the specified DNS names and both ServerAuth/ClientAuth EKUs. Repeat on Node2 (adjusting names accordingly). Alternatively, you can generate a temporary root CA certificate and then issue child certificates to each node (PowerShell’s -TestRoot switch simplifies this by generating a root and end-entity cert together). If you created individual self-signed certs per node, export each node’s certificate (without the private key) and import it into the Trusted People or Trusted Root store of the other nodes. (Trusted People works for peer trust of specific certs; Trusted Root works if you created a root CA and issued from it). For example, if Node1 and Node2 each have self-signed certs, import Node1’s cert as a Trusted Root on Node2 and vice versa. This is required because self-signed certs are not automatically trusted. Using CA-issued certs is strongly recommended for production. Self-signed certs should only be used in test environments, and if used, monitor and manually renew them before expiration (since there’s no CA to do it). A lot of problems have occurred in production systems because people used self signed certs and forgot that they expire. Setting Up WinRM over HTTPS for Remote Management With certificates in place, we can configure Windows Remote Management (WinRM) to use them. WinRM is the service behind PowerShell Remoting and many remote management tools. By default, WinRM uses HTTP (port 5985) and authenticates via Kerberos or NTLM. In a workgroup scenario, NTLM over HTTP would be used – we want to avoid that. Instead, we will enable WinRM over HTTPS (port 5986) with our certificates, providing encryption and the ability to use certificate-based authentication for management sessions. Perform these steps on each cluster node: Verify certificate for WinRM: WinRM requires a certificate in the Local Computer Personal store that has a Server Authentication EKU and whose Subject or SAN matches the hostname. We have already enrolled such a certificate for each node. Double-check that the certificate’s Issued To (CN or one of the SAN entries) exactly matches the hostname that clients will use (e.g. the FQDN). If you plan to manage via short name, ensure the short name is in SAN; if via FQDN, that’s covered by CN or SAN. The certificate must not be expired or revoked, and it should be issued by a CA that the clients trust (not self-signed unless the client trusts it). Enable the HTTPS listener: Open an elevated PowerShell on the node and run: winrm quickconfig -transport:https This command creates a WinRM listener on TCP 5986 bound to the certificate. If it says no certificate was found, you may need to specify the certificate manually. You can do so with: # Find the certificate thumbprint (assuming only one with Server Auth) $thumb = (Get-ChildItem Cert:\LocalMachine\My | Where-Object {$_.EnhancedKeyUsageList -match "Server Authentication"} | Select-Object -First 1 -ExpandProperty Thumbprint) New-Item -Path WSMan:\LocalHost\Listener -Transport HTTPS -Address * -CertificateThumbprint $thumb -Force Verify listeners with: winrm enumerate winrm/config/listener You should see an HTTPS listener with hostname, listening on 5986, and the certificate’s thumbprint. WinRM will automatically choose a certificate that meets the criteria (if multiple are present, it picks the one with CN matching machine name, so ideally use a unique cert to avoid ambiguity). Disable unencrypted/HTTP access (optional but recommended): Since we want all remote management encrypted and to eliminate NTLM, you can disable the HTTP listener. Run: Remove-WSManInstance -ResourceURI winrm/config/Listener -SelectorSet @{Address="*", Transport="HTTP"} This ensures WinRM is only listening on HTTPS. Also, you may configure the WinRM service to reject unencrypted traffic and disallow Basic authentication to prevent any fallback to insecure methods: winrm set winrm/config/service '@{AllowUnencrypted="false"}' winrm set winrm/config/service/auth '@{Basic="false"}' (By default, AllowUnencrypted is false anyway when HTTPS is used, and Basic is false unless explicitly enabled.) TrustedHosts (if needed): In a workgroup, WinRM won’t automatically trust hostnames for authentication. However, when using certificate authentication, the usual TrustedHosts requirement may not apply in the same way as for NTLM/Negotiate. If you plan to authenticate with username/password over HTTPS (e.g. using Basic or default CredSSP), you will need to add the other nodes (or management station) to the TrustedHosts list on each node. This isn’t needed for the cluster’s internal communication (which uses certificates via clustering, not WinRM), but it might be needed for your remote PowerShell sessions depending on method. To allow all (not recommended for security), you could do: Set-Item WSMan:\localhost\Client\TrustedHosts -Value "*" Or specify each host: Set-Item WSMan:\localhost\Client\TrustedHosts -Value "Node1,Node2,Cluster1" This setting allows the local WinRM client to talk to those remote names without Kerberos. If you will use certificate-based authentication for WinRM (where the client presents a cert instead of username/password), TrustedHosts is not required – certificate auth doesn’t rely on host trust in the same way. (Optional) Configure certificate authentication for admin access: One of the benefits of HTTPS listener is you can use certificate mapping to log in without a password. For advanced users, you can issue a client certificate for yourself (with Client Authentication EKU), then configure each server to map that cert to a user (for example, map to the local Administrator account). This involves creating a mapping entry in winrm/config/service/certmapping. For instance: # Example: map a client cert by its subject to a local account winrm create winrm/config/service/certmapping @{CertificateIssuer= "CN=YourCA"; Subject="CN=AdminUserCert"; Username="Administrator"; Password="<adminPassword>"; Enabled="true"} Then from your management machine, you can use that certificate to authenticate. While powerful, this goes beyond the core cluster setup, so we won’t detail it further. Without this, you can still connect to the nodes using Enter-PSSession -ComputerName Node1 -UseSSL -Credential Node1\Administrator (which will prompt for the password but send it safely over the encrypted channel). At this point, we have each node prepared with a trusted certificate and WinRM listening securely. Test the connectivity: from one node, try to start a PowerShell remote session to the other using HTTPS. For example, on Node1 run: Test-WsMan Node2 -UseSSL Enter-PSSession -ComputerName Node2 -UseSSL -Credential Node2\Administrator You should connect without credential errors or warnings (you may get a certificate trust prompt if the client machine doesn’t trust the server cert — make sure the CA root is in the client’s trust store as well). Once you can manage nodes remotely over HTTPS, you’re ready to create the cluster. Installing the Hyper-V and Failover Clustering Roles All cluster nodes need the Hyper-V role (for running VMs) and the Failover Clustering feature. We will use PowerShell to install these simultaneously on each server. On each node: Open an elevated PowerShell (locally or via your new WinRM setup) and run: Install-WindowsFeature -Name Failover-Clustering, Hyper-V -IncludeManagementTools -Restart This installs the Hyper-V hypervisor, the clustering feature, and management tools (including the Failover Cluster Manager and Hyper-V Manager GUI, and PowerShell modules). The server will restart if Hyper-V was not previously enabled (we include -Restart for convenience). After reboot, run the command on the next node (if doing it remotely, do one at a time). Alternatively, use the Server Manager GUI or Install-WindowsFeature without -Restart and reboot manually. After all nodes are back up, verify the features: Get-WindowsFeature -Name Hyper-V, Failover-Clustering It should show both as Installed. Also confirm the Failover Clustering PowerShell module is available (Get-Module -ListAvailable FailoverClusters) and the Cluster service is installed (though not yet configured). Cluster service account: Windows Server 2016+ automatically creates a local account called CLIUSR used by the cluster service for internal communication. Ensure this account was created (Computer Management > Users). We won’t interact with it directly, but be aware it exists. Do not delete or disable CLIUSR – the cluster uses it alongside certificates for bootstrapping. (All cluster node communications will now use either Kerberos or certificate auth; NTLM is not needed in WS2019+ clusters.) Now that you've backflipped and shenaniganed with all the certificates, you can actually get around to building the cluster. Creating the Failover Cluster (Using DNS as the Access Point) Here we will create the cluster and add nodes to it using PowerShell. The cluster will use a DNS name for its administrative access point (since there is no Active Directory for a traditional cluster computer object). The basic steps are: Validate the configuration (optional but recommended). Create the cluster (initially with one node to avoid cross-node authentication issues). Join additional node(s) to the cluster. Configure cluster networking, quorum, and storage (CSV). Validate the Configuration (Cluster Validation) It’s good practice to run the cluster validation tests to catch any misconfiguration or hardware issues before creating the cluster. Microsoft supports a cluster only if it passes validation or if any errors are acknowledged as non-critical. Run the following from one of the nodes (this will reach out to all nodes): Test-Cluster -Node Node1.mylocal.net, Node2.mylocal.net Replace with your actual node names (include all 2 or 4 nodes). The cmdlet will run a series of tests (network, storage, system settings). Ensure that all tests either pass or only have warnings that you understand. For example, warnings about “no storage is shared among all nodes” are expected if you haven’t yet configured iSCSI or if using SMB (you can skip storage tests with -Skip Storage if needed). If critical tests fail, resolve those issues (networking, disk visibility, etc.) before proceeding. Create the Cluster (with the First Node) On one node (say Node1), use the New-Cluster cmdlet to create the cluster with that node as the first member. By doing it with a single node initially, we avoid remote authentication at cluster creation time (no need for Node1 to authenticate to Node2 yet): New-Cluster -Name "Cluster1" -Node Node1 -StaticAddress "10.0.0.100" -AdministrativeAccessPoint DNS Here: -Name is the intended cluster name (this will be the name clients use to connect to the cluster, e.g. for management or as a CSV namespace prefix). We use “Cluster1” as an example. -Node Node1 specifies which server to include initially (Node1’s name). -StaticAddress sets the cluster’s IP address (choose one in the same subnet that is not in use; this IP will be brought online as the “Cluster Name” resource). In this example 10.0.0.100 is the cluster IP. -AdministrativeAccessPoint DNS indicates we’re creating a DNS-only cluster (no AD computer object). This is the default in workgroup clusters, but we specify it explicitly for clarity. The command will proceed to create the cluster service, register the cluster name in DNS (if DNS is configured and dynamic updates allowed), and bring the core cluster resources online. It will also create a cluster-specific certificate (self-signed) for internal use if needed, but since we have our CA-issued certs in place, the cluster may use those for node authentication. Note: If New-Cluster fails to register the cluster name in DNS (common in workgroup setups), you might need to create a manual DNS A record for “Cluster1” pointing to 10.0.0.100 in whatever DNS server the nodes use. Alternatively, add “Cluster1” to each node’s hosts file (as we did in prerequisites). This ensures that the cluster name is resolvable. The cluster will function without AD, but it still relies on DNS for name resolution of the cluster name and node names. At this point, the cluster exists with one node (Node1). You can verify by running cluster cmdlets on Node1, for example: Get-Cluster (should list “Cluster1”) and Get-ClusterNode (should list Node1 as up). In Failover Cluster Manager, you could also connect to “Cluster1” (or to Node1) and see the cluster. Add Additional Nodes to the Cluster Now we will add the remaining node(s) to the cluster: On each additional node, run the following (replace “Node2” with the name of that node and adjust cluster name accordingly): Add-ClusterNode -Cluster Cluster1 -Name Node2 Run this on Node2 itself (locally). This instructs Node2 to join the cluster named Cluster1. Because Node2 can authenticate the cluster (Node1) via the cluster’s certificate and vice versa, the join should succeed without prompting for credentials. Under the hood, the cluster service on Node2 will use the certificate (and CLIUSR account) to establish trust with Node1’s cluster service. Repeat the Add-ClusterNode command on each additional node (Node3, Node4, etc. one at a time). After each join, verify by running Get-ClusterNode on any cluster member – the new node should show up and status “Up”. If for some reason you prefer a single command from Node1 to add others, you could use: # Run on Node1: Add-ClusterNode -Name Node2, Node3 -Cluster Cluster1 This would attempt to add Node2 and Node3 from Node1. It may prompt for credentials or require TrustedHosts if no common auth is present. Using the local Add-ClusterNode on each node avoids those issues by performing the action locally. Either way, at the end all nodes should be members of Cluster1. Configure Quorum (Witness) Quorum configuration is critical, especially with an even number of nodes. The cluster will already default to Node Majority (no witness) or may try to assign a witness if it finds eligible storage. Use a witness to avoid a split-brain scenario. If you have a small shared disk (LUN) visible to both nodes, that can be a Disk Witness. Alternatively, use a Cloud Witness (Azure). To configure a disk witness, first make sure the disk is seen as Available Storage in the cluster, then run: Get-ClusterAvailableDisk | Add-ClusterDisk Set-ClusterQuorum -Cluster Cluster1 -NodeAndDiskMajority 0 /disk:<DiskResourceName> (Replace <DiskResourceName> with the name or number of the disk from Get-ClusterResource). Using Failover Cluster Manager, you can run the Configure Cluster Quorum wizard and select “Add a disk witness”. If no shared disk is available, the Cloud Witness is an easy option (requires an Azure Storage account and key). For cloud witness: Set-ClusterQuorum -Cluster Cluster1 -CloudWitness -AccountName "<StorageAccount>" -AccessKey "<Key>" Do not use a File Share witness – as noted earlier, file share witnesses are not supported in workgroup clusters because the cluster cannot authenticate to a remote share without AD. A 4-node cluster can sustain two node failures if properly configured. It’s recommended to also configure a witness for even-number clusters to avoid a tie (2–2) during a dual-node failure scenario. A disk or cloud witness is recommended (same process as above). With 4 nodes, you would typically use Node Majority + Witness. The cluster quorum wizard can automatically choose the best quorum config (typically it will pick Node Majority + Witness if you run the wizard and have a witness available). You can verify the quorum configuration with Get-ClusterQuorum. Make sure it lists the witness you configured (if any) and that the cluster core resources show the witness online. Add Cluster Shared Volumes (CSV) or Configure VM Storage Next, prepare storage for Hyper-V VMs. If using a shared disk (Block storage like iSCSI/SAN), after adding the disks to the cluster (they should appear in Storage > Disks in Failover Cluster Manager), you can enable Cluster Shared Volumes (CSV). CSV allows all nodes to concurrently access the NTFS/ReFS volume, simplifying VM placement and live migration. To add available cluster disks as CSV volumes: Get-ClusterDisk | Where-Object IsClustered -eq $true | Add-ClusterSharedVolume This will take each clustered disk and mount it as a CSV under C:\ClusterStorage\ on all nodes. Alternatively, right-click the disk in Failover Cluster Manager and choose Add to Cluster Shared Volumes. Once done, format the volume (if not already formatted) with NTFS or ReFS via any node (it will be accessible as C:\ClusterStorage\Volume1\ etc. on all nodes). Now this shared volume can store all VM files, and any node can run any VM using that storage. If using an SMB 3 share (NAS or file server), you won’t add this to cluster storage; instead, each Hyper-V host will connect to the SMB share directly. Ensure each node has access credentials for the share. In a workgroup, that typically means the NAS is also in a workgroup and you’ve created a local user on the NAS that each node uses (via stored credentials) – this is outside the cluster’s control. Each node should be able to New-SmbMapping or simply access the UNC path. Test access from each node (e.g. Dir \\NAS\HyperVShare). In Hyper-V settings, you might set the Default Virtual Hard Disk Path to the UNC or just specify the UNC when creating VMs. Note: Hyper-V supports storing VMs on SMB 3.0 shares with Kerberos or certificate-based authentication, but in a workgroup you’ll likely rely on a username/password for the share (which is a form of local account usage at the NAS). This doesn’t affect cluster node-to-node auth, but it’s a consideration for securing the NAS. Verify Cluster Status At this stage, run some quick checks to ensure the cluster is healthy: Get-Cluster – should show the cluster name, IP, and core resources online. Get-ClusterNode – all nodes should be Up. Get-ClusterResource – should list resources (Cluster Name, IP Address, any witness, any disks) and their state (Online). The Cluster Name resource will be of type “Distributed Network Name” since this is a DNS-only cluster. Use Failover Cluster Manager (you can launch it on one of the nodes or from RSAT on a client) to connect to “Cluster1”. Ensure you can see all nodes and storage. When prompted to connect, use <clustername> or <clusterIP> – with our certificate setup, it may be best to connect by cluster name (make sure DNS/hosts is resolving it to the cluster IP). If a certificate trust warning appears, it might be because the management station doesn’t trust the cluster node’s cert or you connected with a name not in the SAN. As a workaround, connect directly to a node in cluster manager (e.g. Node1), which then enumerates the cluster. Now you have a functioning cluster ready for Hyper-V workloads, with secure authentication between nodes. Next, we configure Hyper-V specific settings like Live Migration. Configuring Hyper-V for Live Migration in the Workgroup Cluster One major benefit introduced in Windows Server 2025 is support for Live Migration in workgroup clusters (previously, live migration required Kerberos and thus a domain). In WS2025, cluster nodes use certificates to mutually authenticate for live migration traffic. This allows VMs to move between hosts with no downtime even in the absence of AD. We will enable and tune live migration for our cluster. By default, the Hyper-V role might have live migration disabled (for non-clustered hosts). In a cluster, it may be auto-enabled when the Failover Clustering and Hyper-V roles are both present, but to ensure it it, run: Enable-VMMigration This enables the host to send/receive live migrations. In PowerShell, no output means success. (In Hyper-V Manager UI, this corresponds to ticking “Enable incoming and outgoing live migrations” in the Live Migrations settings.) In a workgroup, the only choice in UI would be CredSSP (since Kerberos requires domain). CredSSP means you must initiate the migration from a session where you are logged onto the source host so your credentials can be delegated. We cannot use Kerberos here, but the cluster’s internal PKU2U certificate mechanism will handle node-to-node auth for us when orchestrated via Failover Cluster Manager. No explicit setting is needed for cluster-internal certificate usage & Windows will use it automatically for the actual live migration operation. If you were to use PowerShell, the default MigrationAuthenticationType is CredSSP for workgroup. You can confirm (or set explicitly, though not strictly required): Set-VMHost -VirtualMachineMigrationAuthenticationType CredSSP (This can be done on each node; it just ensures the Hyper-V service knows to use CredSSP which aligns with our need to initiate migrations from an authenticated context.) If your cluster nodes were domain-joined, Windows Server 2025 enables Credential Guard which blocks CredSSP by default. In our case (workgroup), Credential Guard is not enabled by default, so CredSSP will function. Just be aware if you ever join these servers to a domain (or they were once joined to a domain before being demoted to a workgroup), you’d need to configure Kerberos constrained delegation or disable Credential Guard to use live migration. For security and performance, do not use the management network for VM migration if you have other NICs. We will designate the dedicated network (e.g. “LMNet” or a specific subnet) for migrations. You can configure this via PowerShell or Failover Cluster Manager. Using PowerShell, run the following on each node: # Example: allow LM only on 10.0.1.0/24 network (where 10.0.1.5 is this node's IP on that network) Set-VMMigrationNetwork 10.0.1.5 Set-VMHost -UseAnyNetworkForMigration $false The Set-VMMigrationNetwork cmdlet adds the network associated with the given IP to the allowed list for migrations. The second cmdlet ensures only those designated networks are used. Alternatively, if you have the network name or interface name, you might use Hyper-V Manager UI: under each host’s Hyper-V Settings > Live Migrations > Advanced Features, select Use these IP addresses for Live Migration and add the IP of the LM network interface. In a cluster, these settings are typically per-host. It’s a good idea to configure it identically on all nodes. Verify the network selection by running: Get-VMHost | Select -ExpandProperty MigrationNetworks. It should list the subnet or network you allowed, and UseAnyNetworkForMigration should be False. Windows can either send VM memory over TCP, compress it, or use SMB Direct (if RDMA is available) for live migration. By default in newer Windows versions, compression is used as it offers a balance of speed without special hardware. If you have a very fast dedicated network (10 Gbps+ or RDMA), you might choose SMB to leverage SMB Multichannel/RDMA for highest throughput. To set this: # Options: TCPIP, Compression, SMB Set-VMHost -VirtualMachineMigrationPerformanceOption Compression (Do this on each node; “Compression” is usually default on 2022/2025 Hyper-V.) If you select SMB, ensure your cluster network is configured to allow SMB traffic and consider enabling SMB encryption if security is a concern (SMB encryption will encrypt the live migration data stream). Note that if you enable SMB encryption or cluster-level encryption, it could disable RDMA on that traffic, so only enable it if needed, or rely on the network isolation as primary protection. Depending on your hardware, you may allow multiple VMs to migrate at once. The default is usually 2 simultaneous live migrations. You can increase this if you have capacity: Set-VMHost -MaximumVirtualMachineMigrations 4 -MaximumStorageMigrations 2 Adjust numbers as appropriate (and consider that cluster-level property (Get-Cluster).MaximumParallelMigrations might override host setting in a cluster). This setting can also be found in Hyper-V Settings UI under Live Migrations. With these configured, live migration is enabled. Test a live migration: Create a test VM (or if you have VMs, pick one) and attempt to move it from one node to another using Failover Cluster Manager or PowerShell: In Failover Cluster Manager, under Roles, right-click a virtual machine, choose Live Migrate > Select Node… and pick another node. The VM should migrate with zero downtime. If it fails, check for error messages regarding authentication. Ensure you initiated the move from a node where you’re an admin (or via cluster manager connected to the cluster with appropriate credentials). The cluster will handle the mutual auth using the certificates (this is transparent – behind the scenes, the nodes use the self-created PKU2U cert or our installed certs to establish a secure connection for VM memory transfer). Alternatively, use PowerShell: Move-ClusterVirtualMachineRole -Name "<VM resource name>" -Node <TargetNode> This cmdlet triggers a cluster-coordinated live migration (the cluster’s Move operation will use the appropriate auth). If the migration succeeds, congratulations – you have a fully functional Hyper-V cluster without AD! Security Best Practices Recap and Additional Hardening Additional best practices for securing a workgroup Hyper-V cluster include: Certificate Security: The private keys of your node certificates are powerful – protect them. They are stored in the machine store (and likely marked non-exportable). Only admins can access them; ensure no unauthorized users are in the local Administrators group. Plan a process for certificate renewal before expiration. If using an enterprise CA, you might issue certificates with a template that allows auto-renewal via scripts or at least track their expiry to re-issue and install new certs on each node in time. The Failover Cluster service auto-generates its own certificates (for CLIUSR/PKU2U) and auto-renews them, but since we provided our own, we must manage those. Stagger renewals to avoid all nodes swapping at once (the cluster should still trust old vs new if the CA is the same). It may be wise to overlap: install new certs on all nodes and only then remove the old, so that at no point a node is presenting a cert the others don't accept (if you change CA or template). Trusted Root and Revocation: All nodes trust the CA – maintain the security of that CA. Do not include unnecessary trust (e.g., avoid having nodes trust public CAs that they don’t need). If possible, use an internal CA that is only used for these infrastructure certs. Keep CRLs (Certificate Revocation Lists) accessible if your cluster nodes need to check revocation for each other’s certs (though cluster auth might not strictly require online revocation checking if the certificates are directly trusted). It’s another reason to have a reasonably long-lived internal CA or offline root. Disable NTLM: Since clustering no longer needs NTLM as of Windows 2019+, you can consider disabling NTLM fallback on these servers entirely for added security (via Group Policy “Network Security: Restrict NTLM: Deny on this server” etc.). However, be cautious: some processes (including cluster formation in older versions, or other services) might break. In our configuration, cluster communications should use Kerberos or cert. If these servers have no need for NTLM (no legacy apps), disabling it eliminates a whole class of attacks. Monitor event logs (Security log events for NTLM usage) if you attempt this. The conversation in the Microsoft tech community indicates by WS2022, cluster should function with NTLM disabled, though a user observed issues when CLIUSR password rotated if NTLM was blocked. WS2025 should further reduce any NTLM dependency. PKU2U policy: The cluster uses the PKU2U security provider for peer authentication with certificates. There is a local security policy “Network security: Allow PKU2U authentication requests to this computer to use online identities” – this must be enabled (which it is by default) for clustering to function properly. Some security guides recommend disabling PKU2U; do not disable it on cluster nodes (or if your organization’s baseline GPO disables it, create an exception for these servers). Disabling PKU2U will break the certificate-based node authentication and cause cluster communication failures. Firewall: We opened WinRM over 5986. Ensure Windows Firewall has the Windows Remote Management (HTTPS-In) rule enabled. The Failover Clustering feature should have added rules for cluster heartbeats (UDP 3343, etc.) and SMB (445) if needed. Double-check that on each node the Failover Cluster group of firewall rules is enabled for the relevant profiles (if your network is Public, you might need to enable the rules for Public profile manually, or set network as Private). Also, for live migration, if using SMB transport, enable SMB-in rules. If you enabled SMB encryption, it uses the same port 445 but encrypts payloads. Secure Live Migration Network: Ideally, the network carrying live migration is isolated (not routed outside of the cluster environment). If you want belt-and-suspenders security, you could implement IPsec encryption on live migration traffic. For example, require IPsec (with certificates) between the cluster nodes on the LM subnet. However, this can be complex and might conflict with SMB Direct/RDMA. Another simpler approach: since we can rely on our certificate mutual auth to prevent unauthorized node communication, focus on isolating that traffic so even if someone tapped it, you can optionally turn on SMB encryption for LM (when using SMB transport) which will encrypt the VM memory stream. At minimum, treat the LM network as sensitive, as it carries VM memory contents in clear text if not otherwise encrypted. Secure WinRM/management access: We configured WinRM for HTTPS. Make sure to limit who can log in via WinRM. By default, members of the Administrators group have access. Do not add unnecessary users to Administrators. You can also use Local Group Policy to restrict WinRM service to only allow certain users or certificate mappings. Since this is a workgroup, there’s no central AD group; you might create a local group for “Remote Management Users” and configure WSMan to allow members of that group (and only put specific admin accounts in it). Also consider enabling PowerShell Just Enough Administration (JEA) if you want to delegate specific tasks without full admin rights, though that’s advanced. Hyper-V host security: Apply standard Hyper-V best practices: enable Secure Boot for Gen2 VMs, keep the host OS minimal (consider using Windows Server Core for fewer attack surface, if feasible), and ensure only trusted administrators can create or manage VMs. Since this cluster is not in a domain, you won’t have AD group-based access control; consider using Authentication Policies like LAPS for unique local admin passwords per node. Monitor cluster events: Monitor the System event log for any cluster-related errors (clustering will log events if authentication fails or if there are connectivity issues). Also monitor the FailoverClustering event log channel. Any errors about “unable to authenticate” or “No logon servers” etc., would indicate certificate or connectivity problems. Test failover and failback: After configuration, test that VMs can failover properly. Shut down one node and ensure VMs move to other node automatically. When the node comes back, you can live migrate them back. This will give confidence that the cluster’s certificate-based auth holds up under real failover conditions. Consider Management Tools: Tools like Windows Admin Center (WAC) can manage Hyper-V clusters. WAC can be configured to use the certificate for connecting to the nodes (it will prompt to trust the certificate if self-signed). Using WAC or Failover Cluster Manager with our setup might require launching the console from a machine that trusts the cluster’s cert and using the cluster DNS name. Always ensure management traffic is also encrypted (WAC uses HTTPS and our WinRM is HTTPS so it is).Some tools and techniques for hardening Windows Server
In this post I go over some tools and techniques exist for hardening Windows Server. As always, apply controls according to the server's role, test them against representative workloads before production rollout, document approved exceptions, and maintain tested console and recovery access in case a security control affects management or application compatibility. Apply the role-specific Windows Server 2025 baseline with OSConfig Security baselines turn hundreds of individual security decisions into a consistent, role-aware desired state. Using OSConfig reduces exposure caused by insecure defaults, legacy protocols, inconsistent administrator choices, and configuration drift that attackers can exploit for credential theft, lateral movement, or persistence. You can use use OSConfig at build time to apply the Microsoft security baseline that matches the server role: SecurityBaseline/WindowsServer/2025/MemberServer , SecurityBaseline/WindowsServer/2025/DomainController , or SecurityBaseline/WindowsServer/2025/WorkgroupMember . The baseline contains more than 300 settings covering network exposure, credentials, lateral movement, persistence resistance, and auditing. It can be managed through PowerShell, Windows Admin Center, or Azure Policy for Azure Arc-enabled servers. You can keep OSConfig drift control enabled so unauthorized or accidental changes are detected and corrected. Pilot the baseline with each workload, record required exceptions, and manage those exceptions centrally rather than weakening the baseline broadly. Implementation steps: Identify the server's role and the management authority that will own its settings, install the OSConfig PowerShell module, review the matching scenario, and apply it first to a representative test server. Validate application and management access, deploy the scenario in controlled rings, schedule and complete the restart required after applying the baseline, verify the desired configuration and compliance results, enable drift control, and record any approved exceptions and recovery procedures. Possible drawbacks: A baseline can disrupt legacy applications, authentication methods, network flows, or management tools that depend on weaker settings. Drift control can also reverse intentional emergency changes if they aren't recorded through the correct authority, so staged testing, documented exceptions, and tested recovery access are essential. Documentation on Learn: OSConfig security configuration for Windows Server | Deploy Windows Server 2025 security baselines with OSConfig | OSConfig security settings repository Use Secured-core hardware and enable platform security Windows Server secured-core combines hardware, firmware, virtualization, and operating-system protections to establish trust before Windows starts and preserve that trust while it runs. These controls mitigate bootkits, malicious or vulnerable kernel drivers, direct memory access attacks, firmware tampering, and attempts to extract credentials from the operating system. You deploy on hardware or virtual machines that support TPM 2.0, UEFI Secure Boot, virtualization-based security, DMA protection, and the other Secured-core requirements. Enable the OSConfig SecuredCore scenario and verify that Credential Guard, hypervisor-protected code integrity, kernel protections, and the signed boot chain are active. You need to keep system firmware, TPM firmware, hypervisor components, and hardware drivers current. Test older drivers before enabling enforcement because incompatible kernel drivers can prevent security features from activating or can affect boot reliability. Implementation steps: Confirm that the physical server or virtual-machine platform meets the Secured-core requirements, update firmware and drivers, and enable TPM 2.0, Secure Boot, virtualization extensions, and DMA or IOMMU protection in the platform configuration. Apply the OSConfig SecuredCore scenario or configure the features through Windows Admin Center, restart as required, verify that each protection is active, record evidence of the hardware capabilities and running Windows protections rather than only the assigned policy, and monitor for driver or workload compatibility issues. Possible drawbacks: Secured-core features require compatible hardware, firmware, hypervisors, and signed drivers, which can increase procurement costs or rule out older systems. Virtualization-based protections can introduce a workload-dependent performance impact, and incompatible drivers or firmware can cause application failures, feature activation problems, or difficult boot recovery. Documentation: What is Secured-core server? | Configure Secured-core server | Windows Server 2025 secured-core hardware requirements Deploy Server Core and minimize installed components Attack-surface reduction removes code, services, interfaces, and utilities that an attacker could exploit or misuse after gaining access. A minimal Server Core deployment lowers the number of vulnerabilities that require patching and reduces opportunities for interactive attacks, malicious browsing, persistence, and abuse of unnecessary administrative tools. Install Server Core unless a supported workload specifically requires Desktop Experience. Server Core has a smaller local interface and component footprint, reducing exposed code, maintenance requirements, and opportunities for interactive misuse. Windows Server 2025 can't convert between Server Core and Server with Desktop Experience after installation, so changing this choice later requires a clean installation. Install only the roles, features, management agents, and application components required for the server's purpose. Remove obsolete utilities and unused software, avoid browsing the web from servers, and disable unnecessary services only after confirming role and application dependencies. Where practical, dedicate each server to a single security or workload role. Implementation steps: Confirm that the workload and vendor support Server Core, formally record the installation-option decision before deployment, select Server Core during installation, and define the minimum roles, features, agents, and software required for the server's purpose. Install only those components, configure remote management and recovery access, remove or disable unused components after dependency testing, and verify that the application, monitoring, backup, patching, and support processes still function. Possible drawbacks: Server Core can make local troubleshooting less familiar and increases reliance on remote management, automation, and command-line skills. Some vendor applications, support tools, or administrators require Desktop Experience, and removing roles or disabling services without dependency testing can break workloads, monitoring, backup, or recovery operations. Documentation: What is the Server Core installation option? | Server Core and Desktop Experience installation options Ensure rapid patching and continuous vulnerability management Patching and vulnerability management identify and close known weaknesses before attackers can reliably exploit them. This practice reduces exposure to remote-code execution, privilege escalation, ransomware, vulnerable drivers, compromised third-party components, and attacks that target publicly documented vulnerabilities soon after disclosure. Make sure you are aware if any of your server workloads have not got the latest security updates deployed Maintain an inventory of operating-system, application, driver, firmware, and management-agent versions. Use deployment rings to test updates quickly, meet defined remediation deadlines, install out-of-band security updates when required, and monitor update compliance and pending restarts. Azure Update Manager can provide centralized assessment and orchestration for Azure and Azure Arc-enabled servers. Use Microsoft Defender Vulnerability Management or an equivalent platform to discover exposures, prioritize remediation by exploitability and business impact, and verify that fixes actually remove the vulnerability. Patching Windows while leaving internet-facing applications, drivers, or firmware obsolete does not adequately harden the server. Implementation steps: Inventory servers and every supported update source, define remediation deadlines and deployment rings, and configure Azure Update Manager or another actively developed orchestration platform. Windows Server Update Services remains supported and available but is deprecated and should be treated as a legacy option rather than the preferred platform for a new long-term design. Run vulnerability assessments, prioritize exposed and actively exploited weaknesses, test updates, deploy them with coordinated reboots, verify compliance after installation, and maintain rollback and exception procedures. Possible drawbacks: Updates can require reboots, consume maintenance windows, or introduce application, driver, and performance regressions. Vulnerability scanners and management agents also consume resources and can generate false positives, so organizations need test rings, rollback procedures, maintenance coordination, and a risk-based process for temporary deferrals. Documentation: Azure Update Manager overview | Cloud-native patch management for Azure Arc-enabled servers | Microsoft Defender Vulnerability Management | Deprecated Windows Server features Implement Microsoft Defender Antivirus and endpoint detection and response Antivirus and endpoint detection and response combine prevention with behavioral monitoring and investigation. They mitigate malicious files, ransomware, web and network-delivered payloads, suspicious process activity, persistence mechanisms, credential theft, and attacks that evade simple signature-based detection. Run Microsoft Defender Antivirus in active mode unless a documented and tested security architecture requires another antimalware product. Enable real-time protection, behavior monitoring, cloud-delivered protection, automatic sample submission, and frequent security-intelligence updates. Use the OSConfig Defender/Antivirus/WindowsServer/2025 scenario as the recommended Server 2025 configuration starting point. Onboard servers to Microsoft Defender for Endpoint or Microsoft Defender for Servers for endpoint detection and response, investigation, and centralized visibility. Enable tamper protection and keep exclusions narrow, workload-specific, and regularly reviewed; broad path, process, or extension exclusions create useful hiding places for attackers. Implementation steps: Confirm licensing, connectivity, proxy, and third-party antivirus requirements, then apply the OSConfig Defender Antivirus scenario or an equivalent centrally managed policy. Enable real-time, behavior, cloud-delivered, sample-submission, and tamper protections; onboard the server to Defender for Endpoint or Defender for Servers; review Microsoft's built-in, automatic server-role, and workload-specific exclusions before adding any manual exclusion; verify sensor health, signature currency, alert delivery, and investigation access; and periodically confirm that every manual exclusion remains necessary. Possible drawbacks: Real-time scanning and endpoint telemetry can add CPU, memory, disk I/O, network, and licensing costs, particularly on high-throughput workloads. False positives or quarantine actions can interrupt services, while cloud-delivered capabilities can raise connectivity, privacy, or data-residency considerations; performance exclusions must therefore be tested and kept narrowly scoped. Documentation: Microsoft Defender Antivirus in Windows | Microsoft Defender for Endpoint on Windows | Defender Antivirus exclusions | Protect against security-setting tampering Deploy attack surface reduction and network protection on your Windows Server workloads Attack surface reduction rules prevent high-risk behaviors rather than waiting for a specific malicious file to be identified, while network protection blocks access to known or suspicious destinations. Together they mitigate ransomware, credential theft, malicious scripts, abuse of trusted tools, vulnerable drivers, command-and-control traffic, and payload delivery. Configure Microsoft Defender attack surface reduction rules to block common behaviors used by ransomware, credential theft, malicious scripts, vulnerable signed drivers, and executable content. Begin with audit or warning mode, review telemetry for legitimate workload dependencies, create narrowly scoped exclusions, and then move suitable rules to block mode on a defined schedule. Enable network protection where supported to prevent processes from reaching malicious or untrusted destinations. Manage these controls centrally through Group Policy, Microsoft Defender for Endpoint security settings management, or another supported policy platform. Implementation steps: Inventory server workloads and make a rule-by-rule applicability decision for each server role rather than reusing a generic workstation ASR profile unchanged. Create a centrally managed ASR and network-protection policy that initially uses audit or warning mode, collect and review events, confirm business-critical dependencies, create narrowly scoped exclusions, move applicable rules to block mode through deployment rings, verify that protected applications remain functional, and continuously review detections and exception use. Possible drawbacks: ASR rules can block legitimate automation, administrative tools, installers, scripts, or line-of-business applications that exhibit high-risk behavior. Audit mode can produce substantial telemetry, and broad exclusions can undermine the protection, so successful deployment requires workload testing, event review, careful exception design, and ongoing tuning. Documentation: Attack surface reduction capabilities | Evaluate Microsoft Defender Antivirus and ASR rules using Group Policy Allow only trusted code with App Control for Business Application control changes the execution model from allowing everything except known malware to allowing only code that satisfies an approved policy. This technique mitigates unknown malware, ransomware, unauthorized administrative utilities, malicious scripts, unapproved drivers, and opportunistic payloads that antivirus has not yet classified. Use App Control for Business to define which executables, scripts, installers, libraries, and drivers may run. Windows Server 2025 includes OSConfig scenarios for Microsoft's default policy and application blocklist. Start in audit mode, collect Code Integrity event ID 3076, create required supplemental allow policies, and move to enforcement only after representative workload testing. There are some good GUI tools written by MVPs published on GitHub that make this very easy. https://github.com/HotCakeX/Harden-Windows-Security Monitor blocked-code event ID 3077 after enforcement and maintain a controlled process for policy updates and emergency recovery. Application allowlisting is substantially stronger than relying only on malware signatures because unapproved code is prevented from running even when it has not yet been classified as malicious. Implementation steps: Verify that the device is running a production-signed Windows Server 2025 build because the OSConfig default policy doesn't permit flight-signed binaries. Inventory approved applications, scripts, drivers, publishers, and update mechanisms, then deploy the default policy and application blocklist through OSConfig in audit mode. Collect event ID 3076, build and deploy required supplemental policies, and sign policies only when the additional tamper resistance is required and certificate lifecycle, policy servicing, removal, and offline recovery have been tested. Test application updates and recovery, move the policy to enforcement in stages, and monitor event ID 3077 and policy health after deployment. Possible drawbacks: Poorly designed policies can block legitimate applications, updates, scripts, drivers, or boot-critical components and can cause a severe service outage. Maintaining allow policies creates operational overhead, especially for frequently changing software, and approved tools can still be abused, so audit-mode deployment, controlled updates, and offline recovery procedures are necessary. Signed policies provide stronger tamper resistance but are intentionally harder to remove, including during recovery. Documentation: Configure App Control for Business by using OSConfig | App Control for Business Keep Windows Defender Firewall enabled with restrictive rules A host firewall limits which systems and applications can communicate with the server, even when upstream network controls are absent or bypassed. Restrictive rules reduce exposure to service exploitation, scanning, lateral movement, remote administration abuse, command-and-control traffic, and accidental publication of listening services. Enable Windows Defender Firewall on Domain, Private, and Public profiles. Retain the default block for unsolicited inbound traffic and create only the rules required by the server role. Scope rules by program or service, protocol, local port, remote address, interface, and profile rather than creating broad port-based or any-source exceptions. Log dropped packets and successful connections where operationally appropriate, centrally monitor policy changes, and review stale rules. Apply explicit outbound restrictions to high-value or tightly controlled servers where feasible, especially when they should communicate with only a small set of update, identity, management, and application endpoints. Implementation steps: Inventory listening services and required inbound and outbound flows, enable the firewall on all profiles, and create narrowly scoped rules for the server role. Decide whether locally created rules may merge with centrally deployed rules for each profile, verify the effective policy on representative servers, remove obsolete or duplicate rules, test application, domain, cluster, backup, and management traffic, enable appropriate logging, deploy the policy centrally, and monitor rule changes and blocked connections before introducing selective outbound restrictions. Possible drawbacks: Incorrect firewall rules can interrupt application traffic, clustering, domain operations, monitoring, backup, or remote management and can make diagnosis difficult. Detailed connection logging consumes storage, while restrictive outbound policies require continuous maintenance as service endpoints change, so rules should be documented, tested, and deployed with recovery access. Documentation: Windows Firewall rule recommendations | OSConfig baseline network protections Harden Remote Desktop and remote administration Remote administration exposes privileged authentication and interactive control paths that are attractive targets for brute-force attacks, credential theft, session hijacking, and exploitation of internet-facing services. Gateways, multifactor authentication, encrypted sessions, restricted source networks, and credential isolation reduce the likelihood that a stolen password or exposed management port leads directly to server compromise. Disable Remote Desktop Services when it is not required. When it is required, use a VPN or Remote Desktop Gateway, require multifactor authentication and Network Level Authentication, restrict source networks and authorized groups, use trusted TLS certificates, and configure sensible idle and disconnected-session limits. Disable clipboard, drive, printer, port, and device redirection unless the operational need outweighs the data-transfer risk. Use Remote Credential Guard only for compatible direct RDP administration of Active Directory-joined targets using Kerberos so credentials aren't sent to the remote host. Remote Credential Guard isn't supported through Remote Desktop Gateway or Remote Desktop Connection Broker. For helpdesk access to a potentially compromised host, use Restricted Admin mode instead of Remote Credential Guard. Never expose TCP port 3389 directly to the internet, and avoid using saved privileged credentials on ordinary administrator workstations. Implementation steps: Disable RDP on servers that don't require it. For brokered or externally initiated access, place RDP behind a VPN or MFA-protected Remote Desktop Gateway and restrict permitted users and source networks. For compatible direct RDP administration of Active Directory-joined targets, configure Remote Credential Guard separately; use Restricted Admin mode for appropriate helpdesk scenarios. Configure Network Level Authentication, trusted TLS certificates, session limits, and required redirection controls, test routine and emergency access, and monitor remote logons and gateway activity. Possible drawbacks: Gateways, VPNs, MFA services, and secure administrative hosts add licensing, infrastructure, and support dependencies, and their outage can block legitimate administration. Device-redirection restrictions can hinder support workflows, while Network Level Authentication and Remote Credential Guard have compatibility and delegation limitations; a separately secured emergency access path is therefore required. Documentation: Plan multifactor authentication for Remote Desktop Services | Remote Credential Guard Enforce least privilege and separate administrative identities Least privilege limits each identity and session to the minimum actions required for its task. Separating standard and privileged accounts constrains the damage caused by phishing, token or password theft, malicious insiders, vulnerable administrative tools, and compromised lower-trust devices, while reducing opportunities for privilege escalation and persistence. Give administrators standard user accounts for routine work and separate privileged accounts for administrative duties. Minimize membership of local Administrators, Domain Admins, Enterprise Admins, and other powerful groups; review membership and assigned user rights regularly; and prevent highly privileged identities from signing in to lower-trust servers and workstations. Use Just Enough Administration endpoints, Windows Admin Center role-based access control, and time-limited elevation where possible. Delegate specific tasks rather than granting unrestricted interactive or PowerShell access, and maintain separately protected emergency accounts for identity-service outages. Implementation steps: Inventory privileged human accounts, service principals, managed identities, service and automation credentials, scheduled tasks, local group membership, duties, and logon locations. Create separate standard and administrative identities, remove unnecessary standing memberships, delegate tasks through role groups, JEA endpoints, Windows Admin Center RBAC, or time-limited elevation, restrict high-tier logons to secured administrative hosts, test that each role can perform its approved duties, and monitor privileged-group, role, automation, and emergency-account use. Possible drawbacks: Designing roles, JEA endpoints, approval processes, and time-limited access requires ongoing engineering and governance. Excessively narrow delegation can delay troubleshooting or incident response, while separate accounts add friction for administrators, so permissions should be tested against real duties and emergency access should remain tightly controlled but usable. Documentation: Just Enough Administration | Windows Admin Center user access options | Enterprise access model Deploy Windows LAPS for local administrator credentials Windows LAPS replaces shared or manually maintained local administrator passwords with unique, random, automatically rotated credentials. It mitigates password reuse, pass-the-hash attacks, credential dumping, and broad lateral movement in which compromise of one server's local administrator credential grants access to many others. Use Windows Local Administrator Password Solution to assign a unique, random, automatically rotated local administrator password to every server. The backup destination depends on join state: Active Directory-only devices can use only Active Directory, Microsoft Entra-only devices can use only Microsoft Entra ID, and hybrid-joined devices can use either destination but not both simultaneously. Tightly restrict and audit password retrieval, configure password history and post-authentication rotation, and monitor policy-processing failures. Never reuse a common local administrator password across servers because one compromised password or hash can enable broad lateral movement. OSConfig provides the LAPS/WindowsServer/2025/MemberServer scenario for member servers. Workgroup systems can be managed through LAPS for Azure Arc, which Microsoft currently documents as a preview feature. On domain controllers, use Windows LAPS to manage the Directory Services Restore Mode password where appropriate. Implementation steps: Select the password-backup destination permitted by the device's join state, prepare Active Directory or Microsoft Entra ID, and identify the local account to manage. For workgroup systems, evaluate the operational and support implications of the preview LAPS for Azure Arc service before adoption. Configure password length, complexity, age, history, and post-authentication actions through policy or OSConfig; configure DSRM password management for applicable domain controllers; delegate password read and reset permissions to a small approved group; pilot the policy; verify password backup and rotation; test authorized recovery; and monitor LAPS processing and retrieval events. Possible drawbacks: LAPS introduces directory, policy, permissions, auditing, and recovery dependencies that must be designed correctly. Scripts or applications that rely on a fixed local password can fail, password rotation can disrupt active sessions or automation, and overly broad rights to retrieve stored passwords can create a new privileged credential repository for attackers to target. Documentation: What is Windows LAPS? | OSConfig Windows LAPS scenario | LAPS for Azure Arc Replace static service-account passwords with managed service accounts Managed service accounts replace human-managed, long-lived service passwords with complex credentials that Active Directory changes automatically. This reduces exposure to password theft, reuse, weak password selection, expired credentials, secrets embedded in scripts, and persistence based on service accounts whose passwords are rarely rotated. Use group managed service accounts for supported Windows services, scheduled tasks, and application pools in Active Directory environments. gMSAs provide automatic password management and reduce the need to store or manually rotate long-lived service credentials. Windows Server 2025 also introduces delegated Managed Service Accounts for supported migrations from traditional service accounts. A dMSA binds authentication to approved machine identities, uses managed randomized keys, and disables use of the original service-account password. dMSA deployment requires a discoverable Windows Server 2025 domain controller, and an existing gMSA can't be migrated to a dMSA. Grant each gMSA only the logon rights, resource permissions, and password-retrieval scope it requires. Do not make service accounts members of privileged groups unless unavoidable, prohibit interactive sign-in, remove obsolete accounts promptly, and monitor changes to the hosts permitted to retrieve each managed password. Implementation steps: Inventory service identities and application dependencies, then select a gMSA for supported services that can directly use a managed account or evaluate a dMSA for a supported Windows Server 2025 migration from a traditional service account. For a gMSA, confirm Active Directory and key-distribution prerequisites, limit which hosts may retrieve its password, assign only required logon rights, permissions, and service principal names, install and test the account on approved hosts, migrate the service or task, and disable or remove the former static-password account. For a dMSA, confirm a discoverable Windows Server 2025 domain controller and follow the documented migration and rollback process. Possible drawbacks: Managed service accounts depend on Active Directory and are not supported by every application, installer, or cross-platform workload. Migration can involve service-principal-name, delegation, permission, and clustering changes, while an overly broad password-retrieval scope allows additional hosts to use the identity. dMSA also requires Windows Server 2025 domain-controller availability and has migration rules that differ from gMSA, so compatibility, rollback, and access boundaries require careful testing. Documentation: Secure group managed service accounts | Delegated Managed Service Accounts overview | Delegated Managed Service Accounts FAQ Protect credentials and phase out legacy authentication Credential isolation and modern authentication reduce the value of secrets that an attacker can extract or relay. Credential Guard, LSA protection, Kerberos AES, and retirement of weak authentication mitigate memory scraping, pass-the-hash, pass-the-ticket, NTLM relay, downgrade attacks, and cracking of obsolete password representations. Verify that Credential Guard and Local Security Authority protection are active where hardware and workload compatibility permit. Windows Server 2025 enables Credential Guard by default on eligible domain-joined systems that aren't domain controllers, but the state should still be verified and centrally enforced where required. Use Negotiate with Kerberos and modern AES encryption for domain authentication, prevent storage of LM hashes or reversibly encrypted passwords, and keep delegated credentials non-exportable. NTLMv1 is removed in Windows Server 2025, and deprecated NTLMv2 should be treated only as a temporary compatibility fallback rather than an end state. Audit NTLM and other legacy authentication dependencies before restricting or disabling them, then remove those dependencies in a controlled sequence. Do not disable legacy protocols blindly on production servers, but do not leave them enabled indefinitely solely because their consumers have not been inventoried. Implementation steps: Confirm hardware and driver support, verify the default Credential Guard state, and use OSConfig or centrally managed policy to enforce Credential Guard and LSA protection where required. Enable NTLM auditing, inventory clients and services using legacy authentication, configure Negotiate and Kerberos AES, update affected service accounts, remediate dependencies, assign an owner and retirement date to every NTLMv2 exception, introduce NTLM restrictions in stages, and monitor authentication failures before broader enforcement. Possible drawbacks: Virtualization-based credential protection requires compatible hardware and can have a workload-dependent performance or compatibility impact. Legacy devices, applications, trusts, or service configurations may still depend on NTLM or weaker cryptography, and disabling them without complete auditing can cause widespread authentication failures or outages. Documentation: Credential Guard overview | OSConfig baseline credential protections | Deprecated Windows Server features Harden SMB and file-server access SMB hardening protects Windows file sharing and related management traffic against protocol downgrade, relay attacks, on-path tampering, brute-force authentication, guest access, data disclosure, and exploitation of obsolete implementations such as SMBv1. Signing verifies message integrity, while encryption protects sensitive content in transit. Remove SMBv1, prevent insecure guest logons, retain and verify the Windows Server 2025 default requirement for inbound and outbound SMB signing, use SMB encryption for sensitive or untrusted network paths, and use SMB 3.x for modern file services. Treat any relaxation of signing for an incompatible third-party device as a documented, isolated, and time-bound exception. Windows Server 2025 also provides SMB authentication rate limiting and stronger signing and encryption capabilities that should be retained unless a documented compatibility requirement exists. Restrict TCP port 445 to approved clients and servers, apply share and NTFS permissions according to least privilege, enable access-based enumeration where appropriate, and audit access to sensitive shares. Do not publish traditional SMB directly to the internet; use a supported secure access design such as SMB over QUIC when its requirements and threat model fit. Implementation steps: Inventory SMB clients, servers, protocol versions, shares, and access requirements, then remove SMBv1 and insecure guest access. Verify that inbound and outbound signing remain required, document and isolate any temporary third-party compatibility exception, configure encryption, authentication rate limiting, and firewall scope according to the workload, review share and NTFS permissions, pilot changes with older clients and high-throughput workloads, and monitor SMB security, authentication, and performance events after enforcement. Possible drawbacks: Mandatory SMB signing and encryption consume processor resources and can reduce throughput or increase latency on demanding file workloads. Older storage appliances, scanners, applications, or clients might not support modern SMB requirements, and overly restrictive port or permission changes can disrupt file access, administration, Group Policy, or backup operations. Documentation: SMB security hardening | Secure SMB traffic in Windows Server Require modern TLS and manage certificates securely Modern TLS protects application and management traffic by authenticating endpoints and encrypting data in transit. Requiring current protocol versions, strong cipher suites, and trusted certificates mitigates eavesdropping, man-in-the-middle attacks, protocol downgrade, weak-cryptography attacks, and impersonation using invalid or compromised certificates. Require TLS 1.2 or later and prefer TLS 1.3 where the application stack supports it. Windows Server 2025 disables TLS 1.0 and TLS 1.1 by default; verify that these protocols and obsolete SSL versions remain disabled and prevent unauthorized re-enablement. Disable weak cipher suites and obsolete hashes through a tested baseline rather than ad hoc registry changes. Inventory old agents, middleware, and network appliances first so incompatible dependencies can be upgraded instead of becoming permanent exceptions. Use certificates from a trusted public or enterprise certification authority, protect private keys with restrictive access control, select appropriate key sizes and algorithms, monitor expiration, and automate renewal. After dependency review, remove expired, untrusted, orphaned, or unnecessary certificates from server stores. Implementation steps: Inventory listening services, clients, protocol versions, cipher dependencies, and installed certificates, then replace weak or expiring certificates and confirm application support for modern TLS. Verify that TLS 1.0 and TLS 1.1 remain disabled, apply tested Schannel or OSConfig settings in stages, disable other legacy protocols and weak ciphers, validate every client and integration, rescan the endpoints, remove only certificates confirmed to be unnecessary, and implement automated certificate enrollment, renewal, expiration alerting, and private-key access review. Possible drawbacks: Disabling old protocols and ciphers can break legacy clients, middleware, monitoring agents, or network devices with no modern TLS support. Certificate issuance, private-key protection, renewal automation, and revocation checking add operational complexity, and an expired or incorrectly deployed certificate can cause a complete service outage. Documentation: TLS/SSL and Schannel overview | OSConfig baseline protocol protections | Deprecated Windows Server features Encrypt operating-system and data volumes with BitLocker BitLocker encrypts data at rest so possession of a disk or offline copy does not provide immediate access to its contents. It mitigates data theft from lost or stolen servers, removed drives, improperly decommissioned hardware, offline password-reset attacks, and attempts to read files by booting an alternate operating system. Enable BitLocker on operating-system and data volumes, using TPM-backed protectors and additional startup authentication where the physical threat model and availability requirements justify it. Use virtual TPMs and supported host or cloud protections for virtual machines. Encryption protects data on removed drives, decommissioned hardware, stolen systems, and offline copies. Escrow recovery information in a protected, recoverable directory or management service before enforcement. Limit access to recovery keys, audit retrieval, include key recovery in incident procedures, and test recovery on representative systems so encryption does not become an availability risk. Implementation steps: Inventory operating-system and data volumes, confirm TPM or virtual TPM readiness, select protectors that meet the physical and availability threat model, and configure a protected recovery-key escrow location. Enable BitLocker in controlled stages, verify encryption and key backup, test normal reboot and recovery scenarios, document break-glass procedures, and continuously monitor encryption and protector compliance. Suspend protection for firmware or boot-chain maintenance only through an approved procedure, then verify that BitLocker protection resumes afterward. Possible drawbacks: Lost recovery material can make encrypted data permanently inaccessible, while firmware, TPM, boot, or hardware changes can unexpectedly trigger recovery. Encryption can add some performance and operational overhead, and startup PINs can conflict with unattended reboot requirements, so protector selection, key escrow, and recovery testing must reflect the server's availability needs. Documentation: BitLocker planning guide | BitLocker operations guide | BitLocker recovery overview Configure detailed auditing and protect local logs Detailed auditing records security-relevant activity so suspicious behavior can be detected, investigated, and attributed. Authentication, privilege, process, PowerShell, policy, and firewall logs help expose brute-force attempts, credential misuse, privilege escalation, persistence, defense evasion, and attacker efforts to alter system configuration. Enable advanced audit policy for successful and failed logons, credential validation, account and group changes, sensitive privilege use, process creation with command-line capture, policy changes, removable storage, file shares, firewall activity, and other events relevant to the server role. The OSConfig baseline enables a broad audit configuration and increases important log sizes to improve forensic coverage. Enable PowerShell module and script block logging, and use protected event logging where appropriate because command content can contain sensitive data. Increase log capacity and retention for the expected event volume, restrict permissions to clear or modify logs, monitor audit-policy changes, and synchronize time with trusted sources. Implementation steps: Define the activities and events required for detection, investigation, and compliance, then apply advanced audit policy through OSConfig or Group Policy. Enable process command-line and PowerShell logging, measure event volume during a representative pilot, size and protect each log and forwarding path from the observed rates, configure trusted time synchronization, generate representative test events to confirm collection, and review event volume, retention, and policy health regularly. Possible drawbacks: Detailed auditing can generate large volumes of events, consume storage and processing resources, and overwhelm analysts with noise if collection isn't tuned. Command-line and PowerShell logs can contain credentials or other sensitive data, while undersized logs may overwrite useful evidence, so access, retention, filtering, and capacity require deliberate design. Documentation: OSConfig baseline auditing and visibility | Recommended audit policy for Windows Event Forwarding | PowerShell logging on Windows Centralize security telemetry and alert on suspicious activity Centralized telemetry moves evidence away from the system that generated it and correlates activity across servers, identities, and networks. This improves detection of distributed attacks, limits an intruder's ability to erase local evidence, and shortens response time for credential attacks, lateral movement, persistence, defense evasion, and destructive actions. Forward security-relevant logs away from each server using Windows Event Forwarding, Azure Monitor, Microsoft Defender, a SIEM such as Microsoft Sentinel, or another protected collection platform. Include Security, System, Windows Defender, PowerShell, Code Integrity, Windows Firewall, Windows LAPS, and role-specific operational logs. Create actionable alerts for repeated authentication failures, new or changed administrators, unexpected service or scheduled-task creation, security-control changes, Defender detections, App Control blocks, log clearing, unusual remote administration, and backup deletion. Restrict access to collectors and retention systems so an attacker who compromises a server cannot erase the centralized evidence. Implementation steps: Select Windows Event Forwarding, Azure Monitor, Microsoft Defender, a SIEM, or a combination; define prioritized detection use cases and role-specific retention before selecting log channels and verbosity; design resilient collectors, access control, and capacity; and deploy the required agents or subscriptions. Onboard the prioritized channels, verify end-to-end ingestion and timestamps, create and test high-value detections and notifications, restrict access to the monitoring platform, and continuously monitor collection health and tune noisy rules. Possible drawbacks: Central collection introduces bandwidth, storage, ingestion, licensing, retention, and analyst costs and can expose sensitive operational data if the monitoring platform is poorly secured. Collector failures create visibility gaps, while poorly tuned rules produce false positives and alert fatigue, so the design needs resilience, health monitoring, access controls, and continuous tuning. Documentation: Use Windows Event Forwarding for intrusion detection | Microsoft Defender for Endpoint security capabilities Maintain ransomware-resilient backups and test recovery Ransomware-resilient backups preserve a trustworthy recovery path when production data, operating systems, or identity services are encrypted, deleted, or corrupted. Isolated and immutable copies mitigate ransomware, destructive administrators, compromised backup credentials, accidental deletion, hardware failure, and attacks intended to eliminate both systems and their recovery data. Keep multiple protected backup copies, including a copy that is offline, immutable, or otherwise isolated from normal server and domain administrator credentials. Use separate backup administration identities, multifactor authorization for destructive operations, encryption, soft delete or immutability controls, and alerts for policy changes or mass deletion. Hypervisor snapshots alone are not an adequate backup strategy. Back up application data and configuration as well as system state and bare-metal recovery data where required by the server role. Define recovery-point and recovery-time objectives, test file, application, system-state, and full-server restoration regularly, and record the evidence. Domain controllers, certificate authorities, and other identity infrastructure require workload-aware recovery procedures. Implementation steps: Classify workloads and define recovery-point and recovery-time objectives, then select local, offsite, offline, and immutable backup targets appropriate to the risk. Use separate backup identities and MFA, schedule application data, configuration, system-state, and bare-metal backups as required, enable encryption and deletion protections, and monitor every job and policy change. Perform regular isolated restore tests that verify application consistency and role-specific recovery semantics for identity systems, not only successful restoration of files or virtual disks, and maintain documented recovery runbooks. Possible drawbacks: Multiple isolated copies, immutable storage, long retention, and regular restore exercises increase storage, network, licensing, staffing, and operational costs. Backups can create false confidence when they are incomplete, stale, infected, or untested, and strong credential separation can slow routine administration, so restore validation and lifecycle management are as important as backup creation. Documentation: Design a ransomware-resilient backup architecture | Azure Backup security best practices | Back up Windows Server system state --- This isn't everything you can do, but it's a start. What other techniques do you use to harden your Windows Server deployments?Hyper-V Virtual TPMs, Certificates, VM Export and Migration
Virtual Trusted Platform Modules (vTPM) in Hyper-V allow you to run guest operating systems, such as Windows 11 or Windows Server 2025 with security features enabled. One of the challenges of vTPMs is that they rely on certificates on the local Hyper-V server. Great if you’re only running the VM with the vTPM on that server, but a possible cause of issues if you want to move that VM to another server. In this article I’ll show you how to manage the certificates that are associated with vTPMs so that you’ll be able to export or move VMs that use them, such as Windows 11 VMs, to any prepared Hyper-V host you manage. When a vTPM is enabled on a Generation 2 virtual machine, Hyper-V automatically generates a pair of self-signed certificates on the host where the VM resides. These certificates are specifically named: "Shielded VM Encryption Certificate (UntrustedGuardian)(ComputerName)" "Shielded VM Signing Certificate (UntrustedGuardian)(ComputerName)". These certificates are stored in a unique local certificate store on the Hyper-V host named "Shielded VM Local Certificates". By default, these certificates are provisioned with a validity period of 10 years. For a vTPM-enabled virtual machine to successfully live migrate and subsequently start on a new Hyper-V host, the "Shielded VM Local Certificates" (both the Encryption and Signing certificates) from the source host must be present and trusted on all potential destination Hyper-V hosts. Exporting vTPM related certificates. You can transfer certificates from one Hyper-V host to another using the following procedure: On the source Hyper-V host, open mmc.exe. From the "File" menu, select "Add/Remove Snap-in..." In the "Add or Remove Snap-ins" window, select "Certificates" and click "Add." Choose "Computer account" and then "Local Computer". Navigate through the console tree to "Certificates (Local Computer) > Personal > Shielded VM Local Certificates". Select both the "Shielded VM Encryption Certificate" and the "Shielded VM Signing Certificate." Right-click the selected certificates, choose "All Tasks," and then click "Export". In the Certificate Export Wizard, on the "Export Private Key" page, select "Yes, export the private key". The certificates are unusable for their intended purpose without their associated private keys. Select "Personal Information Exchange - PKCS #12 (.PFX)" as the export file format. Select "Include all certificates in the certification path if possible". Provide a strong password to protect the PFX file. This password will be required during the import process. To perform this process using the command line, display details of the certificates in the "Shielded VM Local Certificates" store, including their serial numbers. certutil -store "Shielded VM Local Certificates" Use the serial numbers to export each certificate, ensuring the private key is included. Replace <Serial_Number_Encryption_Cert> and <Serial_Number_Signing_Cert> with the actual serial numbers, and "YourSecurePassword" with a strong password: certutil -exportPFX -p "YourSecurePassword" "Shielded VM Local Certificates" <Serial_Number_Encryption_Cert> C:\Temp\VMEncryption.pfx certutil -exportPFX -p "YourSecurePassword" "Shielded VM Local Certificates" <Serial_Number_Signing_Cert> C:\Temp\VMSigning.pfx Importing vTPM related certificates To import these certificates on a Hyper-V host that you want to migrate a vTPM enabled VM to, perform the following steps: Transfer the exported PFX files to all Hyper-V hosts that will serve as potential live migration targets. On each target host, open mmc.exe and add the "Certificates" snap-in for the "Computer account" (Local Computer). Navigate to "Certificates (Local Computer) > Personal." Right-click the "Personal" folder, choose "All Tasks," and then click "Import". Proceed through the Certificate Import Wizard. Ensure the certificates are placed in the "Shielded VM Local Certificates" store. After completing the wizard, verify that both the Encryption and Signing certificates now appear in the "Shielded VM Local Certificates" store on the new host. You can accomplish the same thing using PowerShell with the following command: Import-PfxCertificate -FilePath "C:\Backup\CertificateName.pfx" -CertStoreLocation "Cert:\LocalMachine\Shielded VM Local Certificates" -Password (ConvertTo-SecureString -String "YourPassword" -Force -AsPlainText) Updating vTPM related certificates. Self signed vTPM certificates automatically expire after 10 years. Resetting the key protector for a vTPM-enabled VM in Hyper-V allows you change or renew the underlying certificates (especially if the private key changes). Here are the requirements and considerations around this process: The VM must be in an off state to change security settings or reset the key protector The host must have the appropriate certificates (including private keys) in the "Shielded VM Local Certificates" store. If the private key is missing, the key protector cannot be set or validated. Always back up the VM and existing certificates before resetting the key protector, as this process can make previously encrypted data inaccessible if not performed correctly. The VM must be at a supported configuration version (typically version 7.0 or higher) to support vTPM and key protector features. To save the Current Key Protector: On the source Hyper-V host, retrieve the current Key Protector for the VM and save it to a file. Get-VMKeyProtector -VMName 'VM001' | Out-File '.\VM001.kp' To reset the key protector with a new local key protector: Set-VMKeyProtector -VMName "<VMNAME>" -NewLocalKeyProtector This command instructs Hyper-V to generate a new key protector using the current local certificates. After resetting, enable vTPM if needed: Enable-VMTPM -VMName "<VMNAME>" It is important to note that if an incorrect Key Protector is applied to the VM, it may fail to start. In such cases, the Set-VMKeyProtector -RestoreLastKnownGoodKeyProtector cmdlet can be used to revert to the last known working Key Protector. More information: Set-VMKeyProtector: https://learn.microsoft.com/en-us/powershell/module/hyper-v/set-vmkeyprotectorInstall and run Azure Foundry Local LLM server & Open WebUI on Windows Server 2025
Foundry Local is an on-device AI inference solution offering performance, privacy, customization, and cost advantages. It integrates seamlessly into your existing workflows and applications through an intuitive CLI, SDK, and REST API. Foundry Local has the following benefits: On-Device Inference: Run models locally on your own hardware, reducing your costs while keeping all your data on your device. Model Customization: Select from preset models or use your own to meet specific requirements and use cases. Cost Efficiency: Eliminate recurring cloud service costs by using your existing hardware, making AI more accessible. Seamless Integration: Connect with your applications through an SDK, API endpoints, or the CLI, with easy scaling to Azure AI Foundry as your needs grow. Foundry Local is ideal for scenarios where: You want to keep sensitive data on your device. You need to operate in environments with limited or no internet connectivity. You want to reduce cloud inference costs. You need low-latency AI responses for real-time applications. You want to experiment with AI models before deploying to a cloud environment. You can install Foundry Local by running the following command: winget install Microsoft.FoundryLocal Once Foundry Local is installed, you download and interact with a model from the command line by using a command like: foundry model run phi-4 This will download the phi-4 model and provide a text based chat interface. If you want to interact with Foundry Local through a web chat interface, you can use the open source Open WebUI project. You can install Open WebUI on Windows Server by performing the following steps: Download OpenWebUIInstaller.exe from https://github.com/BrainDriveAI/OpenWebUI_CondaInstaller/releases. You'll get warning messages from Windows Defender SmartScreen. Copy OpenWebUIInstaller.exe into C:\Temp. In an elevated command prompt, run the following commands winget install -e --id Anaconda.Miniconda3 --scope machine $env:Path = 'C:\ProgramData\miniconda3;' + $env:Path $env:Path = 'C:\ProgramData\miniconda3\Scripts;' + $env:Path $env:Path = 'C:\ProgramData\miniconda3\Library\bin;' + $env:Path conda.exe tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main conda.exe tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r conda.exe tos accept --override-channels --channel https://repo.anaconda.com/pkgs/msys2 C:\Temp\OpenWebUIInstaller.exe Then from the dialog choose to install and run Open WebUI. You then need to take several extra steps to configure Open WebUI to connect to the Foundry Local endpoint. Enable Direct Connections in Open WebUI Select Settings and Admin Settings in the profile menu. Select Connections in the navigation menu. Enable Direct Connections by turning on the toggle. This allows users to connect to their own OpenAI compatible API endpoints. Connect Open WebUI to Foundry Local: Select Settings in the profile menu. Select Connections in the navigation menu. Select + by Manage Direct Connections. For the URL, enter http://localhost:PORT/v1 where PORT is the Foundry Local endpoint port (use the CLI command foundry service status to find it). Note that Foundry Local dynamically assigns a port, so it isn't always the same. For the Auth, select None. Select Save ➡️ What is Foundry Local https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-local/what-is-foundry-local ➡️ Edge AI for Beginners https://aka.ms/edgeai-for-beginners ➡️ Open WebUI: https://docs.openwebui.com/Internet Information Services Learning Path
Internet Information Services (IIS) is the modular web server and application platform that first shipped as an optional add-on for Windows NT 3.51 in 1995, evolving from basic HTTP and FTP services into a full-featured web stack tightly integrated with Windows Server and Client operating systems. Today, IIS remains a core component of Windows Server and client editions, used to host legacy and modern web applications, APIs, and services in on-premises and hybrid environments alongside newer cloud-native options. The following training modules on Microsoft Learn provide you with a structured curriculum to learn how to manage and maintain Internet Information Services. It's relevant to almost all supported versions of IIS, but also includes coverage of some functionality that is available with Windows Server 2025. The modules in this learning path are as follows: Deploy and configure Internet Information Services. Understand how to configure, administer, and validate an IIS installation on Windows Server. Configure Internet Information Services websites and applications. Create and configure websites, web applications, and virtual directories in IIS. Manage Internet Information Services. Scripting bulk configuration changes, watching worker processes for signs of trouble, or run health checks to keep your IIS environment running reliably. Secure and harden Internet Information Services. Authentication, authorization, and other security best practices to protect the server and websites. Troubleshoot and optimize IIS performance. Learn how to troubleshoot IIS problems with logs, tracing, and performance counters, then apply tuning changes to improve reliability and throughput. The Administer Internet Information Services learning path collects all these modules together, providing you with a comprehensive guide to managing and maintaining Internet Information Services.GPU Partitioning in Windows Server 2025 Hyper-V
GPU Partitioning (GPU-P) is a feature in Windows Server 2025 Hyper-V that allows multiple virtual machines to share a single physical GPU by dividing it into isolated fractions. Each VM is allocated a dedicated portion of the GPU’s resources (memory, compute, encoders, etc.) instead of using the entire GPU. This is achieved via Single-Root I/O Virtualization (SR-IOV), which provides a hardware-enforced isolation between GPU partitions, ensuring each VM can access only its assigned GPU fraction with predictable performance and security. In contrast, GPU Passthrough (also known as Discrete Device Assignment, DDA) assigns a whole physical GPU exclusively to one VM. With DDA, the VM gets full control of the GPU, but no other VMs can use that GPU simultaneously. GPU-P’s ability to time-slice or partition the GPU allows higher utilization and VM density for graphics or compute workloads, whereas DDA offers maximum performance for a single VM at the cost of flexibility. GPU-P is ideal when you want to share a GPU among multiple VMs, such as for VDI desktops or AI inference tasks that only need a portion of a GPU’s power. DDA (passthrough) is preferred when a workload needs the full GPU (e.g. large model training) or when the GPU doesn’t support partitioning. Another major difference is mobility: GPU-P supports live VM mobility and failover clustering, meaning a VM using a GPU partition can move or restart on another host with minimal downtime. DDA-backed VMs cannot live-migrate. If you need to move a DDA VM, it must be powered off and then started on a target host (in clustering, a DDA VM will be restarted on a node with an available GPU upon failover, since live migration isn’t supported). Additionally, you cannot mix modes on the same device. A physical GPU can be either partitioned for GPU-P or passed through via DDA, but not both simultaneously. Supported GPU Hardware and Driver Requirements GPU Partitioning in Windows Server 2025 is supported on select GPU hardware that provides SR-IOV or similar virtualization capabilities, along with appropriate drivers. Only specific GPUs support GPU-P and you won’t be able to configure it on a consumer gaming GPU like your RTX 5090. In addition to the GPU itself, certain platform features are required: Modern CPU with IOMMU: The host processors must support Intel VT-d or AMD-Vi with DMA remapping (IOMMU). This is crucial for mapping device memory securely between host and VMs. Older processors lacking these enhancements may not fully support live migration of GPU partitions. BIOS Settings: Ensure that in each host’s UEFI/BIOS, Intel VT-d/AMD-Vi and SR-IOV are enabled. These options may be under virtualization or PCIe settings. Without SR-IOV enabled at the firmware level, the OS will not recognize the GPU as partitionable (in Windows Admin Center it might show status “Paravirtualization” indicating the driver is capable but the platform isn’t). Host GPU Drivers: Use vendor-provided drivers that support GPU virtualization. For NVIDIA, this means installing the NVIDIA virtual GPU (vGPU) driver on the Windows Server 2025 host (the driver package that supports GPU-P). Check the GPU vendor’s documentation for installation for specifics. After installing, you can verify the GPU’s status via PowerShell or WAC. Guest VM Drivers: The guest VMs also need appropriate GPU drivers installed (within the VM’s OS) to make use of the virtual GPU. For instance, if using Windows 11 or Windows Server 2025 as a guest, install the GPU driver inside the VM (often the same data-center driver or a guest-compatible subset from the vGPU package) so that the GPU is usable for DirectX/OpenGL or CUDA in that VM. Linux guests (Ubuntu 18.04/20.04/22.04 are supported) likewise need the Linux driver installed. Guest OS support for GPU-P in WS2025 covers Windows 10/11, Windows Server 2019+, and certain Ubuntu LTS versions. After hardware setup and driver installation, it’s important to verify that the host recognizes the GPU as “partitionable.” You can use Windows Admin Center or PowerShell for this: in WAC’s GPU tab, check the “Assigned status” of the GPU it should show “Partitioned” if everything is configured correctly (if it shows “Ready for DDA assignment” then the partitioning driver isn’t active, and if “Not assignable” then the GPU/driver doesn’t support either method). In PowerShell, you can run: Get-VMHostPartitionableGpu | FL Name, ValidPartitionCounts, PartitionCount This will list each GPU device’s identifier and what partition counts it supports. For example, an NVIDIA A40 might return ValidPartitionCounts : {16, 8, 4, 2 …} indicating the GPU can be split into 2, 4, 8, or 16 partitions, and also show the current PartitionCount setting (by default it may equal the max or current configured value). If no GPUs are listed, or the list is empty, the GPU is not recognized as partitionable (check drivers/BIOS). If the GPU is listed but ValidPartitionCounts is blank or shows only “1,” then it may not support SR-IOV and can only be used via DDA. Enabling and Configuring GPU Partitioning Once the hardware and drivers are ready, enabling GPU Partitioning involves configuring how the GPU will be divided and ensuring all Hyper-V hosts (especially in a cluster) have a consistent setup. Each physical GPU must be configured with a partition count (how many partitions to create on that GPU). You cannot define an arbitrary number – it must be one of the supported counts reported by the hardware/driver. The default might be the maximum supported (e.g., 16). To set a specific partition count, use PowerShell on each host: Decide on a partition count that suits your workloads. Fewer partitions means each VM gets more GPU resources (more VRAM and compute per partition), whereas more partitions means you can assign the GPU to more VMs concurrently (each getting a smaller slice). For AI/ML, you might choose a moderate number – e.g. split a 24 GB GPU into 4 partitions of ~6 GB each for inference tasks. Run the Set-VMHostPartitionableGpu cmdlet. Provide the GPU’s device ID (from the Name field of the earlier Get-VMHostPartitionableGpu output) and the desired -PartitionCount. For example: Set-VMHostPartitionableGpu -Name "<GPU-device-ID>" -PartitionCount 4 This would configure the GPU to be divided into 4 partitions. Repeat this for each GPU device if the host has multiple GPUs (or specify -Name accordingly for each). Verify the setting by running: Get-VMHostPartitionableGpu | FL Name,PartitionCount It should now show the PartitionCount set to your chosen value (e.g., PartitionCount : 4 for each listed GPU). If you are in a clustered environment, apply the same partition count on every host in the cluster for all identical GPUs. Consistency is critical: a VM using a “quarter GPU” partition can only fail over to another host that also has its GPU split into quarters. Windows Admin Center will actually enforce this by warning you if you try to set mismatched counts on different nodes. You can also configure the partition count via the WAC GUI. In WAC’s GPU partitions tool, select the GPU (or a set of homogeneous GPUs across hosts) and choose Configure partition count. WAC will present a dropdown of valid partition counts (as reported by the GPU). Selecting a number will show a tooltip of how much VRAM each partition would have (e.g., selecting 8 partitions on a 16 GB card might show ~2 GB per partition). WAC helps ensure you apply the change to all similar GPUs in the cluster together. After applying, it will update the partition count on each host automatically. After this step, the physical GPUs on the host (or cluster) are partitioned into the configured number of virtual GPUs. They are now ready to be assigned to VMs. The host’s perspective will show each partition as a shareable resource. (Note: You cannot assign more partitions to VMs than the number configured) Assigning GPU Partitions to Virtual Machines With the GPU partitioned at the host level, the next step is to attach a GPU partition to a VM. This is analogous to plugging a virtual GPU device into the VM. Each VM can have at most one GPU partition device attached, so choose the VM that needs GPU acceleration and assign one partition to it. There are two main ways to do this: using PowerShell commands or using the Windows Admin Center UI. Below are the instructions for each method. To add the GPU Partition to the VM use the Add-VMGpuPartitionAdapter cmdlet to attach a partitioned GPU to the VM. For example: Add-VMGpuPartitionAdapter -VMName "<VMName>" This will allocate one of the available GPU partitions on the host to the specified VM. (There is no parameter to specify which partition or GPU & Hyper-V will auto-select an available partition from a compatible GPU. If no partition is free or the host GPUs aren’t partitioned, this cmdlet will return an error) You can check that the VM has a GPU partition attached by running: Get-VMGpuPartitionAdapter -VMName "<VMName>" | FL InstancePath,PartitionId This will show details like the GPU device instance path and a PartitionId for the VM’s GPU device. If you see an entry with an instance path (matching the GPU’s PCI ID) and a PartitionId, the partition is successfully attached. Power on the VM. On boot, the VM’s OS will detect a new display adapter. In Windows guests, you should see a GPU in Device Manager (it may appear as a GPU with a specific model, or a virtual GPU device name). Install the appropriate GPU driver inside the VM if not already installed, so that the VM can fully utilize the GPU (for example, install NVIDIA drivers in the guest to get CUDA, DirectX, etc. working). Once the driver is active in the guest, the VM will be able to leverage the GPU partition for AI/ML computations or graphics rendering. Using Windows Admin Center: Open Windows Admin Center and navigate to your Hyper-V cluster or host, then go to the GPUs extension. Ensure you have added the GPUs extension v2.8.0 or later to WAC. In the GPU Partitions tab, you’ll see a list of the physical GPUs and any existing partitions. Click on “+ Assign partition”. This opens an assignment wizard. Select the VM: First choose the host server where the target VM currently resides (WAC will list all servers in the cluster). Then select the VM from that host to assign a partition to. (If a VM is greyed out in the list, it likely already has a GPU partition assigned or is incompatible.) Select Partition Size (VRAM): Choose the partition size from the dropdown. WAC will list options that correspond to the partition counts you configured. For example, if the GPU is split into 4, you might see an option like “25% of GPU (≈4 GB)” or similar. Ensure this matches the partition count you set. You cannot assign more memory than a partition contains. Offline Action (HA option): If the VM is clustered and you want it to be highly available, check the option for “Configure offline action to force shutdown” (if presented in the UI). Proceed to assign. WAC will automatically: shut down the VM (if it was running), attach a GPU partition to it, and then power the VM back on. After a brief moment, the VM should come online with the GPU partition attached. In the WAC GPU partitions list, you will now see an entry showing the VM name under the GPU partition it’s using. At this point, the VM is running with a virtual GPU. You can repeat the process for other VMs, up to the number of partitions available. Each physical GPU can only support a fixed number of active partitions equal to the PartitionCount set. If you attempt to assign more VMs than partitions, the additional VMs will not get a GPU (or the Add command will fail). Also note that a given VM can only occupy one partition on one GPU – you cannot span a single VM across multiple GPU partitions or across multiple GPUs with GPU-P. GPU Partitioning in Clustered Environments (Failover Clustering) One of the major benefits introduced with Windows Server 2025 is that GPU partitions can be used in Failover Clustering scenarios for high availability. This means you can have a Hyper-V cluster where VMs with virtual GPUs are clustered roles, capable of moving between hosts either through live migration (planned) or failover (unplanned). To utilize GPU-P in a cluster, you must pay special attention to configuration consistency and understand the current limitations: Use Windows Server 2025 Datacenter: As mentioned, clustering features (like failover) for GPU partitions are supported only on Datacenter edition. Homogeneous GPU Configuration: All hosts in the cluster should have identical GPU hardware and partitioning setup. Failover/Live Migration with GPU-P does not support mixing GPU models or partition sizes in a GPU-P cluster. Each host should have the same GPU model. The partition count configured (e.g., 4 or 8 etc.) must be the same on every host. This uniformity ensures that a VM expecting a certain size partition will find an equivalent on any other node. Windows Server 2025 introduces support for live migrating VMs that have a GPU partition attached. However, there are important caveats: Hardware support: Live migration with GPU-P requires that the hosts’ CPUs and chipsets fully support isolating DMA and device state. In practice, as noted, you need Intel VT-d or AMD-Vi enabled, and the CPUs ideally supporting “DMA bit tracking.” If this is in place, Hyper-V will attempt to live migrate the VM normally. During such a migration, the GPU’s state is not seamlessly copied like regular memory; instead, Windows will fallback to a slower migration process to preserve integrity. Specifically, when migrating a VM using GPU-P, Hyper-V automatically uses TCP/IP with compression (even if you have faster methods like RDMA configured). This is because device state transfer is more complex. The migration will still succeed, but you may notice higher CPU usage on the host and a longer migration time than usual. Cross-node compatibility: Ensure that the GPU driver versions on all hosts are the same, and that each host has an available partition for the VM. If a VM is running and you trigger a live migrate, Hyper-V will find a target where the VM can get an identical partition. If none are free, the migration will not proceed (or the VM may have to be restarted elsewhere as a failover). Failover (Unplanned Moves): If a host crashes or goes down, a clustered VM with a GPU partition will be automatically restarted on another node, much like any HA VM. The key difference is that the VM cannot save its state, so it will be a cold start on the new node, attaching to a new GPU partition there. When the VM comes up on the new node, it will request a GPU partition. Hyper-V will allocate one if available. If NodeB had no free partition (say all were assigned to other VMs), the VM might start but not get a GPU (and likely Windows would log an error that the virtual GPU could not start). Administrators should monitor and possibly leverage anti-affinity rules to avoid packing too many GPU VMs on one host if full automatic failover is required. To learn more about GPU-P on Windows Server 2025, consult the documentation on Learn: https://learn.microsoft.com/en-us/windows-server/virtualization/hyper-v/gpu-partitioningGetting Started with Windows Admin Center Virtualization Mode
Getting Started with Windows Admin Center Virtualization Mode Windows Admin Center (WAC) Virtualization Mode is a new, preview experience for managing large Hyper-V virtualization fabrics—compute, networking, and storage—from a single, web-based console. It’s designed to scale from a handful of hosts up to thousands, centralizing configuration and day-to-day operations. This post walks through: • What Virtualization Mode is and its constraints • How to install it on a Windows Server host • How to add an existing Hyper-V host into a resource group Prerequisites and Constraints Before you begin, note the current preview limitations: • The WAC Virtualization Mode server and the Hyper-V hosts it manages must be in the same Active Directory domain. • You cannot install Virtualization Mode side-by-side with a traditional WAC deployment on the same server. • Do not install Virtualization Mode directly on a Hyper-V host you plan to manage. – You can install it on a VM running on that host. • Plan for at least 8 GB RAM on the WAC Virtualization Mode server. For TLS, the walkthrough assumes you have an Enterprise CA and are deploying domain-trusted certificates to servers, so browsers automatically trust the HTTPS endpoint. You can use a self signed certificate, but you’ll end up with all the fun that entails when you use WAC-V from a host on which the self signed cert isn’t installed. Given the domain requirements of WAC-V and the hosts it manages, going the Enterprise CA method seemed the path of least resistance. Step 1 – Install the C++ Redistributable On your Windows Server 2025 host that will run WAC Virtualization Mode: 1. Open Windows Terminal or PowerShell. 2. Use winget to search for the VC++ redistributable: powershell winget search "VC Redist" 3. Identify the package corresponding to “Microsoft Visual C++ 2015–2022 Redistributable” (or equivalent). 4. Install it with winget, for example: powershell winget install "Microsoft.VC++2015-2022Redist-x64" This fulfills the runtime dependency for the WAC Virtualization Mode installer. Step 2 – Install Windows Admin Center Virtualization Mode 1. Download the installer 1. Download the Windows Admin Center Virtualization Mode installer from the Windows Insider Preview location provided in the official documentation. Save it to a local folder on the WAC host. 2. Run the setup wizard 1. Double-click the downloaded binary. 2. Approve the UAC prompt. 3. In the Welcome page, proceed as with traditional WAC setup. 3. Accept the license and choose setup type 1. Accept the license agreement. 2. Choose Express setup (suitable for most lab and PoC deployments). 4. Select a TLS certificate 1. When prompted for a TLS certificate: 1. Select a certificate issued by your Enterprise CA that matches the server name. 2. Using CA-issued certs ensures all domain-joined clients will trust the site without manual certificate import. 5. Configure PostgreSQL for WAC 1. Virtualization Mode uses PostgreSQL as its configuration and state database. 2. When prompted: 1. Provide a strong password for the database account WAC will use. 2. Record this securely if required by your org standards. 6. Configure update and diagnostic settings 1. Choose how WAC should be updated (manual/automatic). 2. Set diagnostic data preferences according to your policy. 7. Complete the installation 1. Click Install to deploy: 1. The WAC Virtualization Mode web service 2. The PostgreSQL database instance 2. When installation completes, click Finish. Step 3 – Sign In to Virtualization Mode 1. Open a browser on a domain-joined machine and browse to the WAC URL (for example, https://wac-vmode01.contoso.internal). 2. Sign in with your domain credentials that have appropriate rights to manage Hyper-V hosts (for example, DOMAIN\adminuser). 3. You’ll see the new Virtualization Mode UI, which differs significantly from traditional WAC and is optimized for fabric-wide management. Step 4 – Create a Resource Group Resource groups help you logically organize Hyper-V servers you’ll manage (for example, by site, function, or cluster membership). 1. In the Virtualization Mode UI, select Resource groups. 2. Click Create resource group. 3. Provide a name, such as Zava-Nested-Vert. 4. Save the resource group. You now have a logical container ready for one or more Hyper-V hosts. Step 5 – Prepare the Hyper-V Host Before adding an existing Hyper-V host: 1. Ensure the host is: 1. Running Hyper-V and reachable by FQDN (for example, zava-hvA.zavaops.internal). 2. In the same AD domain as the WAC Virtualization Mode server. 2. Temporarily open File and Printer Sharing from the Hyper-V host’s firewall to the WAC Virtualization Mode server: 1. This is required for initial onboarding. 2. After onboarding, you can re-lock firewall rules according to your security baseline. Step 6 – Add a Hyper-V Host to the Resource Group 1. In the WAC Virtualization Mode UI, go to your resource group. 2. Click the ellipsis (…) and choose Add resource. 3. On the Add resource page, select Compute (you’re adding a Hyper-V server, not a storage fabric resource). 4. Enter the Hyper-V host’s FQDN (for example, zava-hvA.zavaops.internal). 5. Confirm the host resolves correctly and proceed. Configure Networking Template 1. On the Networking page, assign fabric roles to NICs using the network template model: 1. Each NIC can be tagged for one or more roles: 1. Compute 2. Management 3. Storage 2. In a simple, single-NIC lab scenario, you may assign Compute, Management, and Storage all to Ethernet0. 3. All three roles must be fully assigned across available adapters before you can proceed. Configure Storage 1. On the Storage page, specify the storage model: 1. For an existing host using local disks, choose Use existing storage. 2. In future, you can select SAN or file server storage when those options are available and configured in your environment. Configure Compute Properties 1. On the Compute page, configure host-level defaults: 1. Enable or disable Enhanced Session Mode. 2. Set the maximum concurrent live migrations. 3. Confirm or update the default VM storage path. 2. Review the configuration, click Next, then Submit. 3. The Hyper-V host is registered into the resource group and becomes manageable via Virtualization Mode. Step 7 – Verify Host and VM Management With the host onboarded: 1. Open the resource group and select the Hyper-V host. 2. You’ll see a streamlined view similar to traditional WAC, with nodes for: 1. Event logs 2. Files 3. Networks 4. Storage 5. Windows Update 6. Virtual Machines 3. To validate functionality, create a test VM: 1. Go to Virtual Machines → Add. 2. Provide a VM name (for example, WS25-temp). 3. Set vCPUs (for example, 2). 4. Optionally enable nested virtualization. 5. Select the appropriate virtual switch. 6. Click Create, then attach an ISO or existing VHDX and complete OS setup. ▶️ Public Preview: https://aka.ms/WACDownloadvMode ▶️ Documentation: https://aka.ms/WACvModeDocs
