Understanding the Repair Active Directory Object Recovery Action
Published Mar 15 2019 02:39 PM 42.6K Views
Microsoft

First published on MSDN on Dec 13, 2013

 

One of the responsibilities of cluster Network Name resource is to rotate the password of the computer object in Active Directory associated with it.  When the Network Name resource is online, it will rotate the password according to domain and local machine policy (which is 30 days by default).

 

If the password is different from what is stored in the cluster database, the cluster service will be unable to logon to the computer object and the Network Name will fail to come online.  This may also cause issues such as Kerberos errors, failure to register in a secure DNS zone, and live migration to fail.


The Repair Active Directory Object option is a recovery tool to re-synchronize the password for cluster computer objects.  It can be found in Failover Cluster Manager (CluAdmin.msc) by right-clicking on the Network Name, selecting More Actions…, and then clicking Repair Active Directory Object.

 

    • Cluster Name Object (CNO) - The CNO is the computer object associated with the Cluster Name resource.  When using Repair on the Cluster Name, it will use the credentials of the currently logged on user and reset the computer objects password.  To run Repair, you must have the "Reset Password" permissions to the CNO computer object.
    • Virtual Computer Object (VCO) - The CNO is responsible for managing the passwords on all other computer objects (VCO's) for other cluster network names in the cluster.  If the password for a VCO falls out of sync, the CNO will reset the password and self-heal automatically.  Therefore it is not needed to run Repair to reset the password for a VCO.  In Windows Server 2012 a Repair action was added for all other cluster Network Names, and is a little bit different.  Repair will check to see if the associated computer object exists in Active Directory.  If the VCO had been accidentally deleted, then using Repair will re-create the computer object if it is missing.  The recommended process to recover deleted computer objects is with the AD Recycle Bin feature, using Repair to re-create computer objects when they have been deleted should be a last resort recovery action.  This is because some applications store attributes in the computer object (namely MSMQ), and recreating a new computer object will break the application.  Repair is a safe action to perform on any SQL Server, or File Server deployment.  The CNO must have "Create Computer Objects" permissions on the OU in which it resides to recreate the VCO's.

 

 

To run Repair, the Network Name resource must be in a "Failed" or "Offline" state.   Otherwise the option will be grayed out.


Repair is only available through the Failover Cluster Manager snap-in, there is no Powershell cmdlet available to script the action.

 

If you are running Windows Server 2012 and find that you are having to repeatedly run Repair every ~30 days, ensure you have hotfix KB2838043 installed.

 

Matt Kurjanowicz
Senior Software Development Engineer
Clustering & High-Availability
Microsoft

 

 

14 Comments
Copper Contributor

Hi John - any way to query the password date in the cluster database?  I have 2 node 2016 cluster that is alerting with 1196 DNS bad key.

Microsoft

@IT1776 

 

Unfortunately no, there is not a way.  The password date is a part of the crypto checkpoint key of the network name.

 

Copper Contributor

Thanks John for replying to my question.

Copper Contributor

Hi,

 

@John Marlin : On the newer versions of Windows (2016/2019), is there a way to do a CNO repair from power-shell / command line ?

 

Thanks,

Bogdan

Microsoft

@BogdanB 

Unfortunately no. At this time, the Cluster Manager UX is the only method to accomplish this.

Copper Contributor

Thanks John, hopefully it will make it a new version / update :)

 

Found out that there is an API for managing the cluster that can be used for this (clusapi.dll).
It's not 1 line though :) - my code reached ~400...
Am posting here some instructions for those who are looking for smth like this.

 

First you have to create the computer account in the AD and set it's typical CNO properties (SPNs, DSNHostName, msDS-SupportedEncryptionTypes, KerberosEncryptionType, compoundIdentitySupported and why not, Description). Generate a new password and set it for the CNO.
Then retrieve the ObjectGuid of the new CNO from AD.
Use New/Set/Get-AdComputer and Set-ADAccountPassword for all these.
Set this ObjectGuid in the registry for the Cluster Name resource on all nodes (HKLM:\Cluster\Resources\...).

 

Then comes the fun part. Use Add-Type to import the dll and expose the API functions.
From clusapi.dll you'll need to use:
- OpenCluster
- OpenClusterNode
- OpenClusterResource
- ClusterResourceControl (with the CLUSCTL_RESOURCE_NETNAME_SET_PWD_INFO control code).
You also need to use some additional APIs for creating and handling the memory structure (a byte array) required as input for ClusterResourceControl (containing the guid and password - if you sum up the specs from documentation is a 568 byte array, but you need 680 for the function call, just fill the rest with 0s).
From kernel32.dll and msvcrt.dll use these:
- VirtualAlloc
- memset
- VirtualFree
Before calling ClusterResourceControl set the resource for the cluster name offline and bring it back online after.

 

Then do some additional cleanup / setup like granting CNO permissions on VCO and on it's DNS record.

 

If I missed something from the process, I'll appreciate your input.

 

Here are the useful links for all these:

https://docs.microsoft.com/en-us/windows/win32/api/clusapi/
https://docs.microsoft.com/en-us/previous-versions/windows/desktop/mscs/clusctl-resource-netname-set...
https://winprotocoldoc.blob.core.windows.net/productionwindowsarchives/MS-CMRP/%5BMS-CMRP%5D-190313-...
https://www.fuzzysecurity.com/tutorials/24.html
http://www.exploit-monday.com/2012/05/accessing-native-windows-api-in.html

 

Copper Contributor

Has anyone shared such a script? @BogdanB ? Can you share something?

Copper Contributor

Hello, I have a problem, my computer objects were deleted in AD.
my cluster service is online and all the functions for me to perform this procedure will my services be stopped?

and when I try to perform this procedure in a less important function, it gives me the error of incorrect username and password.

 

 

Capturar.JPG

Copper Contributor

I have an issue with 1207 errors on one of our nodes, let's call it n2. We've already verified permissions in AD for the OU and Objects. We gave the CNO full control of the OU and inherited down, but still getting the errors. Moving roles to n1 or n3, the VCO passwords are successfully updated, and the 1207 errors go away. Would performing this repair procedure fix whatever is going on with n2?

Copper Contributor

Same Issue here, two times Windows Server 2019 (build 1809) cluster  and one Windows Server 2022 (build 2009) cluster , different domains,

Error 1207 every hour; it works fine till mid september 2022. 

Patches installed since lastPwdSet @ Svr2019 kb5016623, kb2016737, kb5012170

- AD Permissions checked

- Port 464 checked

- add new roles checked

- october patched installed

- ClusterLog -> RotatePassword Exit 5 

 

Update: Oct 17 2022
- Traffic checked with Wireshark

 

I have a little hope that time is not running out till we get it fixed cause:

"Question: If a workstation does not change its password, will it not be allowed to log onto the network?
Answer: Machine account passwords as such do not expire in Active Directory. " see the link below

https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/machine-account-password-proc...

 

Update: Oct 18 2022

- Talked to Microsoft Support - there will be a patch soon. next month they said. @AaronB1160 

- disabled GPO for the cluster nodes and objects (MSFT Windows Server 2019/2022 Security Compliance Baseline) to clear the Value "Deny access to this computer from network)

- unfortunatly still errors

 

Update: Nov 17 2022

- Microsoft delivered a private patch to us, it was only for testing purpose on Server 2022 and it worked.

- They are working on a patch for public release.

 

Update: Jan 2 2023

I'v got mail from MS on dec 22, short version: please test KB5020032 (preview)

https://support.microsoft.com/de-de/topic/22-november-2022-kb5020032-betriebssystembuild-20348-1311-...

 

It was to late, the server 2022 cluster already installed the december patches the same night and the error was gone. (KB5021249)

 

@Michael Fisher 

We are still waiting for the Server 2019 solution. 

MS wrote: please install KB5021237 and KB5022554

I'll give an update asap.

 

Update: Jan 13 2023

We installed KB5022286 on a Server 2019 Clusters and it looks like the problem is solved.

 

Copper Contributor

@d1haus ; from your Nov 17, 2022, comment I begin searching the update catalog.

I was not able to find an update that specifically mentioned this issue. I am working with Server 2019 (1809), so perhaps I'm still out of luck.

Do you happen to know if I should be watching for a particular KB number?

I am having the issue with my 2019 clusters:

 

 

get-hotfix|sort hotfixID

Source        Description      HotFixID      InstalledBy          InstalledOn
------        -----------      --------      -----------          -----------
PMV-FSC01N03  Security Update  KB4535680     NT AUTHORITY\SYSTEM  2/8/2022 12:00:00 AM
PMV-FSC01N03  Update           KB4589208     NT AUTHORITY\SYSTEM  2/8/2022 12:00:00 AM
PMV-FSC01N03  Security Update  KB5005112                          8/5/2021 12:00:00 AM
PMV-FSC01N03  Update           KB5009642     NT AUTHORITY\SYSTEM  2/8/2022 12:00:00 AM
PMV-FSC01N03  Update           KB5011574     NT AUTHORITY\SYSTEM  4/26/2022 12:00:00 AM
PMV-FSC01N03  Security Update  KB5012170     NT AUTHORITY\SYSTEM  8/23/2022 12:00:00 AM
PMV-FSC01N03  Update           KB5012675     NT AUTHORITY\SYSTEM  4/26/2022 12:00:00 AM
PMV-FSC01N03  Update           KB5014031     NT AUTHORITY\SYSTEM  6/28/2022 12:00:00 AM
PMV-FSC01N03  Update           KB5014797     NT AUTHORITY\SYSTEM  6/28/2022 12:00:00 AM
PMV-FSC01N03  Update           KB5015896     NT AUTHORITY\SYSTEM  7/26/2022 12:00:00 AM
PMV-FSC01N03  Update           KB5017400     NT AUTHORITY\SYSTEM  9/27/2022 12:00:00 AM
PMV-FSC01N03  Security Update  KB5019966     NT AUTHORITY\SYSTEM  11/22/2022 12:00:00 AM
PMV-FSC01N03  Security Update  KB5020374     NT AUTHORITY\SYSTEM  11/22/2022 12:00:00 AM
PMV-FSC01N03  Update           KB5020627     NT AUTHORITY\SYSTEM  11/22/2022 12:00:00 AM

 

 

However My other 2019 Cluster is not having this issue:

 

 

get-hotfix|sort hotfixID

Source        Description      HotFixID      InstalledBy          InstalledOn
------        -----------      --------      -----------          -----------
PMP-HYPV04    Security Update  KB4535680     NT AUTHORITY\SYSTEM  6/15/2021 12:00:00 AM
PMP-HYPV04    Update           KB4589208     NT AUTHORITY\SYSTEM  6/15/2021 12:00:00 AM
PMP-HYPV04    Security Update  KB5001404                          4/9/2021 12:00:00 AM
PMP-HYPV04    Security Update  KB5003243     NT AUTHORITY\SYSTEM  6/2/2021 12:00:00 AM
PMP-HYPV04    Security Update  KB5003711     NT AUTHORITY\SYSTEM  7/6/2021 12:00:00 AM
PMP-HYPV04    Security Update  KB5005112     NT AUTHORITY\SYSTEM  9/7/2021 12:00:00 AM
PMP-HYPV04    Security Update  KB5005701     NT AUTHORITY\SYSTEM  10/5/2021 12:00:00 AM
PMP-HYPV04    Update           KB5006754     NT AUTHORITY\SYSTEM  12/7/2021 12:00:00 AM
PMP-HYPV04    Security Update  KB5008287     NT AUTHORITY\SYSTEM  1/10/2022 12:00:00 AM
PMP-HYPV04    Update           KB5009642     NT AUTHORITY\SYSTEM  3/8/2022 12:00:00 AM
PMP-HYPV04    Update           KB5011574     NT AUTHORITY\SYSTEM  4/5/2022 12:00:00 AM
PMP-HYPV04    Security Update  KB5012170     PMSI\a-michael.fi... 10/31/2022 12:00:00 AM
PMP-HYPV04    Update           KB5017270     PMSI\a-michael.fi... 10/31/2022 12:00:00 AM
PMP-HYPV04    Update           KB5017400     NT AUTHORITY\SYSTEM  10/31/2022 12:00:00 AM
PMP-HYPV04    Security Update  KB5018419     NT AUTHORITY\SYSTEM  10/31/2022 12:00:00 AM

 

 

Update 12-07-2022:

Installed windows10.0-kb5021655-x64_b3a1b7a804396f73ca22c0cddea466b7b8260617.msu - November 17, 2022—KB5021655 (OS Build 17763.3653) Out-of-band update.

Paused node (drained roles), installed update, restarted node; repeated for second node. No change, still have the same behavior.

KB5012596:April 12, 2022—KB5012596 (OS Build 14393.5066) - Microsoft Support

For Windows Server 2016 1607

Addresses an issue that logs Event ID 37 during certain password change scenarios, including failover cluster name object (CNO) or virtual computer object (VCO) password changes.

Windows Update Catalog:

2022-04 Cumulative Update for Windows Server 2016 for x64-based Systems (KB5012596) > Package Details > This update has been replaced by the following updates:

2022-08 Cumulative Update for Windows Server 2016 for x64-based Systems (KB5016622)

2022-06 Cumulative Update for Windows Server 2016 for x64-based Systems (KB5014702)

2022-11 Cumulative Update for Windows Server 2016 for x64-based Systems (KB5021654)

2022-05 Cumulative Update for Windows Server 2016 for x64-based Systems (KB5015019)

2022-10 Cumulative Update for Windows Server 2016 for x64-based Systems (KB5018411)

2022-10 Cumulative Update for Windows Server 2016 for x64-based Systems (KB5020439)

2022-11 Cumulative Update for Windows Server 2016 for x64-based Systems (KB5019964)

2022-05 Cumulative Update for Windows Server 2016 for x64-based Systems (KB5013952)

2022-09 Cumulative Update for Windows Server 2016 for x64-based Systems (KB5017305)

2022-07 Cumulative Update for Windows Server 2016 for x64-based Systems (KB5015808)

KB5013952 is replaced by KB5021654. as of 12-7-2022 KB5021654 package had no replacements.

Since these (virtual) servers had been 2016 (1607) and were updated to server 2019 (1809) (as part of the cluster migration to new data array), I checked for similar update for server 2019.

 

 

Copper Contributor

Hello :)
we have the same issue with Windows Server 2019 patchlevel 2022-11.
According to the article: https://learn.microsoft.com/en-us/troubleshoot/windows-server/identity/password-change-conflict-reso... the update of the computer password should be send to the DC holding the pdc emulator FSMO. Cluster and PDC are located in the same site.
In the debug logs of the cluster we see an other DC is used for this task.
Can someone with the same issue confirm this behaviour?
Thanks in advance and greetings from germany :)

Copper Contributor

I opened a ticket with Microsoft before the holiday season. I'll be following up this month to see if we can resolve the issue.

Copper Contributor

Repairing may fail, when "Everyone - Read" Permission is missing on CNO and/or Resource Access Point DNS Records:

We noticed that this error is raised, when we tried to "repair" the CNO of a Windows Cluster holding a SQL Server Availability Group:

 

[Main Instruction]

There was an error repairing the active directory object for 'Cluster Name'.

[Expanded Information]

The specified directory service attribute or value does not exist.

 

The CNO and SQL Server Availablity Group were pre-Staged in AD. Also DNS Host entries were pre-staged prior creating the cluster. Manually created Host entries do not have an "Everyone Read" permission per design.

As long as these DNS Host records do not have an "Everyone Read" permission the repair might fail.

 

After adding "Everyone-Read" to the DNS Record of the SQL Listener repairing worked as expected.

 

If the Resource Access Point is created by SMSS automatically the everyone permission of its DNS Record will not be set automatically. You need to add it manually after creating the listener.

 

This might be a bug and might be fix by either updating the repair function to use the authenticated user for access DNS or by adding the "everyone read" permission during the SQL Server Availability creation automatically.

 

Version history
Last update:
‎Nov 09 2023 11:09 AM
Updated by: