SOLVED

Cluster issues with Exchange 2016

%3CLINGO-SUB%20id%3D%22lingo-sub-2213307%22%20slang%3D%22en-US%22%3ECluster%20issues%20with%20Exchange%202016%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2213307%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWe%20recently%20rebuilt%20one%20of%20our%20Exchange%20servers%2C%20and%20have%20come%20across%20an%20issue%20with%20the%20Windows%20Failover%20Clustering%2C%20rather%20than%20the%20Exchange%20side%20of%20things.%26nbsp%3B%20Once%20the%20server%20had%20been%20rebuilt%2C%20we%20added%20that%20note%20back%20into%20the%20DAG%20via%20the%20Exchange%20console.%26nbsp%3B%20We%20then%20proceeded%20to%20re-seed%20the%20passive%20database%20copies.%26nbsp%3B%20All%20of%20that%20worked%20okay%2C%20but%20we%20get%20failures%20when%20we%20test%20the%20replication%20health.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIt%20looks%20like%20the%20process%20of%20adding%20the%20clustering%20service%2C%20but%20without%20being%20told%20it%20was%20waiting%20for%20a%20server%20restart%20to%20complete%2C%20which%20we%20didn't%20do.%26nbsp%3B%20I%20suspect%20that%20is%20the%20reason%20why%20in%20the%20Windows%20Failover%20Clustering%2C%20it%20only%20shows%20a%20single%20node.%26nbsp%3B%20When%20I%20attempt%20to%20add%20the%20newly%20built%20node%20to%20that%20cluster%2C%20it%20fails%20stating%20that%20the%20node%20is%20already%20part%20of%20the%20cluster.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ERunning%20the%20following%20command%20shows%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Ecluster%20%2Fcluster%3ADAG02%20%2Fadd%20%2Fnode%3ASERVER1%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EConfiguring%20node%20SERVER1%3CBR%20%2F%3E---------------------------------------%3CBR%20%2F%3E12%25%20Validating%20cluster%20state%20on%20node%20SERVER1.This%20phase%20encountered%20an%20error%20for%20Cluster%20object%20'Node%20wSERVER1%20appears%20to%20be%20a%20member%20of%20a%20cluster.%20It%20is%20either%20a%20member%20of%20an%20existing%20cluster%20or%20the%20node%20was%20not%20cleaned%20up%20after%20being%20evicted%20from%20a%20cluster.%20If%20you%20are%20sure%20this%20is%20not%20a%20member%20of%20a%20cluster%20run%20the%20Remove-ClusterNode%20cmdlet%20with%20the%20-Force%20parameter%20to%20clean%20up%20the%20cluster%20information%20from%20the%20node%20and%20then%20try%20to%20add%20it%20to%20the%20cluster%20again.'%20but%20will%20continue.%20The%20error%20status%20is%205065%20(0x000013C9).%3CBR%20%2F%3EThis%20phase%20has%20failed%20for%20Cluster%20object%20'SERVER1'%20with%20an%20error%20status%20of%205065%20(0x000013C9).%3CBR%20%2F%3EThis%20phase%20has%20failed%20for%20Cluster%20object%20'SERVER1'%20with%20an%20error%20status%20of%205065%20(0x000013C9).%3CBR%20%2F%3ECleaning%20up%20SERVER1.%3C%2FP%3E%3CP%3ESystem%20error%205065%20has%20occurred%20(0x000013c9).%3CBR%20%2F%3EThe%20cluster%20node%20is%20already%20a%20member%20of%20the%20cluster.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Ecluster%20node%3CBR%20%2F%3EListing%20status%20for%20all%20available%20nodes%3A%3C%2FP%3E%3CP%3ENode%20Node%20ID%20Status%3CBR%20%2F%3E--------------%20-------%20---------------------%3CBR%20%2F%3ESERVER2%202%20Up%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EChecking%20the%20database%20copy%20status%20on%20SERVER1%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EGet-MailboxDatabaseCopyStatus%20-Server%20SERVER1%3C%2FP%3E%3CP%3EName%20Status%20CopyQueue%20ReplayQueue%20LastInspectedLogTime%20ContentIndex%3CBR%20%2F%3ELength%20Length%20State%3CBR%20%2F%3E----%20------%20---------%20-----------%20--------------------%20------------%3CBR%20%2F%3EEDB%20AC%2001%5CSERVER1%20Healthy%200%200%2016%2F03%2F2021%2009%3A50%3A05%20Healthy%3CBR%20%2F%3EEDB%20DG%2001%5CSERVER1%20Healthy%200%200%2016%2F03%2F2021%2009%3A50%3A21%20Healthy%3CBR%20%2F%3EEDB%20HJ%2001%5CSERVER1%20Healthy%200%200%2016%2F03%2F2021%2009%3A49%3A47%20Healthy%3CBR%20%2F%3EEDB%20KM%2001%5CSERVER1%20Healthy%200%200%2016%2F03%2F2021%2009%3A49%3A11%20Healthy%3CBR%20%2F%3EEDB%20NR%2001%5CSERVER1%20Healthy%200%200%2016%2F03%2F2021%2009%3A47%3A09%20Healthy%3CBR%20%2F%3EEDB%20SZ%2001%5CSERVER1%20Healthy%200%200%2016%2F03%2F2021%2009%3A49%3A48%20Healthy%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAnd%20on%20SERVER2%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EGet-MailboxDatabaseCopyStatus%20-Server%20SERVER2%3C%2FP%3E%3CP%3EName%20Status%20CopyQueue%20ReplayQueue%20LastInspectedLogTime%20ContentIndex%3CBR%20%2F%3ELength%20Length%20State%3CBR%20%2F%3E----%20------%20---------%20-----------%20--------------------%20------------%3CBR%20%2F%3EEDB%20DG%2001%5CSERVER2%20Mounted%200%200%20Healthy%3CBR%20%2F%3EEDB%20AC%2001%5CSERVER2%20Mounted%200%200%20Healthy%3CBR%20%2F%3EEDB%20HJ%2001%5CSERVER2%20Mounted%200%200%20Healthy%3CBR%20%2F%3EEDB%20KM%2001%5CSERVER2%20Mounted%200%200%20Healthy%3CBR%20%2F%3EEDB%20NR%2001%5CSERVER2%20Mounted%200%200%20Healthy%3CBR%20%2F%3EEDB%20SZ%2001%5CSERVER2%20Mounted%200%200%20Healthy%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI'm%20not%20sure%20how%20to%20proceed%20here.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20don't%20know%20whether%20it%20would%20be%20safe%20to%20run%20the%20suggested%20command%2C%20%22Remove-ClusterNode%20SERVER1%20-force%22%20to%20cleanup%20the%20metadata%2C%20then%20attempt%20to%20re-join%20it%20to%20to%20failover%20cluster%2C%20without%20upsetting%20anything%20else%20on%20the%20Exchange%20side.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2213307%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3E2016%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EExchange%20Server%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2213516%22%20slang%3D%22en-US%22%3ERe%3A%20Cluster%20issues%20with%20Exchange%202016%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2213516%22%20slang%3D%22en-US%22%3E%CE%A4hank-you%20for%20Q%CF%85erying%20Microsoft%20Comm%CF%85nity%2C%20we%20are%20here%20to%20help%20with%20yo%CF%85%20problem%2C%20we%20are%20assigning%20Level%204%20Certified%20%CE%A4%E2%88%8Achnician%20to%20h%E2%88%8Alp%20with%20yo%CF%85r%20issue%20Pl%E2%88%8Aase%20caII%20on%20%CB%A6-8.8.8-%E1%97%B1.%F0%90%90%9E.%F0%90%90%9E-O.2.2.2%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2216642%22%20slang%3D%22en-US%22%3ERe%3A%20Cluster%20issues%20with%20Exchange%202016%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2216642%22%20slang%3D%22en-US%22%3EAnswer%20was%20that%20the%20for%20whatever%20reason%2C%20it%20failed%20to%20automatically%20add%20the%20node%20back%20in%20to%20the%20Failover%20Cluster.%20When%20I%20tried%20to%20manually%20add%20it%20to%20the%20Failover%20Cluster%2C%20and%20it%20stated%20the%20node%20was%20already%20a%20part%20of%20a%20node%2C%20what%20that%20really%20meant%20was%20I%20had%20manually%20changed%20the%20service%20from%20Disabled%20to%20Automatic%20in%20the%20process%20of%20trying%20to%20find%20why%20it%20wasn't%20working.%3CBR%20%2F%3E%3CBR%20%2F%3ESwitching%20the%20service%20back%20to%20Disabled%2C%20then%20attempting%20to%20add%20the%20node%20to%20the%20cluster%20fixed%20the%20issue.%3CBR%20%2F%3E%3CBR%20%2F%3ENow%20both%20DAG%2C%20and%20Failover%20Cluster%20are%20reporting%20as%20Healthy.%3C%2FLINGO-BODY%3E
New Contributor

Hi,

 

We recently rebuilt one of our Exchange servers, and have come across an issue with the Windows Failover Clustering, rather than the Exchange side of things.  Once the server had been rebuilt, we added that note back into the DAG via the Exchange console.  We then proceeded to re-seed the passive database copies.  All of that worked okay, but we get failures when we test the replication health.

 

It looks like the process of adding the clustering service, but without being told it was waiting for a server restart to complete, which we didn't do.  I suspect that is the reason why in the Windows Failover Clustering, it only shows a single node.  When I attempt to add the newly built node to that cluster, it fails stating that the node is already part of the cluster.

 

Running the following command shows:

 

cluster /cluster:DAG02 /add /node:SERVER1

 

Configuring node SERVER1
---------------------------------------
12% Validating cluster state on node SERVER1.This phase encountered an error for Cluster object 'Node SERVER1 appears to be a member of a cluster. It is either a member of an existing cluster or the node was not cleaned up after being evicted from a cluster. If you are sure this is not a member of a cluster run the Remove-ClusterNode cmdlet with the -Force parameter to clean up the cluster information from the node and then try to add it to the cluster again.' but will continue. The error status is 5065 (0x000013C9).
This phase has failed for Cluster object 'SERVER1' with an error status of 5065 (0x000013C9).
This phase has failed for Cluster object 'SERVER1' with an error status of 5065 (0x000013C9).
Cleaning up SERVER1.

System error 5065 has occurred (0x000013c9).
The cluster node is already a member of the cluster.

 

cluster node
Listing status for all available nodes:

Node Node ID Status
-------------- ------- ---------------------
SERVER2 2 Up

 

Checking the database copy status on SERVER1:

 

Get-MailboxDatabaseCopyStatus -Server SERVER1

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex
Length Length State
---- ------ --------- ----------- -------------------- ------------
EDB AC 01\SERVER1 Healthy 0 0 16/03/2021 09:50:05 Healthy
EDB DG 01\SERVER1 Healthy 0 0 16/03/2021 09:50:21 Healthy
EDB HJ 01\SERVER1 Healthy 0 0 16/03/2021 09:49:47 Healthy
EDB KM 01\SERVER1 Healthy 0 0 16/03/2021 09:49:11 Healthy
EDB NR 01\SERVER1 Healthy 0 0 16/03/2021 09:47:09 Healthy
EDB SZ 01\SERVER1 Healthy 0 0 16/03/2021 09:49:48 Healthy

 

And on SERVER2:

 

Get-MailboxDatabaseCopyStatus -Server SERVER2

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex
Length Length State
---- ------ --------- ----------- -------------------- ------------
EDB DG 01\SERVER2 Mounted 0 0 Healthy
EDB AC 01\SERVER2 Mounted 0 0 Healthy
EDB HJ 01\SERVER2 Mounted 0 0 Healthy
EDB KM 01\SERVER2 Mounted 0 0 Healthy
EDB NR 01\SERVER2 Mounted 0 0 Healthy
EDB SZ 01\SERVER2 Mounted 0 0 Healthy

 

I'm not sure how to proceed here.

 

I don't know whether it would be safe to run the suggested command, "Remove-ClusterNode SERVER1 -force" to cleanup the metadata, then attempt to re-join it to to failover cluster, without upsetting anything else on the Exchange side.

1 Reply
best response confirmed by HowardGyton (New Contributor)
Solution
Answer was that the for whatever reason, it failed to automatically add the node back in to the Failover Cluster. When I tried to manually add it to the Failover Cluster, and it stated the node was already a part of a node, what that really meant was I had manually changed the service from Disabled to Automatic in the process of trying to find why it wasn't working.

Switching the service back to Disabled, then attempting to add the node to the cluster fixed the issue.

Now both DAG, and Failover Cluster are reporting as Healthy.