Exchange 2019 Failover with DAG Question

[mmazurkiewcz269]

I am trying to get an understanding of what happened to my organization a couple of weeks ago and am hoping that someone will help me clarify how Exchange 2019 failover works with a DAG.

This past summer my organization upgraded to Exchange 2019, a DAG was added and we have 2 servers that host 4 databases that my understanding is with the DAG that messages synchronize between the databases on both servers.  There is a load balancer in front of the servers as well so inbound connections will go to both servers.

Last month the servers were scheduled for updates.  Our current configuration is that all databases are active on one server and passive on the second one.  We were able to do updates to the passive database server and reboot it without any incident.  Since it was the first time doing updates, I forgot to change the active database over to the server that was just updated, so when the server hosting the active database rebooted our email system was inaccessible until the server completed its updating and reboot cycle which ended up being about an hour.

That is where I am looking for help.  It is my understanding that the DAG should have recognized that the active database server was down and should have activated the databases on the server that already had updates completed...right?  I was curious if there may have been a validation time or something happen, so I am currently in for overnight testing and manually flipped my active databases between servers and did not experience any issues, delays, hiccups or anything to indicate that there was a change.

So that brings me to trying to find out, what was it about rebooting the server that had the active databases on it that halted access to our entire email system until that server came back up?  This behavior makes me nervous for the event of a real failure and the ability for email to resume availability.

Is there anything I can do to better handle this scenario in the future or in the event of a real failure?  I now know that the next time we do maintenance that I will change the active database between servers before rebooting, but while we had our previous outage I couldn't even access the ECP URL.  I did find the Powershell commands for performing a switchover, if we experienced a failure of the server with active databases, would I be able to run the command from powershell and resume email operation from the second server?

Last, since our current configuration is that all databases are active on the same server, would it make more sense to make 2 active on each server?  My idea being that if either server experienced this problem again that it would only be half of the post office that would be down rather than the whole thing.

Sorry for the wall of text but I'm getting pressure to come up with an answer and haven't been able to find one.  Thanks!

[#2]

@mmazurkiewcz269 

Do you have a File Share Witness?

One is required if there is an even number of servers in the DAG.

[#3]

@Neill Tinlin Yes we have a witness server, its on the same system as the non-Exchange server for the DAG

[#4]

@mmazurkiewcz269 

So the FSW is on AN Other server which isn't an Exchange server, isn't a DC and is in the same AD domain?

1. Check that the domain Exchange Trusted Subsystem group is actually  in the local admins group of the FSW. This catches a lot of people out. e.g. It might be set via group policy or some sort of security scanning util might take it out as not being an approved group.

2. Run Get-DatabaseAvailabilityGroup command and make a note of the FSW directory, on the FSW check if this folder exists and that there are files in it

3. Check that there are no firewalls between any of the 3 servers, especially on port 445

4. If for some reason you are using Windows firewall on the FSW check that it is using the domain profile. I've seen cases where Network Location Awareness sets the wrong profile and thus blocks traffic.

NT

[#5]

@Neill Tinlin 

The FSW is its own dedicated server, is not one of the Exchange servers and isn't a DC, and is on the domain.

Following through the checks that you provided:

1.  It does have the Exchange Trusted Subsystem in the local admins group

2.  I had to use the web interface to get the file path from the FSW.  I found that only one of the Exchange servers have the path to the Witness directory physically on the system.

3/4. Firewall and Network settings are good

So based on this I think that my DAG isn't configured quite right.  From the web interface I can open the DAG settings and do see the Witness server in the correct field with the directory path.  Again, the directory is only present on one of the Exchange servers and not the FSW server.  The DAG members does show the 2 Exchange servers listed as well.

Where do I go from here?  I apologize for the questions, this was originally set up by a consultant months ago and the actual person is on another job so has limited availability to follow up with.