Blog Post

Exchange Team Blog
3 MIN READ

Responding to Managed Availability

The_Exchange_Team's avatar
Dec 16, 2013

 

I’ve written a few blog posts now that get into the deep technical details of Managed Availability. I hope you’ve liked them, and I’m not about to stop! However, I’ve gotten a lot of feedback that we also need some simpler overview articles. Fortunately, we’ve just completed documentation on TechNet with an overview of Managed Availability. This was written to address how the feature may be managed day-to-day.

Even that documentation doesn’t address how you respond when Managed Availability cannot resolve a problem on its own. This is the very most common interaction with Managed Availability, but we haven’t described how specifically to do so.

When Managed Availability is unable to recover the health of a server, it logs an event. Exchange Server has a long history of logging warning, error, and critical events into various channels when things go wrong. However, there are two things about Managed Availability events that make them more generally useful than our other error events:

  • They all go to the same place on a server without any clutter
  • They will only be logged when the standard recovery actions fail to restore health of the component

When one of these events is logged on any server in our datacenters, a member of the product group team responsible for that health set gets an immediate phone call.

No one likes to wake up at 2 AM to investigate and fix a problem with a server. This keeps us motivated to only have Managed Availability alerts for problems that really matter, and also to eliminate the cause of the alert by fixing underlying code bugs or automating the recovery. At the same time, there is nothing worse than finding out about incidents from customer calls to support. Every time that happens we have painful meetings about how we should have detected the condition first and woken someone up. These two conflicting forces strongly motivate the entire engineering team to keep these events accurate and useful.

The GUI

Along with a phone call, the on-call engineer receives an email with some information about the failure. The contents of this email are pulled from the event’s description.

The path in Event Viewer for these events is Microsoft-Exchange-ManagedAvailability/Monitoring. Error event 4 means that a health set has failed and gives the details of the monitor that has detected the failure. Information event 1 means that all monitors of a health set have become healthy.

The Exchange 2013 Management Pack for System Center Operations Manager nicely shows only the health sets that are currently failed instead of the Event Viewer’s method of displaying all health sets that have ever failed. SCOM will also roll up health sets into four primary health groups or three views.

The Shell

This wouldn’t be EHLO without some in-depth PowerShell scripts. The event viewer is nice and SCOM is great, but not everyone has SCOM. It would be pretty sweet to get the same behavior as SCOM to show only the health sets on a server that are currently failed.

Note: these logs serve a slightly different purpose than Get-HealthReport. Get-HealthReport shows the current health state of all of a server’s monitors. On the other hand, events are only logged in this channel once all the recovery actions for that monitor have been exhausted without fixing the problem. Also know that these events detail the failure. If you’re only going to take action based on one health metric, the events in this log is a better one. Get-HealthReport is still the best tool to show you the up-to-the-minute user experience.

We have a sample script that can help you with this; it is commented in a way that you can see what we were trying to accomplish. You can get the Get-ManagedAvailabilityAlerts.ps1 script as an attachment to this blog post.

Either this method or Event Viewer will work pretty well for a handful of servers. If you have tens or hundreds of servers, we really recommend investing in SCOM or another robust and scalable event-collection system.

My other posts have dug deeply into troubleshooting difficult problems, and how Managed Availability gives an overwhelmingly immense amount of information about a server’s health. We rarely need to use these troubleshooting methods when running our datacenters. However, the only thing you need to resolve Exchange problems the way we do in Office 365 is a little bit of event viewer or scheduled script.

Abram Jackson
Program Manager, Exchange Server

Updated Apr 27, 2020
Version 3.0
  • Hi AJ, Thanks for the blog. I got two queries. Please help me with answers.

    #1. Is there anyway, we can find what the probe does to evaluate the condition/status of exchange . I looked into Probe definition(Event Viewer- XML), I could not get the complete detail. I would like to know , what all the test done by probe. Is it possible? This will help to investigate the issue and avoid the issue from coming back and exchange MA reacting to hit.

    #2. Can you please help me with a way to find the list of responders do the server forcereboot/reboot? So far I feel when the responder name ends with reboot, the responder does the reboot. Is that any other way to find more accurate result.

    Thank you for your reply in advance.

  • @KSK2000, I'm planning on writing my next two posts on probes and responders. In short, there is not a general way to find out what most probes do, though some TypeNames follow a convention. You can however tell the type of a responder as there are much fewer types. Force reboot responders will be of one of two types, (...).DagForceRebootServerResponder or (...).ForceRebootServerResponder. You can search for these strings in the ResponderDefinition channel of eventvwr or using the PowerShell method I've detailed in other posts.
  • Good to see an article about MA with pointers for customers without SCOM. Thanks!
  • Good to see an article about MA with pointers for customers without SCOM. Thanks!
  • Great article for us Exchange On-Premises admins/engineers
  • now lets see how much time it takes for someone open source to write a gui app for this:)
  • $EventHash = @{} $FilterHash = @{ LogName= 'Microsoft-Exchange-ManagedAvailability/Monitoring' Level= 2,4 StartTime=$Start } Get-WinEvent -FilterHashTable $FilterHash -ComputerName | foreach { if ( $EventHash.[$_.Properties[0].Value] -eq $null ) { if (

    $_.LevelDisplayName -eq 'Error' ) { $EventHash[$_.Properties[0].Value] = $_.Message } else { $EventHash[$_.Properties[0].Value] = 'Resolved' } } } $EventHash.GetEnumerator() | where { $_.Value -ne 'Resolved' }