Announcing Hybrid Modern Authentication for Exchange On-Premises
We’re very happy to announce support for Hybrid Modern Authentication (HMA) with the next set of cumulative updates (CU) for Exchange 2013 and Exchange 2016, that’s CU8 for Exchange Server 2016, and CU19 for Exchange Server 2013. What is HMA? HMA (not HAM, which Word keeps trying to correct it to for me) provides users the ability to access on-premises application using authorization tokens obtained from the cloud. For Exchange (that’s why you’re here right?), this means on-premises mailbox users get the ability to use these tokens (OAuth tokens specifically) for authentication to on-premises Exchange. Sounds thrilling I know, but what exactly are these tokens? And how do users get hold of them? Rather than repeat many things here, I’m going to suggest you take a break and read the How Hybrid Authentication Really Works post, and if you really want to help boost my YouTube viewing numbers, watch this Ignite session recording too. They will respectively give you a pretty solid grounding in OAuth concepts and to help you understand what HMA is really all about. See how much space we saved in this post by sending you somewhere else? If you ignored my advice, the tl’dr version is this: HMA enables Outlook to obtain Access and Refresh OAuth tokens from Azure AD (either directly for password hash sync or Pass-Through Auth identities, or from their own STS for federated identities) and Exchange on-premises will accept them and provide mailbox access. How users get those tokens, what they have to provide for credentials, is entirely up to you and the capabilities of the identity provider (iDP) – it could be simple username and password, or certificates, or phone auth, or fingerprints, blood, eyeball scanning, the ability to recite poetry, whatever your iDP can do. Note that the user’s identity has to be present in AAD for this to work, and there is some configuration required that the Exchange Hybrid Configuration Wizard does for us. That’s why we put the H in HMA, you need to be configured Hybrid with Exchange Online for this feature. It’s also worth knowing that HMA shares many of the same technology as the upcoming Outlook mobile support for Exchange on-premises with Microsoft Enterprise Mobility + Security feature, which as you’ll see from the blog post also requires Hybrid be in place. Once you have that figured out you’ll be able to benefit from both these features with very little additional work. How Does HMA Work? The video linked above goes into detail, but I’ll share some details here for anyone without the time to watch it. Here’s a diagram that explains HMA when the identity is federated. I think that picture is pretty clear, I spent a lot of time making it pretty clear so I don’t think I need to add much to it other than to say, if it’s not clear, you might want to try reading it again. Why Should I Enable HMA? Great question. There are a few good reasons, but mainly this is a security thing. HMA should be considered ‘more secure’ than the authentication methods previously available in Exchange. That’s a nebulous statement if there ever was one (I could have said it’s more ‘Modern’ but I know you weren’t going to fall for that) but there are a few good arguments as to why that’s true. When you enable HMA you are essentially outsourcing user authentication to your iDP, Exchange becomes the consumer of the resulting authorization tokens. You can enforce whatever authentication the iDP can do, rather than teach Exchange how to handle things like text messaged based MFA, blood analysis or retina scanning. If your iDP can do that, Exchange can consume the result. Exchange doesn’t care how you authenticated, only that you did, and came away with a token it can consume. So it’s clearly ‘more secure’ if you choose to enforce authentication types or requirements stronger than those that come free with Exchange, but even if you stick to usernames and passwords it’s also more secure as passwords are no longer being sent from client to server once the user is authenticated (though of course that depends on whether you are using Basic, NTLM or Kerberos). It’s all token based, the tokens have specific lifetimes, and are for specific applications and endpoints. One other interesting and important benefit to all this is that your auth flow is now exactly the same for both your cloud and on-premises users. Any MFA or Conditional Access policies you have configured are applied the same, regardless of the mailbox location. It’s simpler to stay secure. HMA also results in an improved user experience as there will be less authentication prompts. Once the user logs in once to AAD they can access any app that uses AAD tokens – that’s anything in O365 and even Skype for Business on-premises configured for HMA (read more about Skype for Business’s HMA support here). And don’t forget there’s the fact it’s more ‘Modern’. It’s newer and we put the word Modern on it. So it must be better, or at the very least, newer. Excellent, moving on. Will It Cost Me? Not if you just want to use free Azure ID’s or Federated identities and do MFA at your iDP. If you want to take advantage of advanced Azure features, then yes, you’ll have to pay for those. But to set this up the tenant admin needs only an Exchange and an Azure license assigned, to run the tools and enable the config. What do I need to enable HMA? There are some pre-requisites. The following Identity configurations with AAD are supported Federated Identity with AAD with any on-premises STS supported by Office 365 Password Hash Synchronization Pass Through Authentication In all cases, the entire on-premises directory must be synchronized to AAD, and all domains used for logon must be included in the sync configuration. Exchange Server All servers must be Exchange 2013 (CU19+) and/or Exchange 2016 (CU8+) No Exchange 2010 in the environment MAPI over HTTP enabled. It is usually enabled or True for new installs of Exchange 2013 Service Pack 1 and above. OAuth must be enabled on all Virtual Directories used by Outlook (/AutoDiscover, /EWS, /Mapi, /OAB) In the event your environment utilizes a proxy server infrastructure to allow servers to connect to the Internet, be sure all Exchange servers have the proxy server defined in the InternetWebProxy property. You must use clients that support ADAL (the client-side library that allows the client to work with OAuth tokens) to use the Modern Auth enabled features. Outlook 2013 requires the EnableADAL registry key be set, Outlook 2016 has this key set by default, Outlook 2016 for Mac works as it is, support for Outlook mobile (iOS and Android) is coming. Ensure AAD Connect between on-premises AD and the O365 tenant has the “Exchange hybrid deployment” setting enabled in the Optional Features settings of Azure AD Connect. Ensure SSL offloading is not being used between the load balancer and Exchange servers. Ensure all user networks can reach AAD efficiently. Let’s pick a few of those apart. No Exchange 2010 in the environment. That’s right, if you have E2010 you can’t enable HMA. Why? Because worst case is everyone with a mailbox on E2010 will be cut off from email. You don’t want that. It’s because OAuth happens anonymously upon initial connection. We send the user to AAD to get authenticated before we know where their mailbox is – and if that mailbox is on E2010, when they return with a token we’ll refuse to proxy from E2013/16 to E2010. Game over. Please insert coins. So we have drawn a line here and are stating no support for E2010, and the HCW won’t let you enable OAuth if E2010 exists. Don’t try and make it work, remember that scene from Ghostbusters, the whole crossing the streams thing? It’ll be like that, but worse. Next, MAPI/HTTP – you need to be using MAPI/HTTP not RPC/HTTP (Outlook Anywhere). This feature only works with MAPI/HTTP, and anyway, it’s time to get off RPC/HTTP. That’s very old code and as you might know we ended support for its use in O365, so it would be good to switch. It just works. Then there’s the ‘everyone should be in AAD’ thing. That’s because when you enable HMA, it’s Org wide. It affects every user connecting to Exchange. So, all users trying to access Exchange from a client that support Modern Auth will be sent to AAD. If you only have some users represented in AAD, only those users will be able to auth. The rest will come find you at lunch and make your life a misery. Unless you like misery, I wouldn’t recommend that route. Needing clients that support Modern Auth clearly, makes sense. And you need to make sure all the Exchange VDirs have OAuth enabled on them. Sounds obvious, and they are enabled by default, but some admins like to tinker… so it’s worth checking, and I’ll explain how later. SSL offloading works by terminating the SSL/TLS encryption on the load balancer and transmitting the request as HTTP. In the context of OAuth, using SSL offloading has implications because if the audience claim value specifies a HTTPS record, then when Exchange receives the decrypted request over HTTP, the request is considered not valid. By removing SSL offloading, Exchange will not fail the OAuth session due to a change in the audience claim value. Lastly, the ensuring all user networks can reach AAD comment. This change affects all connectivity from supported clients to Exchange, internal and external. When a user tries to connect to Exchange, whether that server is 10 feet away under the new guys desk or in a datacenter on the other side of the planet the HMA flow will kick in. If the user doesn’t have a valid token the traffic will include a trip to AAD. If you are one of those customers with complex networking in place, consider that. How do I Enable HMA? You’ve checked the pre-reqs, and you think you’re good to go. You can do a lot of this up front without impacting clients, I’ll point out where clients begin to see changes, so you can be prepared. We do recommend trying HMA in your test or lab environment if you can before doing it in production. You are changing auth, it’s something you need to be careful doing, as cutting everyone off from email is never a good thing. Here’s what to do. First, we have some Azure Active Directory Configuration to do. You need to register all the URL’s a client might use to connect to on-premises Exchange in AAD, so that AAD can issue tokens for those endpoints. This includes all internal and external namespaces, as AAD will become the default auth method for all connections, internal and external. Here’s a tip – look at the SSL certificates you have on Exchange and make sure all those names are considered for inclusion. Run the following cmdlets to gather the URL’s you need to add/verify are in AAD. Get-MapiVirtualDirectory | FL server,*url* Get-WebServicesVirtualDirectory | FL server,*url* Get-OABVirtualDirectory | FL server,*url*> Now you need to ensure all URL’s clients may connect to are listed as https service principal names (SPN’s): Connect to your AAD tenant using these instructions. For Exchange-related URL’s, execute the following command (note the AppId ends …02): Get-MsolServicePrincipal -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 | select -ExpandProperty ServicePrincipalNames The output will look similar to the following: [PS] C:\WINDOWS\system32> Get-MsolServicePrincipal -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 | select -ExpandProperty ServicePrincipalNames https://autodiscover.contoso.com/ https://mail.contoso.com/ 00000002-0000-0ff1-ce00-000000000000/*.outlook.com 00000002-0000-0ff1-ce00-000000000000/outlook.com 00000002-0000-0ff1-ce00-000000000000/mail.office365.com 00000002-0000-0ff1-ce00-000000000000/outlook.office365.com 00000002-0000-0ff1-ce00-000000000000/contoso.com 00000002-0000-0ff1-ce00-000000000000/autodiscover.contoso.com 00000002-0000-0ff1-ce00-000000000000/contoso.mail.onmicrosoft.com 00000002-0000-0ff1-ce00-000000000000/autodiscover.contoso.mail.onmicrosoft.com 00000002-0000-0ff1-ce00-000000000000/mail.contoso.com 00000002-0000-0ff1-ce00-000000000000 If you do not already have your internal and external MAPI/HTTP, EWS, OAB and AutoDiscover https records listed (i.e., https://mail.contoso.com and https://mail.corp.contoso.com), add them using the following command (replacing the fully qualified domain names with the correct namespaces and/or deleting the appropriate addition line if one of the records already exists): $x= Get-MsolServicePrincipal -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 $x.ServicePrincipalnames.Add("https://mail.corp.contoso.com/") $x.ServicePrincipalnames.Add("https://owa.contoso.com/") Set-MSOLServicePrincipal -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 -ServicePrincipalNames $x.ServicePrincipalNames Repeat step 2 and verify the records were added. We’re looking for https://namespace entries for all the URL’s, not 00000002-0000-0ff1-ce00-000000000000/namespace entries. For example, [PS] C:\WINDOWS\system32> Get-MsolServicePrincipal -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 | select -ExpandProperty ServicePrincipalNames https://autodiscover.contoso.com/ https://mail.contoso.com/ https://mail.corp.contoso.com/ https://owa.contoso.com/ 00000002-0000-0ff1-ce00-000000000000/*.outlook.com 00000002-0000-0ff1-ce00-000000000000/outlook.com 00000002-0000-0ff1-ce00-000000000000/mail.office365.com 00000002-0000-0ff1-ce00-000000000000/outlook.office365.com 00000002-0000-0ff1-ce00-000000000000/contoso.com 00000002-0000-0ff1-ce00-000000000000/autodiscover.contoso.com 00000002-0000-0ff1-ce00-000000000000/contoso.mail.onmicrosoft.com 00000002-0000-0ff1-ce00-000000000000/autodiscover.contoso.mail.onmicrosoft.com 00000002-0000-0ff1-ce00-000000000000/mail.contoso.com 00000002-0000-0ff1-ce00-000000000000 Then we need to validate the EvoSts authentication provider is present using the Exchange Management Shell (this is created by the Hybrid Configuration Wizard): Get-AuthServer | where {$_.Name -eq "EvoSts"} If it is not present, please download and execute the latest version of the Hybrid Configuration Wizard. Note that this authentication provider is not created if Exchange 2010 (this includes Edge Transport servers) is detected in the environment. Now let’s make sure OAuth is properly enabled in Exchange on all the right virtual directories Outlook might use. Run the following cmdlets (and a tip, don’t use -ADPropertiesOnly as that sometimes tells little white lies, try it and see if you don’t believe me) Get-MapiVirtualDirectory | FL server,*url*,*auth* Get-WebServicesVirtualDirectory | FL server,*url*,*oauth* Get-OABVirtualDirectory | FL server,*url*,*oauth* Get-AutoDiscoverVirtualDirectory | FL server,*oauth* You are looking to make sure OAuth is enabled on each of these VDirs, it will look something like this (and the key things to look at are highlighted); [PS] C:\Windows\system32>Get-MapiVirtualDirectory | fl server,*url*,*auth* Server : EX1 InternalUrl : https://mail.contoso.com/mapi ExternalUrl : https://mail.contoso.com/mapi IISAuthenticationMethods : {Ntlm, OAuth, Negotiate} InternalAuthenticationMethods : {Ntlm, OAuth, Negotiate} ExternalAuthenticationMethods : {Ntlm, OAuth, Negotiate} [PS] C:\Windows\system32> Get-WebServicesVirtualDirectory | fl server,*url*,*auth* Server : EX1 InternalNLBBypassUrl : InternalUrl : https://mail.contoso.com/EWS/Exchange.asmx ExternalUrl : https://mail.contoso.com/EWS/Exchange.asmx CertificateAuthentication : InternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity, OAuth} ExternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity, OAuth} LiveIdNegotiateAuthentication : WSSecurityAuthentication : True LiveIdBasicAuthentication : False BasicAuthentication : False DigestAuthentication : False WindowsAuthentication : True OAuthAuthentication : True AdfsAuthentication : False [PS] C:\Windows\system32> Get-OabVirtualDirectory | fl server,*url*,*auth* Server : EX1 InternalUrl : https://mail.contoso.com/OAB ExternalUrl : https://mail.contoso.com/OAB BasicAuthentication : False WindowsAuthentication : True OAuthAuthentication : True InternalAuthenticationMethods : {WindowsIntegrated, OAuth} ExternalAuthenticationMethods : {WindowsIntegrated, OAuth} [PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory | fl server,*auth* Server : EX1 InternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated, WSSecurity, OAuth} ExternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated, WSSecurity, OAuth} LiveIdNegotiateAuthentication : False WSSecurityAuthentication : True LiveIdBasicAuthentication : False BasicAuthentication : True DigestAuthentication : False WindowsAuthentication : True OAuthAuthentication : True AdfsAuthentication : False Once you have checked these over, you might need to add OAuth here and there. It’s important to make sure all the servers are consistent, there’s really nothing harder to troubleshoot than when one server out of ten is wrong… (Top Nerd Note: I hope you know why we didn’t include *url* in the Get-AutodiscoverVirtualDirectory cmdlet? Answers in the comments section if you do. There are no prizes to be won!) If you need to add an Auth method, here’s a tip. For all except /Mapi, just set the -OAuthAuthentication property to $True. Done. But for /Mapi you need add it explicitly, and not using some fancy @Add PowerShell thing you learned in some online course or from that smart guy in the office who tells everyone he doesn’t use ECP as it’s for kids and dogs. Because I've learned too that sometimes that doesn’t always work the way it should. If you needed to add OAuth to all the Mapi Vdirs in the org, do it like this; Get-MapiVirtualDirectory | Set-MapiVirtualDirectory -IISAuthenticationMethods Ntlm, OAuth, Negotiate Up to this point no clients should have been impacted (unless you messed the Vdir auth up, and if you did, you should only have been adding OAuth, not taking others away…you know that now don’t you). So next we start to impact clients – so this is the bit you want to do out of normal business hours. For career reasons. So, make sure you validate the following: Make sure you have completed the steps above in the Azure AD Configuration section. All the SPN’s you need should be in there. Make sure OAuth is enabled on all virtual directories used by Outlook. Make sure your clients are up to date and HMA capable by validating you have the minimal version as defined in our supportability requirements. Make sure you have communicated what you are doing. Set the EvoSts authentication provider as the default provider (this step affects Outlook 2016 for Mac and native EAS clients that support OAuth right away): Set-AuthServer EvoSTS -IsDefaultAuthorizationEndpoint $true Enable the OAuth client feature for Windows Outlook: Set-OrganizationConfig -OAuth2ClientProfileEnabled $True That’s it. All the prep you did means it comes down to two cmdlets. Wield the power wisely. How do I Know I’m Using HMA? After HMA is enabled, the next time a client needs to authenticate it will use the new auth flow. Just turning on HMA may not immediately trigger a re-auth for any client. To test that HMA is working after you have enabled it, restart Outlook. The client should switch to use the Modern Auth flow. You should see an ADAL generated auth dialog, from Office 365. Once you enter the username you might be redirected to your on-premises IDP, like ADFS (and might not see anything at all if Integrated auth is configured), or you might need to enter a password. You might have to do MFA, it depends on how much stuff you’ve set up in AAD already. Once you get connected (and I hope you do), check Outlook’s Connection Status dialog (Ctrl-Right Click the Outlook tray icon) you will see the word Bearer in the Authn column – which is the sign that it’s using HMA. Well done you. Check everyone else is ok before heading home though, eh? Something Went Wrong. How do I Troubleshoot HMA? Ah, you’re reading this section. It’s panic time, right? I was thinking of not publishing this section until next year, just for giggles. Mine, not yours. But I didn’t. Here’s what to think about if stuff isn’t working like I said it would. Firstly, make sure you did ALL the steps above, not some, not just the ones you understood. We’ve all seen it, 10 steps to make something work, and someone picks the steps they do like it’s a buffet. If you’re sure you’ve done them all, let’s troubleshoot this together. If you need to simply turn this back off then just run the last two cmdlets we ran again, but setting them to False this time. You might need to run IISReset on Exchange more than once, we cache settings all over the place for performance reasons, but those two will put you back to where you were if all hope is lost (hopefully you still have a chance to capture a trace as detailed in a moment before you do this, as it will help identity what went wrong). If you aren’t reverting the settings just yet, you clearly want to troubleshoot this a bit. First thing is – is the client seeing any kind of pop up warning dialog? Are they seeing any certificate errors? Trust or name mismatches, that sort of thing? Anything like that will stop this flow in its tracks. The clients don’t need anything more than trusting the endpoints they need to talk to – Exchange, AAD (login.windows.net and login.microsoftonline.com) and ADFS or your iDP of choice if in use. If they trust the issuer of the certs securing those sites, great. If you have some kind of name translation thing going on somewhere, that might cause a warning, or worse, a silent failure. Here’s an example of this I saw recently. Exchange was published using Web Application Proxy (WAP). You can do that, but only in pass-through mode. The publishing rule for AutoDiscover in this case was using autodiscover.contoso.com to the outside world, but the WAP publishing rule was set up to forward that traffic to mail.contoso.com on the inside. That causes this to fail, as Outlook heads to AAD to get a token for the resource called https://autodiscover.contoso.com and it does. Then it hands that to WAP, who then forwards to Exchange using the https://mail.contoso.com target URI – the uri used in the token isn’t equal to the uri used by WAP… kaboom. So, don’t do that. But I’ll show you later how an error like that shows up and can be discovered. Assuming certificates are good, we need to get deeper. We need to trace the traffic. The tool I prefer to use for this is Fiddler, but there are others out there that can be used. Now, Fiddler or the like can capture everything that happens between client and server – and I mean everything. If you are doing Basic auth, Fiddler will capture those creds. So, don’t run a Fiddler trace capturing everything going on and share it with your buddies or Microsoft. We don’t want your password. Use a test account or learn enough about Fiddler to delete the passwords. I’ll leave it to the Telerik people who create Fiddler to tell you how to install and really use their tool, but I’ll share these few snippets I’ve learned, and how I use it to debug HMA. Once installed and with the Fiddler root certs in the trusted root store (Fiddler acts as a man-in-the-middle proxy) it will capture traffic from whatever clients you choose. You need to enable HTTPS decryption (Tools, Options, HTTPS), as all our traffic is encased in TLS. If you have ADFS you can either choose to configure Fiddler to Skip Decryption for the ADFS url, if you don’t want to see what happens at ADFS, but if you do, you will have to relax the security stance of ADFS a bit to allow the traffic to be properly captured. Only do this while capturing the traffic for debug purposes, then reset it back. Start with bypassing decryption for the iDP first, come back to this if you suspect that is the issue. To set level of extended protection for authentication supported by the federation server to none (off) Set-AdfsProperties -extendedprotectiontokencheck none Then to set it back to the default once you have the capture: Set-AdfsProperties -extendedprotectiontokencheck Allow Read more about all that clever ADFSstuff here. Now you run the capture. Start Fiddler first, then start Outlook. I suggest closing all other apps and browsers, so as not to muddy the Fiddling waters. Keep an eye on Fiddler and Outlook, try and log in using Outlook, or repro the issue, then stop tracing (F12). Now we shall try to figure out what’s going on. I prefer the view where I have the traffic listed in the left hand pane, then on the right the top section is the request, and hte lower right in the response. But you do whatever works for you. But Fiddler shows each frame, then splits each into the Request, and the Response. That’s how you need to orient yourself. So the flow you’ll see will be something like this; Client connects to Exchange, sending an empty ‘Bearer‘ header. This is the hint to tell Exchange it can do OAuth but does not yet have a token. If it sends Bearer and a string of gobbledygook, that’s your token. Here are two examples of this. The header section to look at is Security. This is using Fiddler’s Header view. Do you see how the Security header says just Bearer on the left, but shows Bearer + Token on the right. Exchange responds with (lower pane of the same packet in Fiddler, raw view), here’s where you can get a token (link to AAD). If you scroll all the way to the right you’ll see the authorization_uri (AAD) Normally, Outlook goes to that location, does Auth, gets a token, comes back to Exchange, and then tries to connect using Bearer + Token as above. If it’s accepted, it’s 200’s and beers all round and we’re done. Where could it go wrong? Client Failure Firstly, the client doesn’t send the empty Bearer header. That means isn’t even trying to do Bearer. This could be a few things. It could be that you are testing with Outlook 2010 which doesn’t support Bearer (so stop trying and upgrade). Maybe you are using Outlook 2013 but forgot to set the EnableADAL reg keys set? See the link below for those. But what if this is Outlook 2016, which has EnableADAL set by default and it is still not sending the Header…. Huh? Most likely cause, someone has been tinkering around in the registry or with GPO’s to set registry keys. I knew a guy who edited the registry once and three days later crashed his car. So, do not tell me you were not warned. You need to make sure keys are set as per https://support.office.com/en-us/article/Enable-Modern-Authentication-for-Office-2013-on-Windows-devices-7dc1c01a-090f-4971-9677-f1b192d6c910 Outlook2016 for Mac can also have MA disabled (though it’s enabled by default). You can set it back to the default by running this from Terminal: defaults write com.microsoft.Outlook DisableModernAuth -bool NO That’s how we deal with the client not sending the Header. Check again and see the Header in all its Header glory. Auth_URI Failures Next thing that might happen is the server doesn’t respond with the authorization-uri, or it’s the wrong one. If there’s no authorization_uri at all then the EvoSts AuthServer does not have IsDefaultAuthorizationEndpoint set to $true. Recheck you ran Set-AuthServer EvoSts -IsDefaultAuthorizationEndpoint $true If it comes back, but with some other value than expected, make sure the right AuthServer is set as default, we only support you using AAD for this flow. If you think setting this to your on-premises ADFS endpoint will make this work without AAD… you’re wrong, as you discovered when you tried. If you are thinking of trying it, don’t bother. That’s an Exchange 2019 thing. Oh, did I just let that out of the bag? If HMA is enabled at the org level, but connections still don’t elicit the authorization_uri you expect it’s likely OAuth isn’t enabled on the Virtual Directory Outlook is trying to connect to. You need to simply make sure you have OAuth enabled on all VDirs, on all servers. Go back to the How Do I Enable section and check those VDirs again. Now, sometimes that all comes back ok but the client still doesn’t take the bait. If so, check for the following in the response; HTTP/1.1 401 Unauthorized Content-Length: 0 Server: Microsoft-IIS/8.5 Microsoft-HTTPAPI/2.0 request-id: a8e9dfb4-cb06-4b18-80a0-b110220177e1 Www-Authenticate: Negotiate Www-Authenticate: NTLM Www-Authenticate: Basic realm="autodiscover.contoso.com" X-FEServer: CONTOSOEX16 x-ms-diagnostics: 4000000;reason="Flighting is not enabled for domain 'gregt@contoso.com'.";error_category="oauth_not_available" X-Powered-By: ASP.NET WWW-Authenticate: Bearer client_id="00000002-0000-0ff1-ce00-000000000000", trusted_issuers="00000001-0000-0000-c000-000000000000@f31f3647-5d87-4b69-a0b6-73f62aeab14c", token_types="app_asserted_user_v1 service_asserted_app_v1", authorization_uri="https://login.windows.net/common/oauth2/authorize" Date: Thu, 13 Jul 2017 18:22:13 GMT Proxy-Support: Session-Based-Authentication Now this response is interesting because it says, go get a token (www-authenticate), but in x-ms-diagnostics it says, no, don’t. Is Exchange unsure? This means OAuth is enabled, but not for Outlook for Windows. So, you ran one of the two commands above (or you ran them both but not enough time has passed for them to kick in) Verify that the OAuth2ClientProfileEnabled property is set to $true by checking; (Get-OrganizationConfig).OAuth2ClientProfileEnabled Other Failures We have a token, we know OAuth is enabled at the Org level in Exchange, we know all the Vdirs are good. But it still won’t connect. Dang, what now? Now you’ll have to start to dig into server responses more closely, and start looking for things that look like errors. The errors you’ll see are usually in plain English, though of course that doesn’t mean they make sense. But here are some examples. Missing SPNs Client goes to AAD to get a token and get this: Location: urn:ietf:wg:oauth:2.0:oob?error=invalid_resource&error_description=AADSTS50001%3a+The+application+named+https%3a%2f%2fmail.contoso.com%2f+was+not+found+in+the+tenant+named+contoso.com.++This+can+happen+if+the+application+has+not+been+installed+by+the+administrator+of+the+tenant+or+consented+to+by+any+user+in+the+tenant.++You+might+have+sent+your+authentication+request+to+the+wrong+tenant.%0d%0aTrace+ID%3a+cf03a6bd-610b-47d5-bf0b-90e59d0e0100%0d%0aCorrelation+ID%3a+87a777b4-fb7b-4d22-a82b-b97fcc2c67d4%0d%0aTimestamp%3a+2017-11-17+23%3a31%3a02Z Name Mismatches Here’s one I mentioned earlier. There’s some device between client and server changing the names being used. Tokens are issued for specific uri’s, so when you change the names… HTTP/1.1 401 Unauthorized Content-Length: 0 WWW-Authenticate: Bearer client_id="00000002-0000-0ff1-ce00-000000000000", trusted_issuers="00000001-0000-0000-c000-000000000000@8da56bec-0d27-4cac-ab06-52ee2c40ea22,00000004-0000-0ff1-ce00-000000000000@contoso.com,00000003-0000-0ff1-ce00-000000000000@8da56bec-0d27-4cac-ab06-52ee2c40ea22", token_types="app_asserted_user_v1 service_asserted_app_v1", authorization_uri="https://login.windows.net/common/oauth2/authorize", error="invalid_token" Server: Microsoft-IIS/8.5 Microsoft-HTTPAPI/2.0 request-id: 5fdfec03-2389-42b9-bab9-c787a49d09ca Www-Authenticate: Negotiate Www-Authenticate: NTLM Www-Authenticate: Basic realm="mail.contoso.com" X-FEServer: RGBMSX02 x-ms-diagnostics: 2000003;reason="The hostname component of the audience claim value 'https://autodiscover.contoso.com' is invalid";error_category="invalid_resource" X-Powered-By: ASP.NET Date: Thu, 16 Nov 2017 20:37:48 GMT SSL Offloading As mentioned in the previous section, tokens are issued for a specific uri and that value includes the protocol value ("https://"). When the load balancer offloads the SSL, the request Exchange will receives comes in via HTTP, resulting in a claim mismatch due to the protocol value being "http://": Content-Length →0 Date →Thu, 30 Nov 2017 07:52:52 GMT Server →Microsoft-IIS/8.5 WWW-Authenticate →Bearer client_id="00000002-0000-0ff1-ce00-000000000000", trusted_issuers="00000001-0000-0000-c000-000000000000@00c118a9-2de9-41d3-b39a-81648a7a5e4d", authorization_uri="https://login.windows.net/common/oauth2/authorize", error="invalid_token" WWW-Authenticate →Basic realm="mail.contoso.com" X-FEServer →RGBMSX02 X-Powered-By →ASP.NET request-id →2323088f-8838-4f97-a88d-559bfcf92866 x-ms-diagnostics →2000003;reason="The hostname component of the audience claim value is invalid. Expected 'https://mail.contoso.com'. Actual 'http://mail.contoso.com'.";error_category="invalid_resource" Who’s This? Perhaps you ignored my advice about syncing all your users to AAD? HTTP/1.1 401 Unauthorized Cache-Control: private Server: Microsoft-IIS/7.5 request-id: 63b3e26c-e7fe-4c4e-a0fb-26feddcb1a33 Set-Cookie: ClientId=E9459F787DAA4FA880A70B0941F02AC3; expires=Wed, 25-Oct-2017 11:59:16 GMT; path=/; HttpOnly X-CalculatedBETarget: ex1.contoso.com WWW-Authenticate: Bearer client_id="00000002-0000-0ff1-ce00-000000000000", trusted_issuers="00000001-0000-0000-c000-000000000000@cc2e9d54-565d-4b36-b7f0-9866c19f9b17" x-ms-diagnostics: 2000005;reason="The user specified by the user-context in the token does not exist.";error_category="invalid_user" X-AspNet-Version: 4.0.30319 WWW-Authenticate: Basic realm="mail.contoso.com" WWW-Authenticate: Negotiate WWW-Authenticate: NTLM X-Powered-By: ASP.NET X-FEServer: E15 Date: Tue, 25 Oct 2016 11:59:16 GMT Content-Length: 0 Password Changed? When the user changes their password they must re-authenticate to get a new Refresh/Access token pair. HTTP/1.1 400 Bad Request Cache-Control: no-cache, no-store Pragma: no-cache Content-Type: application/json; charset=utf-8 Expires: -1 Server: Microsoft-IIS/8.5 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Content-Type-Options: nosniff x-ms-request-id: f840b3e7-8740-4698-b252-d759825e0300 P3P: CP="DSP CUR OTPi IND OTRi ONL FIN" Set-Cookie: esctx=AQABAAAAAABHh4kmS_aKT5XrjzxRAtHz3lyJfwgypqTMzLvXD-deUmtaub0aqU_17uPZe3xCZbgKz8Ws99KNxVJSM0AglTVLUEtzTz8y8wTTavHlEG6on2cOjXqRtbgr2DLezsw_OZ7JP4M42qZfMd1mR0BlTLWI3dSllBFpS9Epvh5Yi0Of5eQkOHL7x97IDk_o1EWB7lEgAA; domain=.login.windows.net; path=/; secure; HttpOnly Set-Cookie: x-ms-gateway-slice=008; path=/; secure; HttpOnly Set-Cookie: stsservicecookie=ests; path=/; secure; HttpOnly X-Powered-By: ASP.NET Date: Thu, 16 Nov 2017 20:36:16 GMT Content-Length: 605 {"error":"invalid_grant","error_description":"AADSTS50173: The provided grant has expired due to it being revoked. The user might have changed or reset their password. The grant was issued on '2017-10-28T17:20:13.2960000Z' and the TokensValidFrom date for this user is '2017-11-16T20:27:45.0000000Z'\r\nTrace ID: f840b3e7-8740-4698-b252-d759825e0300\r\nCorrelation ID: f3fc8b2f-7cf1-4ce8-b34d-5dd41aba0a02\r\nTimestamp: 2017-11-16 20:36:16Z","error_codes":[50173],"timestamp":"2017-11-16 20:36:16Z","trace_id":"f840b3e7-8740-4698-b252-d759825e0300","correlation_id":"f3fc8b2f-7cf1-4ce8-b34d-5dd41aba0a02"} Unicorn Rampage? When a Unicorn Rampage has taken place and all tokens are invalidated you’ll see this. HTTP/1.1 400 Bad Unicorn Cache-Control: no-cache, no-store, not-bloody-safe Pragma: no-cache Content-Type: application/json; charset=utf-8 Expires: -1 Server: Microsoft-IIS/8.5 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Content-Type-Options: nosniff x-ms-request-id: f840b3e7-8740-4698-b252-d759825e0300 P3P: CP="DSP CUR OTPi IND OTRi ONL FIN" Set-Cookie: esctx=AQABAAAAAABHh4kmS_aKT5XrjzxRAtHz3lyJfwgypqTMzLvXD-deUmtaub0aqU_17uPZe3xCZbgKz8Ws99KNxVJSM0AglTVLUEtzTz8y8wTTavHlEG6on2cOjXqRtbgr2DLezsw_OZ7JP4M42qZfMd1mR0BlTLWI3dSllBFpS9Epvh5Yi0Of5eQkOHL7x97IDk_o1EWB7lEgAA; domain=.login.windows.net; path=/; secure; HttpOnly Set-Cookie: x-ms-gateway-slice=008; path=/; secure; HttpOnly Set-Cookie: stsservicecookie=ests; path=/; secure; HttpOnly X-Powered-By: ASP.NET Date: Thu, 16 Nov 2017 20:36:16 GMT Content-Length: 605 {"error":"unicorn_rampage","error_description":"The Unicorns are on a rampage. It’s time go home” '2017-11-16T20:27:45.0000000Z'\r\nTrace ID: f840b3e7-8740-4698-b252-d759825e0300\r\nCorrelation ID: f3fc8b2f-7cf1-4ce8-b34d-5dd41aba0a02\r\nTimestamp: 2017-11-16 20:36:16Z","error_codes":[50173],"timestamp":"2017-11-16 20:36:16Z","trace_id":"f840b3e7-8740-4698-b252-d759825e0300","correlation_id":"f3fc8b2f-7cf1-4ce8-b34d-5dd41aba0a02"} And so on. You can see there are a few things that can go wrong, but Fiddler is your friend, so use it to debug and look closely and often the answer is staring you right there in the face. Viewing Tokens Lastly, and just for fun, if you want to see what an actual, real life Access token looks like, I’ll show you how… calm down, it’s not that exciting. In Fiddler, in the Request (upper pane), where you see Header + Value (begins ey…), you can right click the value and choose Send to Text Wizard, and set Transform to ‘From Base64’. Or you can copy the entire value and use a web site such as https://jwt.io to transform them into a readable format like this. { "aud": "https://autodiscover.contoso.com/", "iss": "https://sts.windows.net/f31f3647-5d87-4b69-a0b6-73f62aeab14c/", "acr": "1", "aio": "ASQA2/8DAAAAn27t2aiyI+heHYucfj0pMmQhcEEYkgRP6+2ox9akUsM=", "amr": [ "pwd" ], "appid": "d3590ed6-52b3-4102-aeff-aad2292ab01c", "appidacr": "0", "e_exp": 262800, "enfpolids": [], "family_name": "Taylor", "given_name": "Greg", "ipaddr": “100.100.100.100", "name": "Greg Taylor (sounds like a cool guy)", "oid": "7f199a96-50b1-4675-9db0-57b362c5d564", "onprem_sid": "S-1-5-21-2366433183-230171048-1893555995-1654", "platf": "3", "puid": "1003BFFD9ACA40EE", "scp": "Calendars.ReadWrite Contacts.ReadWrite Files.ReadWrite.All Group.ReadWrite.All Mail.ReadWrite Mail.Send Privilege.ELT Signals-Internal.Read Signals-Internal.ReadWrite Tags.ReadWrite user_impersonation", "sub": "32Q7MW8A7kNX5dPed4_XkHP4YwuC6rA8yBwnoROnSlU", "tid": "f31f3647-5d87-4b69-a0b6-73f62aeab14c", "unique_name": "GregT@contoso.com", "upn": "GregT@contoso.com", "ver": "1.0" } Fun times, eh? I was just relieved to see my enfpolids claim was empty when I saw that line, that sounds quite worrying and something I was going to ask my doctor about. Summary We’ve covered why HMA is great, why it’s more secure, how to get ready for it and how to enable it. And even how to troubleshoot it. Like all changes it requires careful planning and execution, and particularly when messing with auth, be super careful, please. If people can’t connect, that’s bad. We’ve been running like this for months inside Microsoft, and we too missed an SPN when we first did it, so it can happen. But if you take your time and do it right, stronger, better and heck, a more Modern auth can be yours. Good luck Greg Taylor Principal PM Manager Office 365 Customer Experience
Problem with Teams and Exchange Integration
Hi everyone, My goal is to integrate Exchange Server (Exchange 2016 CU23) with Teams so that the Teams calendar can be used. I have set up Hybrid Configuration (Classic Full Hybrid) and completed the setup without any errors. I have also checked Entourage, and it shows that Exchange has been registered, but the Teams calendar is still not working. Check items: 1. Get-AuthServer 2. Get-IntraOrganizationConnector → Confirm the "Enabled" status. 3. Get-PartnerApplication → Confirm it's enabled. 4. External resolution of the Autodiscover and EWS virtual directories 5. Run Teams Exchange Integration → The only thing that works is "Verifying if the user's mailbox is discoverable by the Teams service," and everything else is fine.The user's mailbox is not discoverable by the Teams service. Please ask your administrators to verify the user has a mailbox and to confirm the connectivity between Teams and Exchange. Does anyone have any other ideas or suggestions to help me? Thanks!
Free/Busy Hybrid Exchange 2019
Hello everyone, I have a hybrid environment with Exchange Server 2019, with a dedicated Exchange application, we renewed the Exchange Auth certificate and then re-ran the HCW, however, free/busy is not working for 365 Pro local users. Has anyone had this problem? Could you help me?Resolved: Hybrid Exchange Duplicate (Ghost) Mailbox Created After Assigning Exchange Online License
Summary During a hybrid Exchange migration, a user’s mailbox failed to migrate and mail flow broke due to a duplicate (ghost) mailbox automatically created in Exchange Online. Root Cause An Exchange Online Plan 2 license was mistakenly assigned to the user before migration. Azure AD sync then provisioned a cloud mailbox, even though the user already had an on-prem mailbox. This caused a hybrid mismatch — the user appeared in both environments, and migration failed with mailbox lookup errors. Resolution Steps Removed the Exchange Online Plan 2 license from the user account. Forced a DirSync (AAD Connect) synchronization. Verified that the mailbox existed only on-prem via PowerShell Get-Mailbox -Identity email address removed for privacy reasons | fl Name,RecipientTypeDetails,ExchangeGuid Confirmed the ghost mailbox was removed from Exchange Online. Re-ran migration batch successfully to Exchange Online. Verification Get-MailboxStatistics -Identity email address removed for privacy reasons | fl TotalItemSize,ItemCount,LastLogonTime Ensure only one mailbox object exists and mail flow routes correctly. Prevention Tips Don’t assign Exchange Online licenses to hybrid mailboxes before migration. Always verify mailbox location prior to assigning any license. Use PowerShell or EAC to check where the mailbox resides (on-prem vs. cloud). Environment Hybrid Exchange Deployment Exchange 2016 On-Premises Exchange Online (M365) Azure AD Connect This issue is not caused by connectors or mail flow settings, but by improper licensing before migration. Removing the license and resyncing resolves the ghost mailbox problem.
