EDIT 9/19/2023: This blog post has received significant update.
In this second part of the Demystifying Hybrid Free/Busy, we will cover troubleshooting of Hybrid Free/Busy scenarios, more specifically – how and where to find an actual error that will indicate where the problem is. Before venturing forth, please make sure that you have seen Part 1 of this demystifying series! Here is the graphics we posted in the previous post; use this as a reference for users that we will be referring to when troubleshooting:
Usually when a user creates a new meeting in Outlook on the web (OWA) or Outlook, clicks on Scheduling Assistant, adds his or her colleague to the meeting, they try to see when the user is available to meet. If they see the hash marks \\\\\\\ instead of seeing if the other user is free or busy, there is an issue.
Here, we do seem to have a bunch of Free/Busy issues:
You can often see an error message by hovering over hash marks, however we usually find that the error is not very specific. Instead, we would need to take slightly more advanced steps to diagnose the issues by checking things like the Remote Connectivity Analyzer tool, Fiddler, F12 Network tab, Outlook logging or SARA tool.
Where is the actual Free/Busy error message?
First, we need to understand in which direction we have a lookup problem. Please see Part 1 for discussion of directionality. Sources of logs:
These steps are important for us to see the relevant message error for Free/Busy issues. Once we know the error message, it’s much easier to resolve the issue.
Remote Connectivity Analyzer
A few things to know about this tool:
Connectivity Test Results:
With these 3 buttons on the top right corner, you can expand all the results and save them as XML or HTML files. Usually, support people appreciate these files a lot, so please do upload them in your support case workspace.
When you expand the results, there are 3 important checks:
Quick tip: on your side, in Windows PoweShell, you can also use the following commands to see the External EWS url of an user based on the Autodiscover call to Office 365, replace what is in Email= with your actual email addresses.
Invoke-RestMethod -Uri "https://outlook.office365.com/autodiscover/autodiscover.json?Email=CLOUDUSER@CONTOSO.COM&Protocol=EWS"
Invoke-RestMethod -Uri "https://outlook.office365.com/autodiscover/autodiscover.json?Email=ONPREMUSER@CONTOSO.COM&Protocol=EWS"
Performing the Free/Busy Lookup. This will be Success or Failed.
If it failed, look under the Additional details to see the error message.
If success, be happy, maybe the issue is resolved, or not be happy as it might be an intermittent issue (which is harder to troubleshoot) or a local issue only (happening in your specific network, machine, Outlook version).
In my case, I see that I have a NoFreeBusyAccessException, given by the Exchange on-premises server HHE1601.
OUTLOOK
Note: The Modern Outlook clients log Free/Busy information in Outlook ETL files and you won’t be able to see the Free/Busy error in plain text here. This was possible with Outlook 2010 logs, back in the old days. But this method is still useful, because you can provide the Outlook ETL log containing the error to Microsoft Support to parse it for you and help you fix it also.
If you want to see the error for yourself, check the Fiddler method.
For the Outlook F/B error, we need to first enable Outlook logging and after this we will need to reproduce the issue (\\\\\\).
After repro, we will collect the Outlook logs. Steps:
SARA
I would also like to mention that there is a Free/Busy troubleshooter in Beta version, incorporated into SARA tool (Microsoft Support and Recovery Assistant for Office 365) which you can download it from here : https://diagnostics.outlook.com/#/
Open SARA and select Outlook scenario, click Next, then select I’m having problems with my calendar, input email address and password of the source mailbox (cloud mailbox if direction not working is cloud > on-premises) and then select I can’t see when someone is free or busy.
Due to the underlying complexity of it all, this is not a completely reliable way of determining the cause of free/busy issues in Hybrid Deployments, but it is a good start when troubleshooting.
This F/B test from SARA covers mostly cloud to cloud scenarios but I recommend it here because it does connectivity and additional checks on tenant, licensing and Autodiscover.
And sometimes it shows the underlying Free/Busy error message. Here are some screenshots with the SARA process:
After the Office 365 readiness checks, the tool will ask you for the email address of the Target Mailbox:
In the failed results, expand the Support Message and User Message:
OWA / Outlook on the web F12 Network Tab
Cloud OWA F12 Network tab You need to login to OWA as the source mailbox, hit F12 (Developer Tools for browser) and select the Network Tab. You would then lookup Free/Busy for the target mailbox (reproduce the issue).
If the source mailbox is hosted in Cloud, to look for the F/B here, you can find the Search Icon and type there “GetSchedule” or find the Filter Icon and type “graphql”, then look at Response or Preview tab to see the error message by expanding GetSchedule until you reach to the error.
(click thumbnail to view larger)
If the Source Mailbox is hosted in Exchange On-Premises, you would look after GetUserAvailabilityInternal:
Fiddler –Outlook or OWA
You would need to download and install Fiddler tool from the internet, enable HTTPS decryption in Fiddler and then reproduce the Free/Busy issue in Outlook or OWA or both.
Fiddler - Exchange Online Source Mailbox logged in Outlook desktop.
Look for “GetUserAvailability” calls and then on the right side, you have Request on the top and Response on the bottom. Switch to XML tabs for a nicer view. In the Request you will see the attendees’ email addresses and, in the Response, you will have ResponseMessage with ResponseClass=Error or ResponseClass=Success.
Fiddler – Exchange Online Source Mailbox logged in OWA.
In Fiddler, you can check in the Request pane, under Raw tab the ClientRequestID, you can for example search after this specific value in your on-premises Exchange server logs: IIS W3SVC2 logs, HTTPProxy EWS logs and EWS logs (more information on these logs, location and extracts, later in the article). Example here from a lab:
ClientRequestID: {72741DFF-A6AC-402B-991B-C6B5D56B1422}
Date: Mon, 11 Sep 2023 19:01:25 GMT
If you are fan of SQL language, you can use a tool like Log Parser Studio and search through these logs, for example, here is a query on the ClientRequestID from earlier:
SELECT DateTime, ClientRequestID, RequestID, UserAgent, SoapAction, ErrorCode, GenericErrors, GenericInfo, FileName FROM '[LOGFILEPATH]'
WHERE ClientRequestID LIKE '%{72741DFF-A6AC-402B-991B-C6B5D56B1422}%'
You can also use findstr.exe utility to look for the client request id or other keywords like the requester’s email address or “CrossForest”.
Example of command:
findstr.exe /I /S "{72741DFF-A6AC-402B-991B-C6B5D56B1422}" *.log
When troubleshooting Free/Busy issues, the following on-premises logs can be very useful, especially for Cloud to On-Premises Free/Busy direction.
IIS logs Default Web Site (DWS)
Path: %SystemDrive%\inetpub\logs\LogFiles\W3SVC1
Path example: C:\inetpub\logs\LogFiles\W3SVC1
Extract of Autodiscover and EWS log entries with IOC Enabled in IIS W3SVC1 logs:
Autodiscover – OAUTH (autodiscover.svc without /WSSecurity)
2016-01-06 17:45:27 10.0.0.5 POST /autodiscover/autodiscover.svc &CorrelationID=<empty>;&ClientId=QNFNHKEEKYENCJITQQ&cafeReqId=7972d1fc-a9d9-44c6-8851-480d3601cbd7; 443 S2S~00000002-0000-0ff1-ce00-000000000000 132.245.65.28 ASAutoDiscover/CrossForest/EmailDomain//15.01.0361.007 200 0 0 109
EWS – OAUTH (exchange.asmx without /WSSecurity)
2016-01-06 17:45:27 10.0.0.5 POST /ews/exchange.asmx &CorrelationID=<empty>;&ClientId=WSIVGUUAUWWRFACJBWDA&cafeReqId=6ce8864c-74a0-4ad2-a3dc-7b69e0415403; 443 <unverified>actas1(sip:joe@contoso.com|smtp:joe@contoso.com|upn:joe@contoso.com) 132.245.65.28 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 703
Example of EWS entry with Organization Relationship Enabled in IIS W3SVC1 logs: EWS – DAUTH (exchange.asmx with /WSSecurity)
2016-01-06 18:04:41 10.0.0.5 POST /ews/exchange.asmx/WSSecurity &CorrelationID=<empty>;&ClientId=VOMGJKAWURSVKOXQLBVA&cafeReqId=18fd3a2e-7b1c-4828-8943-6b20912e2e44; 443 - 132.245.65.28 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 296
IIS logs Exchange BackEnd (BE)
Path: %SystemDrive%\inetpub\logs\LogFiles\W3SVC2
Path example: C:\inetpub\logs\LogFiles\W3SVC2
Example of EWS entry with Organization Relationship Enabled (DAUTH) in IIS W3SVC2 logs:
2016-01-06 18:04:41 fe80::f17f:beef:a5e3:7d3c%25 POST /ews/exchange.asmx/WSSecurity - 444 - fe80::f17f:beef:a5e3:7d3c%25 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 93
HTTPProxy logs for Autodiscover
Path: %ExchangeInstallPath%Logging\HttpProxy\Autodiscover
Path example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Autodiscover
Example of Autodiscover entry with Organization Relationship Enabled (DAUTH)
2016-01-06T18:05:20.552Z,bcdfbed5-f11f-4250-a616-e38cb475cd3f,15,0,1104,2,,Autodiscover,autodiscover.contoso.com,/autodiscover/autodiscover.svc /WSSecurity,,,false,,contoso.com,Smtp~joe@contoso.com,ASAutoDiscover/CrossForest/EmailDomain/ /15.01.0361.007,132.245.65.28,exch-2013,200,200,,POST,Proxy,exch-2013.contoso.com,15.00.1104.000,IntraForest,AnchorMailboxHeader-SMTP,[…],BeginRequest=2016-01-06T18:05:20.192Z;CorrelationID=<empty>;ProxyState-Run=None;FEAuth=BEVersion-1941996624;NewConnection=fe80::f17f:beef:a5e3:7d3c%25&0;
HTTPProxy logs for EWS
Path: %ExchangeInstallPath%Logging\HttpProxy\Ews
Path example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews
Example of EWS entry with Organization Relationship Enabled (DAUTH):
2016-01-06T18:04:41.490Z,4757ab2c-8ccc-4d1a-ae39-0780ecc8eabb,15,0,1104,2,{02CD833F-18AB-413A-83CB-0E86F4DA5362},Ews,mail.contoso.com,/ews/exchange.asmx/WSSecurity,,,false,,contoso.com, Smtp~joe@contoso.com,ASProxy/CrossForest/EmailDomain//15.01.0361.007,132.245.65.28,exch-2013,200,200,,POST,Proxy,exch-2013.contoso.com,15.00.1104.000,IntraForest,AnchorMailboxHeader-SMTP,[…],BeginRequest=2016-01-06T18:04:41.380Z;
EWS logs
Path: %ExchangeInstallPath%Logging\Ews
Path example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\Ews
Example of EWS entry with Organization Relationship Enabled (DAUTH):
2016-01-06T18:04:41.490Z,4757ab2c-8ccc-4d1a-ae39-0780ecc8eabb,15,0,1104,2,{02CD833F-18AB-413A-83CB-0E86F4DA5362}, External,true,jane@contoso.mail.onmicrosoft.com,, ASProxy/CrossForest/EmailDomain//15.01.0361.007,Target=None;Req=Exchange2012/Exchange2013; ,132.245.65.28,exch-2013,exch-2013.contoso.com,GetUserAvailability,200,12150,,,,,,ebd34d71ac7342c19d947d881db4ad55,f866c73e-6c91-475e-bdec-0428bdeaa423,PrimaryServer; Requester=jane@contoso.mail.onmicrosoft.com; Failures=0
Event Viewer Application logs on Exchange Server References here and here.
Example of Event ID 4002 for MSExchange Availability:
Log Name: Application
Source: MSExchange Availability
Event ID: 4002
Task Category: Availability Service
Level: Error
Description:
Process 4568: ProxyWebRequest CrossSite from S-1-5-21-391720751-1508397712-925700815-508779 to https://hybrid.contoso.com/ews/exchange.asmx failed. Caller SIDs: NetworkCredentials. The exception returned is Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequestProcessingException: System.Web.Services.Protocols.SoapException: You have exceeded the available concurrent connections for your account. Try again once your other requests have completed.
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
IIS tracing for the error code in the IIS logs Reference here.
Based on cumulative support team experience, we created a table (see the attachment to this post) with Free/Busy errors encountered so far and their possible resolutions. We cannot cover all possible scenarios and errors even though we have a good-sized list. This is meant to illustrate ways we can resolve specific errors and these suggestions might not work for you even if you have the same error. If you know the exact Free/Busy error that you get and checked configuration as discussed in part 1 of this series, this is already a tremendous progress, and this will help us resolve your issue faster. Of course, you can follow these suggestions on your own as most of the actions are harmless but if you don’t feel confident in troubleshooting on your own or you fear that actions are dangerous or irreversible, please contact us.
Thanks to all that contributed to this content: Ray Fong, Nino Bilic, Tim Heeney, Greg Taylor and Brian Day.
Mirela Buruiana
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.