Demystifying Hybrid Free/Busy: Finding errors and troubleshooting
Published Mar 02 2018 06:59 AM 166K Views

EDIT 9/19/2023: This blog post has received significant update.

In this second part of the Demystifying Hybrid Free/Busy, we will cover troubleshooting of Hybrid Free/Busy scenarios, more specifically – how and where to find an actual error that will indicate where the problem is. Before venturing forth, please make sure that you have seen Part 1 of this demystifying series! Here is the graphics we posted in the previous post; use this as a reference for users that we will be referring to when troubleshooting: FB2_1

Do you really have a Free/Busy issue?

Usually when a user creates a new meeting in Outlook on the web (OWA) or Outlook, clicks on Scheduling Assistant, adds his or her colleague to the meeting, they try to see when the user is available to meet. If they see the hash marks \\\\\\\ instead of seeing if the other user is free or busy, there is an issue.

Here, we do seem to have a bunch of Free/Busy issues:

fbnew01.jpg

You can often see an error message by hovering over hash marks, however we usually find that the error is not very specific. Instead, we would need to take slightly more advanced steps to diagnose the issues by checking things like the Remote Connectivity Analyzer tool, Fiddler, F12 Network tab, Outlook logging or SARA tool.

Where is the actual Free/Busy error message?

First, we need to understand in which direction we have a lookup problem. Please see Part 1 for discussion of directionality. Sources of logs:

  • Remote Connectivity Analyzer tool
  • Outlook logging
  • SARA tool
  • OWA F12 Network Tab
  • Fiddler – Outlook and OWA

These steps are important for us to see the relevant message error for Free/Busy issues. Once we know the error message, it’s much easier to resolve the issue.

Remote Connectivity Analyzer

A few things to know about this tool:

fbnew02.jpg

  • Source Mailbox: the user that will be requesting the free/busy information. This will be the user that is logged in Outlook or OWA and cannot see free/busy for other people. This is also called Requester or Organizer of the meeting.
  • Authentication type for Source Mailbox: you will choose Modern Authentication
  • Source Mailbox credentials: you will need to authenticate with the credentials of the Source Mailbox.
  • The tool doesn’t support Basic Authentication for Exchange Online mailboxes because this is disabled in Exchange Online. While it is still used by Exchange On-premises environments, currently, if you select Basic Authentication for the on-premises source mailbox, the test will fail before doing the actual Free/Busy process.
  • It works if your Exchange on-premises has enabled Modern Authentication for client protocols. In conclusion, Source Mailbox login needs to be using OAuth for this test to work, regardless of where it is hosted.

fbnew03.jpg

  • Target Mailbox: the user that the Source Mailbox is requesting free/busy for. This is the Attendee of the meeting.
  • The tool simulates Outlook’s way of querying Free/Busy. If you have a free/busy issue that is only happening in OWA but not in Outlook Desktop, then this test will likely not catch the error.
  • To be able to perform the test, you must allow connectivity for the Remote Connectivity Analyzer tool’s IP addresses. These are part of the "Microsoft 365 Common and Office Online" ranges published in the Office 365 URLs and IP address ranges. The IPs for the Remote Connectivity Analyzer are part of the range specified as "Allow Required" (currently ID 46 in the documentation).
    Check https://testconnectivity.microsoft.com/Pages/ChangeList.htm for any future changes.
  • Note that you can only insert one Target Mailbox email address per test. If you have errors for multiple target mailboxes, run multiple tests, for each user.

Connectivity Test Results:

fbnew04.jpg

 

With these 3 buttons on the top right corner, you can expand all the results and save them as XML or HTML files. Usually, support people appreciate these files a lot, so please do upload them in your support case workspace.

When you expand the results, there are 3 important checks:

  1. Determining where the source mailbox is hosted (cloud or not).
    1. If the Mailbox is hosted in cloud, you will see something like this: IsOffice365Mailbox=True. The mailbox is hosted in Office 365. <ASURL>https://outlook.office365.com/EWS/Exchange.asmx</ASURL>
    2. If the Mailbox is not hosted in cloud, you will see something like this: IsOffice365Mailbox=False. The mailbox isn't hosted in Office 365.
  2. Determining where the target mailbox is hosted (cloud or not).
  3. Test Autodiscover for the Target Mailbox SMTP to retrieve External EWS url.

Quick tip: on your side, in Windows PoweShell, you can also use the following commands to see the External EWS url of an user based on the Autodiscover call to Office 365, replace what is in Email= with your actual email addresses.

 

Invoke-RestMethod -Uri "https://outlook.office365.com/autodiscover/autodiscover.json?Email=CLOUDUSER@CONTOSO.COM&Protocol=EWS"
Invoke-RestMethod -Uri "https://outlook.office365.com/autodiscover/autodiscover.json?Email=ONPREMUSER@CONTOSO.COM&Protocol=EWS"

 

Performing the Free/Busy Lookup. This will be Success or Failed.

If it failed, look under the Additional details to see the error message.

If success, be happy, maybe the issue is resolved, or not be happy as it might be an intermittent issue (which is harder to troubleshoot) or a local issue only (happening in your specific network, machine, Outlook version).

fbnew06.jpg

In my case, I see that I have a NoFreeBusyAccessException, given by the Exchange on-premises server HHE1601.

OUTLOOK

Note: The Modern Outlook clients log Free/Busy information in Outlook ETL files and you won’t be able to see the Free/Busy error in plain text here. This was possible with Outlook 2010 logs, back in the old days. But this method is still useful, because you can provide the Outlook ETL log containing the error to Microsoft Support to parse it for you and help you fix it also.

If you want to see the error for yourself, check the Fiddler method.

For the Outlook F/B error, we need to first enable Outlook logging and after this we will need to reproduce the issue (\\\\\\).

After repro, we will collect the Outlook logs.  Steps:

  • Enable Outlook logging: Follow this KB article and check the “Enable troubleshooting logging (this requires restarting Outlook)” option.
  • Restart Outlook. 
  • Reproduce the issue for the non-working free/busy direction. Suppose Free/Busy direction not working is cloud to on-premises, you will be logged on as a cloud user (Source Mailbox), go to Calendar tab, New Meeting, Scheduling Assistant, add some on-premises users to a meeting until you see the hash marks (instead of Free/Busy information). You do not need to save or send a meeting request.
  • Collect the Outlook-#####.etl log from %temp%\Outlook Logging folder (reference here). You would need to send the ETL file to Microsoft Support to get it analyzed as we are parsing this log with an internal tool. You might not know this, but Hybrid free/busy support cases are free of charge! Of course, you can still use the other methods (fiddler for Outlook/OWA or browser for OWA) to see Free/Busy error yourself, however we (Support) might ask you additionally to get this log as well for a further dive into the Free/Busy errors.

SARA

I would also like to mention that there is a Free/Busy troubleshooter in Beta version, incorporated into SARA tool (Microsoft Support and Recovery Assistant for Office 365) which you can download it from here : https://diagnostics.outlook.com/#/ 

Open SARA and select Outlook scenario, click Next, then select I’m having problems with my calendar, input email address and password of the source mailbox (cloud mailbox if direction not working is cloud > on-premises) and then select I can’t see when someone is free or busy.

fbnew07.jpg

Due to the underlying complexity of it all, this is not a completely reliable way of determining the cause of free/busy issues in Hybrid Deployments, but it is a good start when troubleshooting.
This F/B test from SARA covers mostly cloud to cloud scenarios but I recommend it here because it does connectivity and additional checks on tenant, licensing and Autodiscover.

And sometimes it shows the underlying Free/Busy error message.  Here are some screenshots with the SARA process:

fbnew08.jpg

After the Office 365 readiness checks, the tool will ask you for the email address of the Target Mailbox:

fbnew09.jpg

In the failed results, expand the Support Message and User Message:

fbnew10.jpg

OWA / Outlook on the web F12 Network Tab

Cloud OWA F12 Network tab You need to login to OWA as the source mailbox, hit F12 (Developer Tools for browser) and select the Network Tab. You would then lookup Free/Busy for the target mailbox (reproduce the issue).

If the source mailbox is hosted in Cloud, to look for the F/B here, you can find the Search Icon and type there “GetSchedule” or find the Filter Icon and type “graphql”, then look at Response or Preview tab to see the error message by expanding GetSchedule until you reach to the error.

fbnew11.jpg

(click thumbnail to view larger) 

If the Source Mailbox is hosted in Exchange On-Premises, you would look after GetUserAvailabilityInternal:

fbnew12.jpg

Fiddler –Outlook or OWA

You would need to download and install Fiddler tool from the internet, enable HTTPS decryption in Fiddler and then reproduce the Free/Busy issue in Outlook or OWA or both.
Fiddler - Exchange Online Source Mailbox logged in Outlook desktop.

Look for “GetUserAvailability” calls and then on the right side, you have Request on the top and Response on the bottom. Switch to XML tabs for a nicer view. In the Request you will see the attendees’ email addresses and, in the Response, you will have ResponseMessage with ResponseClass=Error or ResponseClass=Success.

Fiddler – Exchange Online Source Mailbox logged in OWA.

fbnew13.jpg

 

fbnew14.jpg

In Fiddler, you can check in the Request pane, under Raw tab the ClientRequestID, you can for example search after this specific value in your on-premises Exchange server logs: IIS W3SVC2 logs, HTTPProxy EWS logs and EWS logs (more information on these logs, location and extracts, later in the article). Example here from a lab:

fbnew15.jpg

ClientRequestID: {72741DFF-A6AC-402B-991B-C6B5D56B1422}

Date: Mon, 11 Sep 2023 19:01:25 GMT

fbnew16.jpg

If you are fan of SQL language, you can use a tool like Log Parser Studio and search through these logs, for example, here is a query on the ClientRequestID from earlier:

 

SELECT DateTime, ClientRequestID, RequestID, UserAgent, SoapAction, ErrorCode, GenericErrors, GenericInfo, FileName FROM '[LOGFILEPATH]'
WHERE ClientRequestID LIKE '%{72741DFF-A6AC-402B-991B-C6B5D56B1422}%'

 

 

fbnew17.jpg

You can also use findstr.exe utility to look for the client request id or other keywords like the requester’s email address or “CrossForest”.

Example of command:

 

findstr.exe /I /S "{72741DFF-A6AC-402B-991B-C6B5D56B1422}" *.log

 

When troubleshooting Free/Busy issues, the following on-premises logs can be very useful, especially for Cloud to On-Premises Free/Busy direction. 
IIS logs Default Web Site (DWS) 

 

Path: %SystemDrive%\inetpub\logs\LogFiles\W3SVC1
Path example: C:\inetpub\logs\LogFiles\W3SVC1

 

Extract of Autodiscover and EWS log entries with IOC Enabled in IIS W3SVC1 logs: 

Autodiscover – OAUTH (autodiscover.svc without /WSSecurity)

 

2016-01-06 17:45:27 10.0.0.5 POST /autodiscover/autodiscover.svc &CorrelationID=<empty>;&ClientId=QNFNHKEEKYENCJITQQ&cafeReqId=7972d1fc-a9d9-44c6-8851-480d3601cbd7; 443 S2S~00000002-0000-0ff1-ce00-000000000000 132.245.65.28 ASAutoDiscover/CrossForest/EmailDomain//15.01.0361.007 200 0 0 109

 

EWS – OAUTH (exchange.asmx without /WSSecurity)

 

2016-01-06 17:45:27 10.0.0.5 POST /ews/exchange.asmx &CorrelationID=<empty>;&ClientId=WSIVGUUAUWWRFACJBWDA&cafeReqId=6ce8864c-74a0-4ad2-a3dc-7b69e0415403; 443 <unverified>actas1(sip:joe@contoso.com|smtp:joe@contoso.com|upn:joe@contoso.com) 132.245.65.28 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 703

 

Example of EWS entry with Organization Relationship Enabled in IIS W3SVC1 logs: EWS – DAUTH (exchange.asmx with /WSSecurity)

 

2016-01-06 18:04:41 10.0.0.5 POST /ews/exchange.asmx/WSSecurity &CorrelationID=<empty>;&ClientId=VOMGJKAWURSVKOXQLBVA&cafeReqId=18fd3a2e-7b1c-4828-8943-6b20912e2e44; 443 - 132.245.65.28 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 296

 

IIS logs Exchange BackEnd (BE) 

 

Path: %SystemDrive%\inetpub\logs\LogFiles\W3SVC2
Path example: C:\inetpub\logs\LogFiles\W3SVC2

 

Example of EWS entry with Organization Relationship Enabled (DAUTH) in IIS W3SVC2 logs:

 

2016-01-06 18:04:41 fe80::f17f:beef:a5e3:7d3c%25 POST /ews/exchange.asmx/WSSecurity - 444 - fe80::f17f:beef:a5e3:7d3c%25 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 93

 

HTTPProxy logs for Autodiscover 

 

Path: %ExchangeInstallPath%Logging\HttpProxy\Autodiscover
Path example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Autodiscover

 

Example of Autodiscover entry with Organization Relationship Enabled (DAUTH)

 

2016-01-06T18:05:20.552Z,bcdfbed5-f11f-4250-a616-e38cb475cd3f,15,0,1104,2,,Autodiscover,autodiscover.contoso.com,/autodiscover/autodiscover.svc /WSSecurity,,,false,,contoso.com,Smtp~joe@contoso.com,ASAutoDiscover/CrossForest/EmailDomain/ /15.01.0361.007,132.245.65.28,exch-2013,200,200,,POST,Proxy,exch-2013.contoso.com,15.00.1104.000,IntraForest,AnchorMailboxHeader-SMTP,[…],BeginRequest=2016-01-06T18:05:20.192Z;CorrelationID=<empty>;ProxyState-Run=None;FEAuth=BEVersion-1941996624;NewConnection=fe80::f17f:beef:a5e3:7d3c%25&0;

 

HTTPProxy logs for EWS

 

Path: %ExchangeInstallPath%Logging\HttpProxy\Ews
Path example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews

 

Example of EWS entry with Organization Relationship Enabled (DAUTH):

 

2016-01-06T18:04:41.490Z,4757ab2c-8ccc-4d1a-ae39-0780ecc8eabb,15,0,1104,2,{02CD833F-18AB-413A-83CB-0E86F4DA5362},Ews,mail.contoso.com,/ews/exchange.asmx/WSSecurity,,,false,,contoso.com, Smtp~joe@contoso.com,ASProxy/CrossForest/EmailDomain//15.01.0361.007,132.245.65.28,exch-2013,200,200,,POST,Proxy,exch-2013.contoso.com,15.00.1104.000,IntraForest,AnchorMailboxHeader-SMTP,[…],BeginRequest=2016-01-06T18:04:41.380Z;

 

EWS logs 

 

 

Path: %ExchangeInstallPath%Logging\Ews
Path example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\Ews

 

 

Example of EWS entry with Organization Relationship Enabled (DAUTH):

 

 

2016-01-06T18:04:41.490Z,4757ab2c-8ccc-4d1a-ae39-0780ecc8eabb,15,0,1104,2,{02CD833F-18AB-413A-83CB-0E86F4DA5362}, External,true,jane@contoso.mail.onmicrosoft.com,, ASProxy/CrossForest/EmailDomain//15.01.0361.007,Target=None;Req=Exchange2012/Exchange2013; ,132.245.65.28,exch-2013,exch-2013.contoso.com,GetUserAvailability,200,12150,,,,,,ebd34d71ac7342c19d947d881db4ad55,f866c73e-6c91-475e-bdec-0428bdeaa423,PrimaryServer; Requester=jane@contoso.mail.onmicrosoft.com; Failures=0

 

 

Event Viewer Application logs on Exchange Server References here and here.

Example of Event ID 4002 for MSExchange Availability:

 

Log Name: Application
Source: MSExchange Availability
Event ID: 4002
Task Category: Availability Service
Level: Error
Description:
Process 4568: ProxyWebRequest CrossSite from S-1-5-21-391720751-1508397712-925700815-508779 to https://hybrid.contoso.com/ews/exchange.asmx failed. Caller SIDs: NetworkCredentials. The exception returned is Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequestProcessingException: System.Web.Services.Protocols.SoapException: You have exceeded the available concurrent connections for your account. Try again once your other requests have completed.
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)

IIS tracing for the error code in the IIS logs Reference here.

Free/Busy errors and fixes

Based on cumulative support team experience, we created a table (see the attachment to this post) with Free/Busy errors encountered so far and their possible resolutions. We cannot cover all possible scenarios and errors even though we have a good-sized list. This is meant to illustrate ways we can resolve specific errors and these suggestions might not work for you even if you have the same error. If you know the exact Free/Busy error that you get and checked configuration as discussed in part 1 of this series, this is already a tremendous progress, and this will help us resolve your issue faster. Of course, you can follow these suggestions on your own as most of the actions are harmless but if you don’t feel confident in troubleshooting on your own or you fear that actions are dangerous or irreversible, please contact us.

Free/Busy Errors discussed in the attached document (FB_Errors_FixesV7):

  1. “An internal server error occurred. The operation failed” LID: 59916. 500 Internal Server error.
  2. "The remote user mailbox must specify the the explicit local mailbox in the header"
  3. "An error occurred when verifying security for the message"
  4. "Unable to connect to the remote server"
  5. “Autodiscover failed for email address <> with error ‘The request failed with HTTP status 404: Not Found’ ”
  6. “The request failed with HTTP status 401: Unauthorized - The user specified by the user-context in the token is ambiguous” LID: 43532
  7. "An existing connection was forcibly closed by the remote host - An unexpected error occurred on a receive "
  8. "An existing connection was forcibly closed by the remote host - An unexpected error occurred on a send ”
  9. "Configuration information for forest/domain could not be found in Active Directory"
  10. "Proxy web request failed.,inner exception: The request failed with HTTP status 401: Unauthorized."
  11. "The response from the Autodiscover service at 'https://autodiscover/autodiscover.svc/WSSecurity' failed due to an error in user setting 'ExternalEwsUrl'. Error message: InvalidUser." LID: 33676
  12. “The caller does not have access to free/busy data" LID: 47652 LID: 44348
  13. “The request failed with HTTP status 403: Forbidden (The server denied the specified Uniform Resource Locator (URL). “ LID: 43532
  14. “Unable to resolve e-mail address user@notes.domain.com to an Active Directory object” LID: 57660
  15. “An error occurred when processing the security tokens in the message.” LID: 59916
  16. “The cross-organization request for mailbox yyy@contoso.com is not allowed because the requester is from a different organization” LID: 39660
  17. “The request failed with HTTP status 401: Unauthorized - Microsoft.Exchange.Security.OAuth.OAuth TokenRequestFailedException: Missing signing certificate “
  18. “The application is missing a linked account for RBAC roles, or the linked account has no RBAC role assignments, or the calling users account is logon disabled”
  19. “The entered and stored passwords do not match“
  20. “The password has to be changed.”
  21. “The password for the account has expired” or “Provision is needed before federated account can be logged in”
  22. “The request timed out”
  23. “The specified member name is either invalid or empty”
  24. “The result set contains too many calendar entries” LID: 54796
  25. “The request failed with HTTP status 401: Unauthorized - The token has an invalid signature.”
  26. “The request failed with HTTP status 401: Unauthorized - Client assertion contains an invalid signature. [Reason - The key was not found., Thumbprint of key used by client: '<>’ “
  27. “Proxy web request failed., inner exception: Response is not well-formed XML “
  28. “Failed to communicate with https://login.microsoftonline.com/extSTS.srf., inner exception: Unable to connect to the remote server”
  29. “Autodiscover failed for E-Mail Address <> with error System.Net.WebException: The remote name could not be resolved: '<>'”
  30. “Failed to get ASURL. Error 8004010F”
  31. “Proxy web request failed. , inner exception: System.Net.WebException: The request failed with the error message: -- &lt;head&gt;&lt;title&gt;Object moved”
  32. “The request was aborted: Could not create SSL/TLS secure channel.”
  33. “The user specified by the user-context in the token does not exist.";error_category="invalid_user“
  34. “The hostname component of the audience claim value 'https://<>’ is invalid";error_category="invalid_resource“
  35. “Proxy web request failed. , inner exception: System.Net.WebException: The request failed with HTTP status 503: Service Unavailable”
  36. “Proxy web request failed. , inner exception: System.Net.WebException: The request failed with HTTP status 504: Gateway Timeout.”

Thanks to all that contributed to this content: Ray Fong, Nino Bilic, Tim Heeney, Greg Taylor and Brian Day.

Mirela Buruiana

87 Comments
Copper Contributor

Ciao Mirela

I have an Exchange Hybrid Deployment (with Exchange 2016)

Everything was working fine, but after we have switched the publication on Web Application Proxy, the availability stopt to work.

All the services are published in Pass trough and all the other exchange services are ok autodiscover included.

There some settings that we have to check on the WAP server?

Thank you

Michele Foffano

Microsoft

Ciao Michele,

We would need the Free/Busy error for the affected direction. Assuming it is cloud to on-premises, check the error with Fiddler or OWA F12 (look at GetUserAvailability* as explained above) and check Get-IntraOrganizationConnector and Get-OrganizationRelationship settings in Cloud. And also on exrca.com the free/busy test under O365 for Cloud user to OnPrem user (assuming direction is cloud -> onprem). This will say if pass through is there indeed at least for Basic Auth (what exrca tests now) and other things like ports , certificates. Could be also a TLS issue, I cannot guess, so we need the F/B error message and you can also check the most common errors here: FB_Errors.FixesV6

Microsoft

For everyone: my name is Mirela Buruiana and I am the main author of this blog. Reply to this comment if you want me to get notified about your question /comment.

Copper Contributor

Hi Mirela,

quering Get-IntraOrganizationConnector we have noticed that the discovery endpoint URL is  https://webmail.domain.com/autodiscover/autodiscover.svc.

On the Web Application Proxy server we have published all the services individually :

We didn't pubblish the url https://webmail.domain.com/autodiscover .

This was the problem  :smile:

do you know if on the WAP services we can publish the root URL https://webmail.domain.com/ ?

or we must create a pubblication for every service?

 

Thank you for your help

best regards

Ciao

Michele

 

Microsoft

Hi Michele, I don't know WAP :) but you can change the Discovery endpoint on the IOC to autodiscover.domain.com. Unfortunately, HCW by default sets the discoveryEndpoint to the EWS Fqdn /autodiscover . 

Copper Contributor

I am having a free busy issue from Cloud to On-Prem, I've run the free/busy test from EXRCA and all results show that the mailbox is accessible, the trace shows calendar info but testing from a cloud account (OWA) to on-prem does not return results. If the EXRCA works what else should I look at. Is this a permission error?

Microsoft

 

check  "Where can we see actual Free/Busy error message?", look at OWA F12 method or Fiddler and search for GetUserAvailability call as shown above in the blog.
Once you get the error message, check the PDF with errors and fixes, see if something similar.
Copper Contributor

I'm puzzled why The connectivity analyzer shows it all but the actual OWA user it's not working. I would think I'd see some indicator on the report. Thanks for the reply, I'll see if I can see anything in the OWA analysis. I do a lot of hybrids and Free/Busy is the most likely place I run into problems.

Microsoft

RCA test is not accurate. We are working on it.

Copper Contributor

It's showing data from the mailbox that we are looking to get free/busy data from, I see a list of meetings that should be coming back to the cloud user. So even though I see this it may not be a real result? Blocked from sending back to the OWA user in the cloud?

Microsoft

Sorry for delay, @Robert Styles , can you please retry RCA Free/Busy test now and let me know if there is now an error?

Copper Contributor

@Mirela_Buru Thanks for looking at this. The free busy actually looked great. We dug a little deeper and found during the scheduling assistant look up we were getting an authentication error... 

 

The hostname component of the audience claim value 'https://email.customer.com' is invalid


It had been a while since the client had run the HCW, and once we ran through that again it repaired the federation and we were immediately able to see free/busy. In these situations I think I'll try that earlier and hopefully save some frustration!

Copper Contributor
Any thoughts on what the issue is?
 
Performing Free/Busy LookupFree/Busy Lookup failed.
Additional Details
Free/Busy Lookup failed with exception: Proxy web request failed. , inner exception: System.Net.WebException: The request failed with HTTP status 504: Gateway Timeout. at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.Proxy.Service.EndGetUserAvailability(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.FreeBusyApplication.EndProxyWebRequest(ProxyWebRequest proxyWebRequest, QueryList queryList, IService service, IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequest.EndInvoke(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.AsyncWebRequest.EndInvokeWithErrorHandling()
 
Microsoft

@chris280 , I assume it is from EXO to Exchange onpremises? it fails at ews request and usually network related issues. You need to check httpproxy logs / iis logs to see if the request reaches Exchange when you repro the issue or it is stuck at some network device in front. If you are in Modern Hybrid Topology and your Cloud IntraOrganizationConnector or OrganizationRelationship's TargetSharingEpr is pointing to the Hybrid Agent, you need to see if the agent is active /still installed and your firewall/ proxy is allowing outbound traffic to EXO. You can check the troubleshooting suggestions from Scenario 1 from my other blog on Hybrid Migration Endpoints (EWS), these are applicable also here if f/b direction is cloud -> onpremises: https://techcommunity.microsoft.com/t5/exchange-team-blog/troubleshooting-hybrid-migration-endpoints...

Copper Contributor

Hi Mirela_Buru

 

We have an issue with not seeing free/busy from only online exchange to on-prem exchange (works in the other direction fine) I have reviewed you post (great) and have a getuseravailabiltyinternal that is kind of like #3 from your list but not quite. We suspect that something on our cloud configuration is not correct. Do you have any thoughts? 

 

{"Header":{"ServerVersionInfo":{"MajorVersion":15,"MinorVersion":20,"MajorBuildNumber":3433,"MinorBuildNumber":45,"Version":"V2018_01_08"}},"Body":{"Responses":[{"ResponseMessage":{"__type":"SingleResponseMessage:#Exchange","MessageText":"Proxy web request failed. , inner exception: System.Web.Services.Protocols.SoapHeaderException: An error occurred when verifying security for the message.\r\n at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)\r\n at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)\r\n at Microsoft.Exchange.InfoWorker.Common.Availability.Proxy.Service.EndGetUserAvailability(IAsyncResult asyncResult)\r\n at Microsoft.Exchange.InfoWorker.Common.Availability.FreeBusyApplication.EndProxyWebRequest(ProxyWebRequest proxyWebRequest, QueryList queryList, IService service, IAsyncResult asyncResult)\r\n at Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequest.EndInvoke(IAsyncResult asyncResult)\r\n at Microsoft.Exchange.InfoWorker.Common.Availability.AsyncWebRequest.EndInvokeWithErrorHandling()","ResponseCode":"ErrorProxyRequestProcessingFailed","MessageXml":"<?xml version=\"1.0\"?>\r\n<XmlNodeArray xmlns:t=\"http://schemas.microsoft.com/exchange/services/2006/types\" xmlns:m=\"http://schemas.microsoft.com/exchange/services/2006/messages\">\r\n <ExceptionType xmlns=\"http://schemas.microsoft.com/exchange/services/2006/errors\">ProxyWebRequestProcessingException</Exc... <ExceptionCode xmlns=\"http://schemas.microsoft.com/exchange/services/2006/errors\">5016</ExceptionCode>\r\n <ExceptionServerName xmlns=\"http://schemas.microsoft.com/exchange/services/2006/errors\">CH2PR19MB3832</ExceptionServerName>\r\n <ExceptionMessage xmlns=\"http://schemas.microsoft.com/exchange/services/2006/errors\">Proxy web request failed. , inner exception: An error occurred when verifying security for the message. LID: 59916</ExceptionMessage>\r\n</XmlNodeArray>","ResponseClass":"Error"},"CalendarView":{"MergedFreeBusy":null,"Items":null,"WorkingHours":null,"FreeBusyViewType":"None"}}],"ResponseCode":"NoError","ResponseClass":"Success"}}

 

 

I have also ran Get-IntraOrganizationConnector  on both on-prem and exch online and both do not show any output?

 

 

thanks!

Eric

Copper Contributor

Hi Mirela_Buru

 

We have an issue with not seeing free/busy from only online exchange to on-prem exchange (works in the other direction fine) I have reviewed you post (great) and have a getuseravailabiltyinternal that is kind of like #3 from your list but not quite. We suspect that something on our cloud configuration is not correct. Do you have any thoughts? 

 

{"Header":{"ServerVersionInfo":{"MajorVersion":15,"MinorVersion":20,"MajorBuildNumber":3433,"MinorBuildNumber":45,"Version":"V2018_01_08"}},"Body":{"Responses":[{"ResponseMessage":{"__type":"SingleResponseMessage:#Exchange","MessageText":"Proxy web request failed. , inner exception: System.Web.Services.Protocols.SoapHeaderException: An error occurred when verifying security for the message.\r\n at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)\r\n at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)\r\n at Microsoft.Exchange.InfoWorker.Common.Availability.Proxy.Service.EndGetUserAvailability(IAsyncResult asyncResult)\r\n at Microsoft.Exchange.InfoWorker.Common.Availability.FreeBusyApplication.EndProxyWebRequest(ProxyWebRequest proxyWebRequest, QueryList queryList, IService service, IAsyncResult asyncResult)\r\n at Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequest.EndInvoke(IAsyncResult asyncResult)\r\n at Microsoft.Exchange.InfoWorker.Common.Availability.AsyncWebRequest.EndInvokeWithErrorHandling()","ResponseCode":"ErrorProxyRequestProcessingFailed","MessageXml":"<?xml version=\"1.0\"?>\r\n<XmlNodeArray xmlns:t=\"http://schemas.microsoft.com/exchange/services/2006/types\" xmlns:m=\"http://schemas.microsoft.com/exchange/services/2006/messages\">\r\n <ExceptionType xmlns=\"http://schemas.microsoft.com/exchange/services/2006/errors\">ProxyWebRequestProcessingException</Exc... <ExceptionCode xmlns=\"http://schemas.microsoft.com/exchange/services/2006/errors\">5016</ExceptionCode>\r\n <ExceptionServerName xmlns=\"http://schemas.microsoft.com/exchange/services/2006/errors\">CH2PR19MB3832</ExceptionServerName>\r\n <ExceptionMessage xmlns=\"http://schemas.microsoft.com/exchange/services/2006/errors\">Proxy web request failed. , inner exception: An error occurred when verifying security for the message. LID: 59916</ExceptionMessage>\r\n</XmlNodeArray>","ResponseClass":"Error"},"CalendarView":{"MergedFreeBusy":null,"Items":null,"WorkingHours":null,"FreeBusyViewType":"None"}}],"ResponseCode":"NoError","ResponseClass":"Success"}}

 

 

I have also ran Get-IntraOrganizationConnector  on both on-prem and exch online and both do not show any output?

 

 

thanks!

Eric

Copper Contributor

Hi Mirela_Buru

 

We have an issue with not seeing free/busy from only online exchange to on-prem exchange (works in the other direction fine) I have reviewed you post (great) and have a getuseravailabiltyinternal that is kind of like #3 from your list but not quite. We suspect that something on our cloud configuration is not correct. Do you have any thoughts? 

 

{"Header":{"ServerVersionInfo":{"MajorVersion":15,"MinorVersion":20,"MajorBuildNumber":3433,"MinorBuildNumber":45,"Version":"V2018_01_08"}},"Body":{"Responses":[{"ResponseMessage":{"__type":"SingleResponseMessage:#Exchange","MessageText":"Proxy web request failed. , inner exception: System.Web.Services.Protocols.SoapHeaderException: An error occurred when verifying security for the message.\r\n at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)\r\n at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)\r\n at Microsoft.Exchange.InfoWorker.Common.Availability.Proxy.Service.EndGetUserAvailability(IAsyncResult asyncResult)\r\n at Microsoft.Exchange.InfoWorker.Common.Availability.FreeBusyApplication.EndProxyWebRequest(ProxyWebRequest proxyWebRequest, QueryList queryList, IService service, IAsyncResult asyncResult)\r\n at Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequest.EndInvoke(IAsyncResult asyncResult)\r\n at Microsoft.Exchange.InfoWorker.Common.Availability.AsyncWebRequest.EndInvokeWithErrorHandling()","ResponseCode":"ErrorProxyRequestProcessingFailed","MessageXml":"<?xml version=\"1.0\"?>\r\n<XmlNodeArray xmlns:t=\"http://schemas.microsoft.com/exchange/services/2006/types\" xmlns:m=\"http://schemas.microsoft.com/exchange/services/2006/messages\">\r\n <ExceptionType xmlns=\"http://schemas.microsoft.com/exchange/services/2006/errors\">ProxyWebRequestProcessingException</Exc... <ExceptionCode xmlns=\"http://schemas.microsoft.com/exchange/services/2006/errors\">5016</ExceptionCode>\r\n <ExceptionServerName xmlns=\"http://schemas.microsoft.com/exchange/services/2006/errors\">CH2PR19MB3832</ExceptionServerName>\r\n <ExceptionMessage xmlns=\"http://schemas.microsoft.com/exchange/services/2006/errors\">Proxy web request failed. , inner exception: An error occurred when verifying security for the message. LID: 59916</ExceptionMessage>\r\n</XmlNodeArray>","ResponseClass":"Error"},"CalendarView":{"MergedFreeBusy":null,"Items":null,"WorkingHours":null,"FreeBusyViewType":"None"}}],"ResponseCode":"NoError","ResponseClass":"Success"}}

 

 

I have also ran Get-IntraOrganizationConnector  on both on-prem and exch online and both do not show any output?

 

 

thanks!

Eric

Microsoft

Hi Eric, as mentioned in the first blog comment, make sure we are looking at the right Organization Relationship in Exchange Online (Office 365 > On-Premises ) and that is configured properly. If indeed the request reaches on-premises servers and not EXO servers,  then please recycle EWS App Pool on-premises /IISreset like I mentioned in the PDF with errors.

Copper Contributor

(deleted)

Microsoft

@ADynes , provide me with the case number (you can also send me a private message) so that I can check it.

Copper Contributor

Hi @Mirela_Buru 

 

Your help was great and helped me figure out our main problem. We used a service to help us with setting up hybrid, they had us change our on-prem autodiscover dns record to point to outlook.com before we ran the wizard. I changed it back to our on-prem ip address after the wizard was run initially.  After I ran the wizard again (with the correct autodiscover record)  we are now seeing on-prem free/busy for our exchange 2013 users. We have exchange 2010 and 2013 on-prem and unfortunately we are unable to see the exchange 2010 users free/busy. The free/busy troubleshooter comes up with the error message below when I point to an exchange 2010 mailbox:

 

Any ideas where I should look next?

 

Performing Free/Busy Test. See guidance about the Hybrid Configuration Wizard at http://aka.ms/HCWFree/Busy test failed
Test Steps
Determining where the source mailbox is hosted.The check to determine if the source mailbox is hosted in Office 365 completed successfully.
Additional Details
Determining where the target mailbox is hosted. The check to determine if the target mailbox is hosted in Office 365 completed successfully.
Additional Details
The Microsoft Connectivity Analyzer is attempting to test Autodiscover for xx .Autodiscover was tested successfully.
Test Steps
Verifying connectivity to the specified endpointSuccessfully verified pass-through connectivity.
Additional Details
Verifying connectivity to the specified endpointSuccessfully verified pass-through connectivity.
Additional Details
Performing Free/Busy LookupFree/Busy Lookup failed.
Additional Details
Free/Busy Lookup failed with exception: Proxy web request failed. , inner exception: System.Web.Services.Protocols.SoapHeaderException: An error occurred when verifying security for the message. at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.Proxy.Service.EndGetUserAvailability(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.FreeBusyApplication.EndProxyWebRequest(ProxyWebRequest proxyWebRequest, QueryList queryList, IService service, IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequest.EndInvoke(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.AsyncWebRequest.EndInvokeWithErrorHandling()
Microsoft

Hi @epeterson79 , for the security of the message error, just follow the previous suggestions: 
recycle EWS App Pool on all Exchange servers / run IISreset  like I mentioned in the PDF FB_Errors.FixesV6 (error #3) : https://support.microsoft.com/en-us/help/2752387/users-from-a-federated-organization-cannot-see-the-... 

 

Copper Contributor

Hello @Mirela_Buru 

First of all, Thanks a million, I've solved lots of Hybrid Free / Busy issues by taking help from this blog. However, currently I'm having an issue, I've spent a few hours on it but haven't been able to solve it. Our On-Premise Users can not see Cloud Users free / busy.

- First I found Federation Certificate was expired, I renewed it and recreated federation, Test-Federation is now all good.

- After that I found OAuth Certificate was expired, I was able to renew it and publish to all Exchange Servers, Test-OAuthConnectivity is all good.

- Test-OrganizationRelationship is all good.

- IntraOrganizationConnector is enabled.

- I checked your FB errors table but I couldn't found the error I'm facing.

- I've done tonnes of changes at On Premise OrganizationRelationship populating TargetSharingEPR, Testing with / without WSSecurity, Disabling IOC but no luck. I'm stuck on below error.

- In OWA Network Response for Get-Useravailablity as well as at MS RCA I get error, I suspect this is still related to OAuth between EXO and EX-On-Premise ?

 

Free/Busy Lookup failed with exception: Autodiscover failed for email address USER@INITIALDOMAIN.mail.onmicrosoft.com with error System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 52.98.32.2:443

 

Microsoft

@Mashal_Khan , this is a network issue between Exchange On-Premises Servers and Exchange Online Server IP Addresses (52.98.32.2 is EXO server where probably your target cloud user USER@INITIALDOMAIN.mail.onmicrosoft.comis hosted).

How are your Exchange On-premises servers going out on the internet? Proxy or Firewall? You seem to be blocking outbound connections to 52.98.32.2. If using proxy, make sure this is set on the Exchange Servers, you can do Get-ExchangeServer |FL identity, internetwebproxy to check if there and Set-ExchangeServer to set one. Make sure you allow all your Exchange Servers to connect to all EXO IP addresses https://docs.microsoft.com/en-us/microsoft-365/enterprise/urls-and-ip-address-ranges?view=o365-world... 

You should follow Suggestions from Error 22, points 2 and 3 and make sure you can access from Exchange System Account (on each Exchange Server CAS/MBX) the following URLs:
https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc ( EXO Autodiscover Service)

https://52.98.32.2/autodiscover/autodiscover.svc  (continue to certificate error because we put ip)

http://autodiscover.<YOUR-INITIAL-DOMAIN.mail.onmicrosoft.com/autodiscover/autodiscover.svc  (this should redirect to https EXO Autodiscover Service)

https://outlook.office365.com/ews/exchange.asmx ( EXO EWS Service, this can also be set on the TargetSharingEpr on the on-premises Organization Relationship when On-Premises IOC is disabled). 

Copper Contributor

@Mirela_Buru 

Thank You, Sorry I forgot to mention, I checked on all Exchange Servers, they're doing out to the Internet directly i.e. internetwebproxy is empty on all. We don't have internet access on any Exchange Server but Hybrid Server can make outbound requests to anywhere on port 80, 443. I checked all URLs are accessible on Hybrid Server. Free / Busy was fine previously with the same network access. Do I these URLs need to be accessible on all Exchange Servers or just Hybrid Server is enough ?

 

The other thing you advised is that if I disable On Premise IOC and populate OrganizationRelationship with https://outlook.office365.com/ews/exchange.asmx Free / Busy look up will not use Autodiscover but it will use EWS and that may resolve my issue ?

 

Also I will have my network team allow outbound connections to anywhere on port 80, 443 for a small duration to test and let you know if that worked.

Microsoft

1.  Do I these URLs need to be accessible on all Exchange Servers or just Hybrid Server is enough ?

>> ALL the Exchange 2010 MBX & CAS or 2013 MBX (backend) or 2016 would need outbound Internet access to all these:

Reference: https://docs.microsoft.com/en-us/previous-versions/office/exchange-server-2010/dd638083(v=exchg.141)... 

 

2. The other thing you advised is that if I disable On Premise IOC and populate Organization Relationship with https://outlook.office365.com/ews/exchange.asmx Free / Busy look up will not use Autodiscover but it will use EWS and that may resolve my issue ?

>>Yes, correct, you can try to bypass AutoD query and see if it works by going directly to EWS but this can be any on-premises Exchange Server in your organization, so internet access is needed from each.

Copper Contributor

@Mirela_BuruThank you for great help, I was able to create a User on Hybrid Server, that User is able to see Cloud Users Free / Busy as Hybrid Server has required outbound access to anywhere on port 80,443, We need to grant all Exchange Servers outbound access to the Office 365 IPs/ Urls to solve this, THANKS A BILLION !!!!

Copper Contributor

Can you confirm in section 22 "https://login.microsoftonline.com/extSTS.srf [<-- You should be prompted to
download the file.]" - that the file should be downloadable? (I've tried from a variety of different machines but get the same experience)

 

When i run the psexec test I get a webpage that says

 

"Sorry, but we’re having trouble with signing you in.

AADSTS900561: The endpoint only accepts POST requests. Received a GET request."

 

I'm seeing errors in outlook (onprem) to 365 mailbox f/b lookups that state 

 

"<MessageText>Failed to communicate with https://login.microsoftonline.com/extSTS.srf."

 

so it seems a very similar error but i'm struggling to troubleshoot further.

 

Thanks for any pointers.

Microsoft

Hi @David Hood , you are correct, the PDF is outdated. I will try to update it soon.

The https://login.microsoftonline.com/extSTS.srf   should give [<-- You should see “Sorry, but we’re having trouble signing you in”.]

 

For MFG URLs issues, you can also check my other blog on Federation Trust, especially error 3: https://techcommunity.microsoft.com/t5/exchange-team-blog/how-to-address-federation-trust-issues-in-... 

 

Copper Contributor

Hi all,

 

I tried to find out a problem with F/B - Exchange 2016 on-premises users  are not able to view the free/busy information of the office 365 users. They have in Outlook 2016 (could not be updated),

I check OWA F12 and error is: System.Net.WebException: The request failed with HTTP status 401: Unauthorized.

 

I was able to find almost same error in the Microsoft free/busy troubleshooting guide (Index 26) attached below as well: https://anishjohnes.files.wordpress.com/2019/10/fb_errors.fixesv6.pdf

 

but when i test Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.office365.com/ews/exchange.asmx -Mailbox OnPremises 

my exception are:

System.Net.WebException: The request was aborted: The request was canceled. ---> Microsoft.Exchange.Security.OAuth.OAuthTokenRequestFailedException: The trusted issuers contained the followi
ng entries '00000001-0000-0000-c000-000000000000@*'. None of them are configured locally.

 

Also I check following guide (index 26) check certificate mismatch but i have the same on perm and Azure.

 

I found solution, I explaind what I did on the forum: https://social.technet.microsoft.com/Forums/en-US/a0bb1d0a-6639-4fdf-92e1-7fa350e47057/exchange-2016...

 

Regards,

Rafal

Microsoft

Thanks for your feedback, @rchmielarski , for such error we would normally need more outputs on how the OAuth is configured and it's interesting that apparently IISreset did the trick here. Would have been good to know your CU for Exchange 2016 and if by any chance Get-AuthServer  had previously or maybe it still has the  -DomainName populated with an SMTP domain and if the user used in Test-OauthConnectivity had that domain in primary SMTP address.

Copper Contributor
Issue exchange 2013 cannot see free busy information 2007

Hi @Mirela_Buru, I have enviroment with exchange 2007 and exchange 2013. I Have a event id in my exchange 2013: 

 

Process 15040: ProxyWebRequest CrossSite from S-1-5-21-190996285-3174742192-1521001504-11052 to https://mail.teste.com.br:443/EWS/Exchange.asmx failed. Caller SIDs: NetworkCredentials. The exception returned is Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequestProcessingException: Proxy web request failed. ---> System.Net.WebException: The request failed with HTTP status 401: Unauthorized.

 

 

And there is a problem "exchange 2013 cannot see free busy information 2007" . The o rest all work fine 

 

But i don't have in my enviroment exchange online.

This solutions cab se aply in my enviroment? (https://techcommunity.microsoft.com/legacyfs/online/media/2019/01/FB_Errors.FixesV6.pdf  index 10: Proxy web request failed.,inner exception: The
request failed with HTTP status 401:
Unauthorized.) 

 

 

Thank you for your help

best regards

 

Agostinho Martini

Microsoft

@Agostinho_Martini , the solutions from this PDF don't apply to this scenario.

I would suggest to check HTTPProxy EWS logs on the Ex2013 and IIS logs on Ex2007, eventually FREB logs for 401 sub code.

Check Authentication Methods in EWS on Ex2007 server. Make sure Default user doesn't have NONE in the Get-MailboxFolderPermission TargetUser:\Calendar.  Exchange 2007 users can see Free/Busy between them and set OOF replies successfully?

Copper Contributor

@Mirela_Buru  Thank you very much!

 

Users in exchange 2007 can see free busy information amoung user in exchange 2007.

 

User in exchange 2007 can see free busy information users exchange 2013.

 

Only users exchange 2013 cant seee frre busy information user exchange 2007.

 

EWS autentication exchange 2013: Default and Back End site

 

[PS] C:\Windows\system32>Get-WebServicesVirtualDirectory -server smxpoa05 | fl InternalUrl,ExternalUrl,Identity,Name,*au
th* | fl
Creating a new session for implicit remoting of "Get-WebServicesVirtualDirectory" command...


InternalUrl : https://mail.teste1.com.br/ews/exchange.asmx
ExternalUrl : https://mail.teste1.com.br/ews/exchange.asmx
Identity : servidor1\EWS (Default Web Site)
Name : EWS (Default Web Site)
CertificateAuthentication :
InternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity, OAuth}
ExternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity, OAuth}
LiveIdNegotiateAuthentication :
WSSecurityAuthentication : True
LiveIdBasicAuthentication : False
BasicAuthentication : False
DigestAuthentication : False
WindowsAuthentication : True
OAuthAuthentication : True
AdfsAuthentication : False

 

[PS] C:\Windows\system32>Get-WebServicesVirtualDirectory -ShowMailboxVirtualDirectories -Server smxpoa05 | fl


RunspaceId : ca5a7de9-75a2-4fdb-97f2-284da2ac5af2
CertificateAuthentication :
InternalNLBBypassUrl : https://mail.teste1.local:444/ews/exchange.asmx
GzipLevel : High
MRSProxyEnabled : False
Name : EWS (Exchange Back End)
InternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity}
ExternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity}
LiveIdNegotiateAuthentication :
WSSecurityAuthentication : True
LiveIdBasicAuthentication : False
BasicAuthentication : False
DigestAuthentication : False
WindowsAuthentication : True
OAuthAuthentication : False
AdfsAuthentication : False


EWS autentication exchange 2007: Default


Get-WebServicesVirtualDirectory
-server smxpoa01 | fl InternalUrl,ExternalUrl,Identity,Name,*auth* | fl


InternalUrl : https://legacy.teste1.com.br/EWS/Exchange.asmx
ExternalUrl : https://legacy.teste1.com.br/EWS/Exchange.asmx
Identity : SMXPOA01\EWS (Default Web Site)
Name : EWS (Default Web Site)
InternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated}
ExternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated}
BasicAuthentication : True
DigestAuthentication : False
WindowsAuthentication : True

 

In my logs iis 2007 and 2013 i found:

Log IIS 2007:


C:\WINDOWS\system32\LogFiles\HTTPERR

2020-11-09 19:42:28 10.63.78.187 53295 10.63.70.231 443 HTTP/1.1 RPC_IN_DATA /rpc/rpcproxy.dll?SMXPOA04.teste1.local:6001 400 1 BadRequest DefaultAppPool
2020-11-09 19:42:28 170.233.237.115 16855 10.63.70.231 443 - - - - - Timer_ConnectionIdle -
2020-11-09 19:42:32 10.63.51.188 59668 10.63.70.231 443 - - - - - Timer_ConnectionIdle -
2020-11-09 19:42:34 10.63.78.185 56960 10.63.70.231 443 HTTP/1.1 RPC_IN_DATA /rpc/rpcproxy.dll?SMXPOA03.teste1.local:6001 400 1 BadRequest DefaultAppPool
2020-11-09 19:42:34 10.63.78.186 22997 10.63.70.231 443 HTTP/1.1 RPC_IN_DATA /rpc/rpcproxy.dll?SMXPOA03:6001 400 1 BadRequest DefaultAppPool


Log IIS 2013:

C:\Windows\System32\LogFiles\HTTPERR

2020-11-09 20:52:43 10.63.78.185 19986 10.63.78.185 444 HTTP/1.1 RPC_IN_DATA /rpc/rpcproxy.dll?smxpoa05.teste1.local:6001 400 2 Connection_Dropped MSExchangeRpcProxyAppPool
2020-11-09 20:52:43 10.63.75.43 50880 10.63.78.185 443 HTTP/1.1 POST /Microsoft-Server-ActiveSync?User=abiliolima&DeviceId=7QULD3E1SP1FR5ERG8Q5IHK68G&DeviceType=iPhone&Cmd=Ping - 1 Connection_Dropped MSExchangeSyncAppPool
2020-11-09 20:52:43 10.63.75.43 50908 10.63.78.185 443 HTTP/1.1 POST /Microsoft-Server-ActiveSync?eQAWBBAl2wZcQIhAIrRyDVtNyO9ABOj7SokLV2luZG93c01haWw= - 1 Connection_Dropped MSExchangeSyncAppPool

 

For me i am very confused because i found only event id exchange 2013:

 

Process 15040: ProxyWebRequest CrossSite from S-1-5-21-190996285-3174742192-1521001504-11052 to https://mail.teste1.com.br:443/EWS/Exchange.asmx failed. Caller SIDs: NetworkCredentials. The exception returned is Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequestProcessingException: Proxy web request failed. ---> System.Net.WebException: The request failed with HTTP status 401: Unauthorized.

 

Any idea?

 

 

Microsoft

I would say to make sure that is indeed the error (401 Unauthorized) for this free/busy direction ex2013 -> ex2007 .

You can try free/Busy test https://testconnectivity.microsoft.com/tests/FreeBusy/input or OWA F12 Network tab GetUserAvailability call and check Response body. 

And in logs, you would look at EWS/Exchange.asmx or WebServicesAppPool stuff. 

Copper Contributor

Hello Mirela, amazing article.

 

We have a story where 2 onprem exchange Organizations (org1 and orerg2) share an smtp address space (@company.com). 1 org is already hybrid with O365. All works there, including mail routing via EOP and transport rules to decide left or right.

Now the second org wants their mailboxes also moved to Exchange Online. Both Orgs have company.com as primary SMTP for all users. We will be able to setup a second hybrid (minimal hybrid with manual corrections) to support migration and some level of co-existence. We expect F/B onprem (org2 user) to cloud will work. However Cloud to Org2 user onpremise is likely to fail. What would be required, if at all possible, to make this scenario work ? 


Thanks a lot,
Toni

Copper Contributor

Unless it's a new feature, you'll not be able to assign the primary SMTP to two different tenants. I have used hybrid to migrate 7 different schools into their own tenant, one at a time. We connected the Hybrid to the first domain and migrated, then move it to the second domain and repeat down the line. The assumption is that the 'hybrid' is only needed for free/busy during the migration. Once everyone is moved up, you can still maintain SMTP connectivity for remaining mail flow.

Microsoft

@tonivervloet , So the first question is related to tenants, is it Org1 and Org2 to same Tenant1 or different tenant, Tenant2? 

Then yes, like Robert said above, you won't be able to migrate if the SMTP domain is shared between 2 tenants, you need to have an accepted domain verified, unique in Tenant 2. 
If same tenant and 2 on-premises organizations, then I also don't see how to work-around the free/busy part if the on-premises users will have same SMTP. The Target Address Domain is the cloud organization relationships or cloud intra-organization connectors must be unique (even if you can set targetSharingEpr to point to EWS of each on-premises organization).

Copper Contributor

hello @Mirela_Buru , @Robert Styles ,

 

Indeed the address spaces is only assigned to 1 tenant of course, but shared between a tenant (onprem + hybrid) and an onprem org that soon will be hybrid (to the same tenant) as well. But environments have company.com users (primary) and that works fine. Ideally that would have the shared as secondary, then we could uniquely identify both orgs.

We can get an accepted domain verified for org 2 no problem, but that domain is only used as a proxy. Its the old name space.

 

Seems we are stuck ?


thanks already for your time,both.

 

Toni

Copper Contributor

I didnt see this error on your list.  The issue we are having is an O365 user can not see the free/busy calendar for an Exchange 2010 user.  The Exchange user can see the Free/Busy of the O365 user.  When this issue happened in the past, I would run IISREST on the Exchange 2010 CAS servers and the problem would be fixed.  This time, its not working.  Any ideas?

 

Performing Free/Busy LookupFree/Busy Lookup failed.

 
Additional Details
Free/Busy Lookup failed with exception: Proxy web request failed. , inner exception: System.Web.Services.Protocols.SoapHeaderException: An error occurred when processing the security tokens in the message. at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.Proxy.Service.EndGetUserAvailability(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.FreeBusyApplication.EndProxyWebRequest(ProxyWebRequest proxyWebRequest, QueryList queryList, IService service, IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequest.EndInvoke(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.AsyncWebRequest.EndInvokeWithErrorHandling()
Copper Contributor

Found the solution to my problem.  I ran IISRESET again, waited for a bit and no change.  Then I ran this command on both Exchange 2010 CAS server and now the Free/Busy is showing.

Get-FederationTrust | Set-FederationTrust -RefreshMetadata

 

Microsoft

@JeffP2021 , glad you managed to fix it.
This is  similar to error #3 from the PDF : "An error occurred when verifying security for the message" so you would follow the exact steps / resolution 

Mirela_Buru_0-1611835955607.png

You might want to create a scheduled task for this as mentioned here  https://techcommunity.microsoft.com/t5/exchange-team-blog/keep-your-federation-trust-up-to-date/bc-p... 

 

Copper Contributor

Thank you for the reply.  Im not sure how I missed number 3.  My problem is partially solved.  It worked for the calendar I had previously added.  I am still having the trouble adding new calendars.  I will look at number 3 again and follow the suggested steps.

 

I have call scheduled with MS Support today.  Hopefully it will be fixed by then.

Copper Contributor

Mirela

 

Why can I see one on-premise account (the original one that was working) an no other on-premise account?  The free/busy connectivity test passes for this user, but the others get the error in number 3.

 

I am now working with MS Support.

Copper Contributor

Hello,

 

We have exchange 2010 Sp3 environment coexistence with 2016 exchange server as hybrid. We have recently configured hybrid and migrated successfully migrated user to office 365.

 

We have noticed that free busy is not working from office 365 to On-prem users and vice versa.

 

We have checked the Autodiscover endpoint from outside the network it is giving authentication prompt same for EWS gives the authentication prompt however free busy not working. As per the article I have set the target sharingepr but still we have issues. Also we have toggled wssauthentication for exchange 2010 and 2016 and restarted the app pool but no luck.

 

Can you please suggest us we are kind of stuck and not migrating any mailboxes to cloud.

Microsoft

@Shady111 , you should first check this blog announcement Keep your Federation Trust up-to-date - Microsoft Tech Community

 and make sure you performed RefreshMetadata (recommended it to run it twice).

Second, run the Free/Busy test on exrca.com here: Microsoft Remote Connectivity Analyzer and see what is the error message for free/busy lookup from Cloud Source User to On-Premises Target User(s).  You might get different errors. 

Once you know the free/busy error message, you can lookup the list with 34 errors from above and see if you find your error or similar one in the list. And in the PDF you have some suggestions to fix / troubleshoot this:

Free/Busy Errors discussed in the attached table:

FB_Errors.FixesV6

Microsoft

@JeffP2021 , might be needed to run RefreshMetadata twice and possibly iisreset on each server? Can you provide me with the case number (private message), I can maybe take a look there?

Copper Contributor

Thank you Mirela_Buru.  I have ran RefreshMetadata twice and ran IISRESET on both servers, no change.

 

I will send PM with MS case #.  He could use all the help he can get.  Thank you!

Copper Contributor

Hello,

my free busy is failing with this error on both sides , Any thoughts inputs appreciated

ManjunBN_0-1612463716420.png

 

Microsoft

Hi @ManjunBN , I would need a Test-FederationTrust -Verbose output from onpremises organization executed in Exchange Management Shell if Exchange 2010 server, if Ex2013/2016/2019, then you would do it like this:

Open Windows PowerShell (blue)

add-pssnapin *exchange*

start-transcript

Test-FederationTrust -Verbose | fl

Get-FederationTrust | fl

Get-FederatedOrganizationIdentifier | fl

stop-transcript

You can open a case and upload transcript there and send me the case number or you can send me a private message with the output.

Also, did Free/Busy work so far?

Did you do any modification to the federation trust recently? like Set-FederatedOrganizationIdentifier?

 

Co-Authors
Version history
Last update:
‎Sep 19 2023 11:11 AM
Updated by: