Announcing Original Folder Item Recovery
Cumulative Update 6 (CU6) for Exchange Server 2016 will be released soon TM , but before that happens, I wanted to make you aware of a behavior change in item recovery that is shipping in CU6. Hopefully this information will aid you in your planning, testing, and deployment of CU6. Item Recovery Prior to Exchange 2010, we had the Dumpster 1.0, which was essentially a view stored per folder. Items in the dumpster stayed in the folder where they were soft-deleted (shift-delete or delete from Deleted Items) and were stamped with the ptagDeletedOnFlag flag. These items were special-cased in the store to be excluded from normal Outlook views and quotas. This design also meant that when a user wanted to recover the item, it was restored to its original folder. With Exchange 2010, we moved away from Dumpster 1.0 and replaced it with the Recoverable Items folder. I discussed the details of that architectural shift in the article, Single Item Recovery in Exchange 2010. The Recoverable Items architecture created several benefits: deleted items moved with the mailbox, deleted items were indexable and discoverable, and facilitated both short-term and long-term data preservation scenarios. As a reminder, the following actions can be performed by a user: A user can perform a soft-delete operation where the item is deleted from an Inbox folder and moved to the Deleted Items folder. The Deleted Items folder can be emptied either manually by the user, or automatically via a Retention Policy. When data is removed from the Deleted Items folder, it is placed in the Recoverable Items\Deletions folder. A user can perform a hard-delete operation where the item is deleted from an Inbox folder and moved to the Recoverable Items\Deletions folder, bypassing the Deleted Items folder entirely. A user can recover items stored in the Recoverable Items\Deletions folder via recovery options in Outlook for Windows and Outlook on the web However, this architecture has a drawback – items cannot be recovered to their original folders. Many of you have voiced your concerns around this limitation in the Recoverable Items architecture, through various feedback mechanisms, like at Ignite 2015 in Chicago where we had a panel that included the Mailbox Intelligence team (those who own backup, HA, DR, search, etc.). Due to your overwhelming feedback, I am pleased to announce that beginning with Exchange 2016 CU6, items can be recovered to their original folders! How does it work? When an item is deleted (soft-delete or hard-delete) it is stamped with the LastActiveParentEntryID (LAPEID) MAPI property (property ID 348A). By using the folder ID, it does not matter if the folder is moved in the mailbox’s hierarchy or renamed. When the user attempts a recovery action, the LAPEID is used as the move destination endpoint. The LAPEID stamping mechanism has been in place since Exchange 2016 Cumulative Update 1. This means that as soon as you install CU6, your users can recover items to their original folders! Soft-Deletion: Hard-Deletion Are there limitations? Yes, there are limitations. First, to use this functionality, the user’s mailbox must be on a Mailbox server that has CU6 installed. The user must also use Outlook on the web to recover to the original folder; neither Outlook for Windows or Outlook for Mac support this functionality, today. If an item does not have an LAPEID stamped, then the item will be recovered to its folder type origin – Inbox for mail items, Calendar for calendar items, Contacts for contact items, and Tasks for task items. How could an item not have an LAPEID? Well, if the item was deleted before CU1 was installed, it won’t have an LAPEID. And lastly, this feature does not recover deleted folders. It only recovers items to folders that still exist within the user’s mailbox hierarchy. Once a folder is deleted, recovery will be to the folder type origin for that item. Summary We hope you can take advantage of this long sought-after feature. We continue to look at ways we can improve user recovery actions and minimize the need for third-party backup solutions. If you have questions, please let us know. Ross Smith IV Principal Program Manager Office 365 Customer ExperienceDemystifying Hybrid Free/Busy: Finding errors and troubleshooting
EDIT 9/19/2023: This blog post has received significant update. In this second part of the Demystifying Hybrid Free/Busy, we will cover troubleshooting of Hybrid Free/Busy scenarios, more specifically – how and where to find an actual error that will indicate where the problem is. Before venturing forth, please make sure that you have seen Part 1 of this demystifying series! Here is the graphics we posted in the previous post; use this as a reference for users that we will be referring to when troubleshooting: Do you really have a Free/Busy issue? Usually when a user creates a new meeting in Outlook on the web (OWA) or Outlook, clicks onScheduling Assistant, adds his or her colleague to the meeting, they try to see when the user is available to meet. If they see the hash marks\\\\\\\instead of seeing if the other user is free or busy, there is an issue. Here, we do seem to have a bunch of Free/Busy issues: You can often see an error message by hovering over hash marks, however we usually find that the error is not very specific. Instead, we would need to take slightly more advanced steps to diagnose the issues by checking things like theRemote Connectivity Analyzertool, Fiddler, F12 Network tab, Outlook logging or SARA tool. Where is the actual Free/Busy error message? First, we need to understand in which direction we have a lookup problem. Please seePart 1 for discussion of directionality. Sources of logs: Remote Connectivity Analyzer tool Outlook logging SARA tool OWA F12 Network Tab Fiddler – Outlook and OWA These steps are important for us to see the relevant message error for Free/Busy issues. Once we know the error message, it’s much easier to resolve the issue. Remote Connectivity Analyzer A few things to know about this tool: Source Mailbox: the user that will be requesting the free/busy information. This will be the user that is logged in Outlook or OWA and cannot see free/busy for other people. This is also called Requester or Organizer of the meeting. Authentication type for Source Mailbox: you will choose Modern Authentication Source Mailbox credentials: you will need to authenticate with the credentials of the Source Mailbox. The tool doesn’t support Basic Authentication for Exchange Online mailboxes because this is disabled in Exchange Online. While it is still used by Exchange On-premises environments, currently, if you select Basic Authentication for the on-premises source mailbox, the test will fail before doing the actual Free/Busy process. It works if your Exchange on-premises has enabled Modern Authentication for client protocols. In conclusion, Source Mailbox login needs to be using OAuth for this test to work, regardless of where it is hosted. Target Mailbox: the user that the Source Mailbox is requesting free/busy for. This is the Attendee of the meeting. The tool simulates Outlook’s way of querying Free/Busy. If you have a free/busy issue that is only happening in OWA but not in Outlook Desktop, then this test will likely not catch the error. To be able to perform the test, you must allow connectivity for the Remote Connectivity Analyzer tool’s IP addresses. These are part of the "Microsoft 365 Common and Office Online" ranges published in theOffice 365 URLs and IP address ranges.The IPs for the Remote Connectivity Analyzer are part of the range specified as "Allow Required" (currently ID 46 in the documentation). Check https://testconnectivity.microsoft.com/Pages/ChangeList.htm for any future changes. Note that you can only insert one Target Mailbox email address per test. If you have errors for multiple target mailboxes, run multiple tests, for each user. Connectivity Test Results: With these 3 buttons on the top right corner, you can expand all the results and save them as XML or HTML files. Usually, support people appreciate these files a lot, so please do upload them in your support case workspace. When you expand the results, there are 3 important checks: Determining where the source mailbox is hosted (cloud or not). If the Mailbox is hosted in cloud, you will see something like this:IsOffice365Mailbox=True. The mailbox is hosted in Office 365. <ASURL>https://outlook.office365.com/EWS/Exchange.asmx</ASURL> If the Mailbox is not hosted in cloud, you will see something like this:IsOffice365Mailbox=False. The mailbox isn't hosted in Office 365. Determining where the target mailbox is hosted (cloud or not). Test Autodiscover for the Target Mailbox SMTP to retrieve External EWS url. Quick tip: on your side, in Windows PoweShell, you can also use the following commands to see the External EWS url of an user based on the Autodiscover call to Office 365, replace what is in Email= with your actual email addresses. Invoke-RestMethod -Uri "https://outlook.office365.com/autodiscover/autodiscover.json?Email=CLOUDUSER@CONTOSO.COM&Protocol=EWS" Invoke-RestMethod -Uri "https://outlook.office365.com/autodiscover/autodiscover.json?Email=ONPREMUSER@CONTOSO.COM&Protocol=EWS" Performing the Free/Busy Lookup. This will be Success or Failed. If it failed, look under the Additional details to see the error message. If success, be happy, maybe the issue is resolved, or not be happy as it might be an intermittent issue (which is harder to troubleshoot) or a local issue only (happening in your specific network, machine, Outlook version). In my case, I see that I have a NoFreeBusyAccessException, given by the Exchange on-premises server HHE1601. OUTLOOK Note: The Modern Outlook clients log Free/Busy information in Outlook ETL files and you won’t be able to see the Free/Busy error in plain text here. This was possible with Outlook 2010 logs, back in the old days. But this method is still useful, because you can provide the Outlook ETL log containing the error to Microsoft Support to parse it for you and help you fix it also. If you want to see the error for yourself, check the Fiddler method. For the Outlook F/B error, we need to first enable Outlook logging and after this we will need to reproduce the issue (\\\\\\). After repro, we will collect the Outlook logs. Steps: Enable Outlook logging:Followthis KB articleand check the “Enable troubleshooting logging (this requires restarting Outlook)” option. Restart Outlook. Reproduce the issue for the non-working free/busy direction.Suppose Free/Busy direction not working is cloud to on-premises, you will be logged on as a cloud user (Source Mailbox), go to Calendar tab, New Meeting, Scheduling Assistant, add some on-premises users to a meeting until you see the hash marks (instead of Free/Busy information). You do not need to save or send a meeting request. Collect the Outlook-#####.etl log from%temp%\Outlook Loggingfolder (referencehere). You would need to send the ETL file to Microsoft Support to get it analyzed as we are parsing this log with an internal tool. You might not know this, but Hybrid free/busy support cases are free of charge! Of course, you can still use the other methods (fiddler for Outlook/OWA or browser for OWA) to see Free/Busy error yourself, however we (Support) might ask you additionally to get this log as well for a further dive into the Free/Busy errors. SARA I would also like to mention that there is a Free/Busy troubleshooter in Beta version, incorporated into SARA tool (Microsoft Support and Recovery Assistant for Office 365) which you can download it from here :https://diagnostics.outlook.com/#/ Open SARA and select Outlook scenario, click Next, then selectI’m having problems with my calendar, input email address and password of the source mailbox (cloud mailbox if direction not working is cloud > on-premises) and then selectI can’t see when someone is free or busy. Due to the underlying complexity of it all, this is not a completely reliable way of determining the cause of free/busy issues in Hybrid Deployments, but it is a good start when troubleshooting. This F/B test from SARA covers mostly cloud to cloud scenarios but I recommend it here because it does connectivity and additional checks on tenant, licensing and Autodiscover. And sometimes it shows the underlying Free/Busy error message. Here are some screenshots with the SARA process: After the Office 365 readiness checks, the tool will ask you for the email address of the Target Mailbox: In the failed results, expand the Support Message and User Message: OWA / Outlook on the web F12 Network Tab Cloud OWA F12 Network tab You need to login to OWA as the source mailbox, hit F12 (Developer Tools for browser) and select the Network Tab. You would then lookup Free/Busy for the target mailbox (reproduce the issue). If the source mailbox is hosted in Cloud, to look for the F/B here, you can find the Search Icon and type there “GetSchedule” or find the Filter Icon and type “graphql”, then look at Response or Preview tab to see the error message by expanding GetSchedule until you reach to the error. (click thumbnail to view larger) If the Source Mailbox is hosted in Exchange On-Premises, you would look after GetUserAvailabilityInternal: Fiddler –Outlook or OWA You would need to download and install Fiddler tool from the internet, enable HTTPS decryption in Fiddler and then reproduce the Free/Busy issue in Outlook or OWA or both. Fiddler - Exchange Online Source Mailbox logged in Outlook desktop. Look for “GetUserAvailability” calls and then on the right side, you have Request on the top and Response on the bottom. Switch to XML tabs for a nicer view. In the Request you will see the attendees’ email addresses and, in the Response, you will have ResponseMessage with ResponseClass=Error or ResponseClass=Success. Fiddler – Exchange Online Source Mailbox logged in OWA. In Fiddler, you can check in the Request pane, under Raw tab the ClientRequestID, you can for example search after this specific value in your on-premises Exchange server logs: IIS W3SVC2 logs, HTTPProxy EWS logs and EWS logs (more information on these logs, location and extracts, later in the article). Example here from a lab: ClientRequestID: {72741DFF-A6AC-402B-991B-C6B5D56B1422} Date: Mon, 11 Sep 2023 19:01:25 GMT If you are fan of SQL language, you can use a tool like Log Parser Studio and search through these logs, for example, here is a query on the ClientRequestID from earlier: SELECT DateTime, ClientRequestID, RequestID, UserAgent, SoapAction, ErrorCode, GenericErrors, GenericInfo, FileName FROM '[LOGFILEPATH]' WHERE ClientRequestID LIKE '%{72741DFF-A6AC-402B-991B-C6B5D56B1422}%' You can also use findstr.exe utility to look for the client request id or other keywords like the requester’s email address or “CrossForest”. Example of command: findstr.exe /I /S "{72741DFF-A6AC-402B-991B-C6B5D56B1422}" *.log When troubleshooting Free/Busy issues, the following on-premises logs can be very useful, especially for Cloud to On-Premises Free/Busy direction. IIS logs Default Web Site (DWS) Path: %SystemDrive%\inetpub\logs\LogFiles\W3SVC1 Path example: C:\inetpub\logs\LogFiles\W3SVC1 Extract of Autodiscover and EWS log entries with IOC Enabled in IIS W3SVC1 logs: Autodiscover – OAUTH (autodiscover.svc without /WSSecurity) 2016-01-06 17:45:27 10.0.0.5 POST /autodiscover/autodiscover.svc &CorrelationID=<empty>;&ClientId=QNFNHKEEKYENCJITQQ&cafeReqId=7972d1fc-a9d9-44c6-8851-480d3601cbd7; 443 S2S~00000002-0000-0ff1-ce00-000000000000 132.245.65.28 ASAutoDiscover/CrossForest/EmailDomain//15.01.0361.007 200 0 0 109 EWS – OAUTH (exchange.asmx without /WSSecurity) 2016-01-06 17:45:27 10.0.0.5 POST /ews/exchange.asmx &CorrelationID=<empty>;&ClientId=WSIVGUUAUWWRFACJBWDA&cafeReqId=6ce8864c-74a0-4ad2-a3dc-7b69e0415403; 443 <unverified>actas1(sip:joe@contoso.com|smtp:joe@contoso.com|upn:joe@contoso.com) 132.245.65.28 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 703 Example of EWS entry with Organization Relationship Enabled in IIS W3SVC1 logs:EWS – DAUTH (exchange.asmx with /WSSecurity) 2016-01-06 18:04:41 10.0.0.5 POST /ews/exchange.asmx/WSSecurity &CorrelationID=<empty>;&ClientId=VOMGJKAWURSVKOXQLBVA&cafeReqId=18fd3a2e-7b1c-4828-8943-6b20912e2e44; 443 - 132.245.65.28 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 296 IIS logs Exchange BackEnd (BE) Path: %SystemDrive%\inetpub\logs\LogFiles\W3SVC2 Path example: C:\inetpub\logs\LogFiles\W3SVC2 Example of EWS entry with Organization Relationship Enabled (DAUTH) in IIS W3SVC2 logs: 2016-01-06 18:04:41 fe80::f17f:beef:a5e3:7d3c%25 POST /ews/exchange.asmx/WSSecurity - 444 - fe80::f17f:beef:a5e3:7d3c%25 ASProxy/CrossForest/EmailDomain//15.01.0361.007 200 0 0 93 HTTPProxy logs for Autodiscover Path: %ExchangeInstallPath%Logging\HttpProxy\Autodiscover Path example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Autodiscover Example of Autodiscover entry with Organization Relationship Enabled (DAUTH) 2016-01-06T18:05:20.552Z,bcdfbed5-f11f-4250-a616-e38cb475cd3f,15,0,1104,2,,Autodiscover,autodiscover.contoso.com,/autodiscover/autodiscover.svc /WSSecurity,,,false,,contoso.com,Smtp~joe@contoso.com,ASAutoDiscover/CrossForest/EmailDomain/ /15.01.0361.007,132.245.65.28,exch-2013,200,200,,POST,Proxy,exch-2013.contoso.com,15.00.1104.000,IntraForest,AnchorMailboxHeader-SMTP,[…],BeginRequest=2016-01-06T18:05:20.192Z;CorrelationID=<empty>;ProxyState-Run=None;FEAuth=BEVersion-1941996624;NewConnection=fe80::f17f:beef:a5e3:7d3c%25&0; HTTPProxy logs for EWS Path: %ExchangeInstallPath%Logging\HttpProxy\Ews Path example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews Example of EWS entry with Organization Relationship Enabled (DAUTH): 2016-01-06T18:04:41.490Z,4757ab2c-8ccc-4d1a-ae39-0780ecc8eabb,15,0,1104,2,{02CD833F-18AB-413A-83CB-0E86F4DA5362},Ews,mail.contoso.com,/ews/exchange.asmx/WSSecurity,,,false,,contoso.com, Smtp~joe@contoso.com,ASProxy/CrossForest/EmailDomain//15.01.0361.007,132.245.65.28,exch-2013,200,200,,POST,Proxy,exch-2013.contoso.com,15.00.1104.000,IntraForest,AnchorMailboxHeader-SMTP,[…],BeginRequest=2016-01-06T18:04:41.380Z; EWS logs Path: %ExchangeInstallPath%Logging\Ews Path example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\Ews Example of EWS entry with Organization Relationship Enabled (DAUTH): 2016-01-06T18:04:41.490Z,4757ab2c-8ccc-4d1a-ae39-0780ecc8eabb,15,0,1104,2,{02CD833F-18AB-413A-83CB-0E86F4DA5362}, External,true,jane@contoso.mail.onmicrosoft.com,, ASProxy/CrossForest/EmailDomain//15.01.0361.007,Target=None;Req=Exchange2012/Exchange2013; ,132.245.65.28,exch-2013,exch-2013.contoso.com,GetUserAvailability,200,12150,,,,,,ebd34d71ac7342c19d947d881db4ad55,f866c73e-6c91-475e-bdec-0428bdeaa423,PrimaryServer; Requester=jane@contoso.mail.onmicrosoft.com; Failures=0 Event Viewer Application logs on Exchange ServerReferenceshereandhere. Example of Event ID 4002 for MSExchange Availability: Log Name: Application Source: MSExchange Availability Event ID: 4002 Task Category: Availability Service Level: Error Description: Process 4568: ProxyWebRequest CrossSite from S-1-5-21-391720751-1508397712-925700815-508779 tohttps://hybrid.contoso.com/ews/exchange.asmxfailed. Caller SIDs: NetworkCredentials. The exception returned is Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequestProcessingException: System.Web.Services.Protocols.SoapException: You have exceeded the available concurrent connections for your account. Try again once your other requests have completed. at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) IIS tracing for the error code in the IIS logsReferencehere. Free/Busy errors and fixes Based on cumulative support team experience, we created a table (see the attachment to this post) with Free/Busy errors encountered so far and their possible resolutions. We cannot cover all possible scenarios and errors even though we have a good-sized list. This is meant to illustrate ways we can resolve specific errors and these suggestions might not work for you even if you have the same error. If you know the exact Free/Busy error that you get and checked configuration as discussed in part 1 of this series, this is already a tremendous progress, and this will help us resolve your issue faster. Of course, you can follow these suggestions on your own as most of the actions are harmless but if you don’t feel confident in troubleshooting on your own or you fear that actions are dangerous or irreversible, please contact us. Free/Busy Errors discussed in the attached document (FB_Errors_FixesV7): “An internal server error occurred. The operation failed” LID: 59916. 500 Internal Server error. "The remote user mailbox must specify the the explicit local mailbox in the header" "An error occurred when verifying security for the message" "Unable to connect to the remote server" “Autodiscover failed for email address <> with error ‘The request failed with HTTP status 404: Not Found’ ” “The request failed with HTTP status 401: Unauthorized - The user specified by the user-context in the token is ambiguous” LID: 43532 "An existing connection was forcibly closed by the remote host - An unexpected error occurred on a receive " "An existing connection was forcibly closed by the remote host - An unexpected error occurred on a send ” "Configuration information for forest/domain could not be found in Active Directory" "Proxy web request failed.,inner exception: The request failed with HTTP status 401: Unauthorized." "The response from the Autodiscover service at 'https://autodiscover/autodiscover.svc/WSSecurity' failed due to an error in user setting 'ExternalEwsUrl'. Error message: InvalidUser." LID: 33676 “The caller does not have access to free/busy data" LID: 47652 LID: 44348 “The request failed with HTTP status 403: Forbidden (The server denied the specified Uniform Resource Locator (URL). “ LID: 43532 “Unable to resolve e-mail address user@notes.domain.com to an Active Directory object” LID: 57660 “An error occurred when processing the security tokens in the message.” LID: 59916 “The cross-organization request for mailbox yyy@contoso.com is not allowed because the requester is from a different organization” LID: 39660 “The request failed with HTTP status 401: Unauthorized - Microsoft.Exchange.Security.OAuth.OAuth TokenRequestFailedException: Missing signing certificate “ “The application is missing a linked account for RBAC roles, or the linked account has no RBAC role assignments, or the calling users account is logon disabled” “The entered and stored passwords do not match“ “The password has to be changed.” “The password for the account has expired” or “Provision is needed before federated account can be logged in” “The request timed out” “The specified member name is either invalid or empty” “The result set contains too many calendar entries” LID: 54796 “The request failed with HTTP status 401: Unauthorized - The token has an invalid signature.” “The request failed with HTTP status 401: Unauthorized - Client assertion contains an invalid signature. [Reason - The key was not found., Thumbprint of key used by client: '<>’ “ “Proxy web request failed., inner exception: Response is not well-formed XML “ “Failed to communicate with https://login.microsoftonline.com/extSTS.srf., inner exception: Unable to connect to the remote server” “Autodiscover failed for E-Mail Address <> with error System.Net.WebException: The remote name could not be resolved: '<>'” “Failed to get ASURL. Error 8004010F” “Proxy web request failed. , inner exception: System.Net.WebException: The request failed with the error message: -- <head><title>Object moved” “The request was aborted: Could not create SSL/TLS secure channel.” “The user specified by the user-context in the token does not exist.";error_category="invalid_user“ “The hostname component of the audience claim value 'https://<>’ is invalid";error_category="invalid_resource“ “Proxy web request failed. , inner exception: System.Net.WebException: The request failed with HTTP status 503: Service Unavailable” “Proxy web request failed. , inner exception: System.Net.WebException: The request failed with HTTP status 504: Gateway Timeout.” Thanks to all that contributed to this content: Ray Fong, Nino Bilic, Tim Heeney, Greg Taylor and Brian Day. Mirela Buruiana
