Introducing the IMAP Migration Troubleshooter
Situation If transitioning your organization from a non-Exchange system such as Google or Lotus Notes to Office 365, you typically need to follow the IMAP migration path. As diagnosing and remediating any issues you might run into during such a migration can be difficult for people unfamiliar with the matter, we worked with our Support teams to provide some guidance for exactly such cases and packaged it for you in a wizard-like package. Introducing the IMAP Migration Troubleshooter IMAP Migrations provide organizations with an effective way to move email from any environment that supports the IMAP protocol. This is usually used for any non-Exchange source email system. If Exchange is the source you would most likely opt for the Staged, Cutover, or Hybrid migration path. To help you with troubleshooting, we have released a new guided walkthrough (GWT) for IMAP migrations. This link will soon be surfaced in various KB articles as well as the support ticket creation process via the Office 365 portal. The intent of this GWT is to take a tenant administrator step-by-step though the common troubleshooting tasks to solve their IMAP migration issues. We took the most common migration failure scenarios and put them into this easy to follow guide. https://aka.ms/IMAPMigrationGWT Feedback If you see any issues with the IMAP migration troubleshooter or think there are scenarios that should be added or improved, please let us know at MigrationGWT@microsoft.com. Special thanks to all that contributed to the creation of the GWT: Kevyn Pietsch, Nagesh Mahadev, Timothy Heeney, Shawn Sullivan, and to the writers and KE team for assisting with collaboration and coordination efforts: Charlotte Raymundo and Sharon Shen. If you are looking for assistance on any troubleshooting Hybrid migrations, see the Exchange Online Migration Guided Walk Through (GWT). Kevyn Pietsch, Nagesh Mahadev, Timothy HeeneyDemystifying Hybrid Free/Busy: what are the moving parts?
Hybrid Free/Busy is one of those things that many people do not fully understand. If everything works well, the complexity is hidden from view and people working in various parts of organization can seamlessly work together. But if things go wrong… you will appreciate deeper understanding of what makes it work. This is why we wanted to create the blog post series on the subject. In this article, we will discuss how Free/Busy works in an Exchange Hybrid configuration. In next blog post, you will learn what we (Microsoft Support) see as the most common problems along with how we go about diagnosing those (often) complex issues. Hope you like reading, because there is a lot to cover! So, what is Free/Busy? Free/Busy is a feature that allows you to see when others are free (their calendar shows availability), busy (their calendar shows them as busy), or even Out of Office, or Something Else (tentative or working away) so that you can find an appropriate time for your meetings. Calling it all “Free/Busy/OOF/Something-Else” didn’t sound so cool to marketing hence “Free/Busy”. In a Hybrid deployment, we usually have some mailboxes in Exchange On-Premises and some mailboxes in Exchange Online (users are in different premises) and this has to work there too. One of the most important parts in Hybrid Configurations is the Federation Trust and many features, including Free/Busy can rely upon this. A quick overview of a hybrid deployment with a focus on Federation Trust components In Contoso – On-Premises side we have on-premises Exchange Servers and mailboxes. We also have a federation trust with the Microsoft Federated Gateway (MFG) - now called Azure Authentication System. A Federation trust is not set by default for Exchange On-Premises organizations and we can either create it manually or run the Hybrid Configuration Wizard (HCW) which will do this for us. If you don’t have already a federation trust established on-premises (usually for purposes to share F/B with another on-premises organization) and you plan for a Hybrid deployment, then we strongly recommend you run HCW to automatically create the Federation Trust. In Contoso – Cloud Side there are Office 365 Exchange Online servers, your Office 365 tenant and mailboxes migrated from on-premises. A Federation trust with the MFG is automatically created for cloud-only-based Exchange organizations whom do not have a Hybrid relationship to an on-premises organization. If you run Get-FederationTrust cmdlet in Exchange Online PowerShell (see here how to connect to Exchange Online PowerShell) you would see two trusts: “WindowsLiveId” (Consumer Instance of Microsoft Federation Gateway) and “MicrosoftOnline” (Business Instance of Microsoft Federation Gateway) . Note: As a troubleshooting tip, you might want to make sure the Application Identifier is “260563” and the Application Uri is “outlook.com” for both; in case you have a different App ID (292841) and a different App URI (outlook.live.com) for a Cloud trust, this means your tenant has an old reference pointing to MFG and most probably Free/Busy from on-premises to cloud would fail with a quite generic 401 Unauthorized error. If you were to find yourself in a such situation, please open a support case with Microsoft to get it resolved. About Free/Busy lookup directionality To guide you to the correct troubleshooting steps we first need to determine what direction Free/Busy queries are failing in. Understanding how Free/Busy works in general and the direction that lookups are failing can greatly simplify the troubleshooting steps. Simply said, there are 2 possible Free/Busy directions (referring to our example above): When Joe looks up Jane’s Free/Busy, the Free/Busy direction is On-premises to Cloud. When Jane looks up Joe’s Free/Busy, the Free/Busy direction is Cloud to On-Premises Now that you are familiar with the Free/Busy directions, we should take a moment to discuss how it all works. Let me set the stage for the below diagram: joe@contoso.com has his mailbox in Exchange on-premises and jane@contoso.com has been moved to Exchange Online. In Exchange on-premises, joe@contoso.com is a Mailbox User. Because Directory Synchronization is a requirement for Hybrid Deployments, joe@contoso.com (mailbox) is synced to the cloud and represented as a Mail Enabled User (MEU) object in Cloud with Target Address (TA) of joe@contoso.com. jane@contoso.com was once a Mailbox User in Exchange on-premises. Her mailbox was then migrated to Exchange Online. Jane now has a Mailbox User in Exchange Online and is represented as a Remote Mailbox (RM) object in Exchange On-Premises. Jane has a Primary SMTP address (SMTP: jane@contoso.com) and a secondary smtp address (smtp: jane@contoso.mail.onmicrosoft.com). Notice how we refer to a primary email address (SMTP) and to a secondary email address (smtp). According to TechNet, the difference between primary and secondary addresses is that the primary address serves as the return e-mail address. When mail is received from a recipient, the primary address determines which address the mail appears to have come from. Recipients can receive mail sent to any of the addresses associated with them (primary or secondary). The Target Address (TA) of the on-premises Remote Mailbox object is represented as jane@tenant.mail.onmicrosoft.com (secondary smtp) and this is crucial for Autodiscover and email routing from on-premises to cloud. A successful Autodiscover query is important in the Free/Busy process as it provides us with the Availability Service URL of the user which is the External EWS URL of an Exchange Organization. This TechNet article gives us more information on this <domain>.mail.onmicrosoft.com secondary email address: The wizard (HCW) adds an accepted domain to the on-premises organization for hybrid mail flow and Autodiscover requests for the cloud organization. This domain, referred to as the coexistence domain, is added as a secondary proxy domain (contoso.mail.onmicrosoft.com) to any email address policies which have PrimarySmtpAddress templates for domains selected in the Hybrid Configuration wizard(contoso.com). By default, this domain is <domain>.mail.onmicrosoft.com. Table below illustrates the hybrid user objects discussed above as well as how to look them up. On-premises commands = Exchange Management Shell Cloud commands = Exchange Online PowerShell Typical Hybrid user objects configuration On-premises commands On-premises objects Corresponding cloud objects Cloud commands ON-PREM MAILBOX USER MAIL ENABLED USER IN CLOUD Get-mailbox Joe@contoso.com | FT PrimarySMTPaddress On-premises mailbox user with SMTP: Joe@contoso.com Cloud mail enabled user with SMTP: Joe@contoso.com Get-MailUser Joe@contoso.com | FT PrimarySMTPAddress (Get-mailbox Joe@contoso.com). EmailAddresses On-premises mailbox user has a secondary smtp address of @contoso.mail. onmicrosoft.com configured by HCW (Email Address Policies) Cloud mail enabled user has this secondary smtp Joe@contoso.mail. onmicrosoft.com (Get-mailuser Joe@contoso.com). EmailAddresses (Get-mailbox Joe@contoso.com). EmailAddresses On-premises mailbox user has a secondary smtp Joe@contoso.mail. onmicrosoft.com Cloud mail enabled user has an ExternalEmailAddress Joe@contoso.com Get-MailUser Joe@contoso.com |FT ExternalEmailAddress REMOTE MAILBOX ON-PREM MAILBOX USER IN CLOUD Get-RemoteMailbox Jane@contoso.com |FT primarySMTPaddress Remote mailbox SMTP: Jane@contoso.com Mailbox User SMTP: Jane@contoso.com Get-Mailbox Jane@contoso.com |FT PrimarySMTPAddress Get-RemoteMailbox Jane@contoso.com | FT RemoteRoutingAddress Remote mailbox TargetAddress: Jane@contoso.mail. onmicrosoft.com Mailbox User smtp: Jane@contoso.mail. onmicrosoft.com (Get-Mailbox Jane@contoso.com). EmailAddresses Now that we understand user objects and their relevant properties, we should come back to Free/Busy directions. Between the on-premises organization and cloud organization there are bidirectional Organization Relationships and/or bidirectional Intra Organization Connectors (for Exchange 2013 or later) that are created during Hybrid Configuration. The source of these Relationships / Connectors plays an important role in the F/B directionality. The Free/Busy directions are: Direction On-premises to Cloud You are logged in to Outlook on the Web or Outlook on Windows as an on-premises user (joe@contoso.com), you’re your mailbox hosted on your on-premises Exchange server. As an on-premises user you wish to have a meeting with the cloud user (jane@contoso.com) but will first have to check their availability for the meeting. Let’s walk through the process: 1. Joe creates a meeting request and adds Jane as an attendee. 2. The on-premises Exchange server in Contoso determines (usually based on Target Address of the mail-enabled user) that Jane is not a local mailbox and has a different domain name of contoso.mail.onmicrosoft.com set as the Target Address. 3. The Availability Service on the on-premises Exchange server (Client Access Server if Ex2010 or Mailbox Server if 2013/2016) in Contoso then checks to see if there is a path to query Jane’s Free/Busy information for Contoso’s cloud side. First, we check if we have an Intra-Organization Connector (1) with the domain name of contoso.mail.onmicrosoft.com (assuming the Exchange server is version 2013 or later). If an IOC does not exist, then we look to see if an Organization Relationship (2) is configured by looking for the domain name of contoso.mail.onmicrosoft.com in the list of any existing Organization Relationships. If neither an IOC nor an Organization Relationship for the domain contoso.mail.onmicrosoft.com exists, then we look for the domain name contoso.mail.onmicrosoft.com as an Availability Address Space (3). 4. Assuming we would have an Exchange 2010-only environment in the on-premises and we’ve ran HCW successfully, we should expect to see both an Organization Relationship as well as a Federation Trust which combined is the second option from step #3. The Target ApplicationURI in the Contoso on-premises Organization Relationship is set to outlook.com, which is an identifier for the Contoso Cloud organization trust in the Azure Authentication System. A request is then made to the Azure Authentication System for a delegation token that will be accepted by Contoso Cloud Organization. 5. Once the delegation token is received back from the Azure Authentication System, the Exchange server in Contoso on-premises sends an Autodiscover request to Exchange Online, and upon a successful Autodiscover response will send an EWS request to Exchange Online for Jane’s availability information. 6. The Exchange Online server validates the token provided by the Azure Authentication System and once confirmed, returns the requested Free/Busy data for Jane’s mailbox. Here is a flowchart which illustrates the Free/Busy lookup process: Direction Cloud to on-premises You are logged in to Outlook on the Web or Outlook on Windows as a cloud user (jane@contoso.com), whose mailbox is in Exchange Online. Jane wants to have a meeting with an on-premises user (joe@contoso.com), but must first check their availability for the meeting. 1. Jane creates a meeting request and adds Joe as an attendee 2. The Exchange Online servers in Contoso-cloud side organization determine (usually based on target address of a mail-enabled user) that Joe is not a local mailbox and has a domain name of contoso.com set as the target address. 3. The Availability Service on Exchange Online servers checks to see if there is a path for it to find the Free/Busy information for Contoso on-premises side organization. First, we check if we have an Intra-Organization Connector (1) with the domain name of contoso.com (assuming the Exchange server is 2013 or later on-premises). If an IOC does not exist, then we look to see if an Organization Relationship (2) is configured by looking for the domain name of contoso.com in the list of any existing Organization Relationships. If neither an IOC nor an Organization Relationship for the domain contoso.com exists, then we then look for the domain name contoso.com as an Availability Address Space (3). 4. Assuming we would have an Exchange 2010-only environment in the on-premises and we’ve ran HCW successfully, we should expect to see both an Organization Relationship as well as a Federation Trust which combined is the second option from step #3. The Target ApplicationURI in the Contoso Cloud Organization Relationship is set to FYDIBOHF25SPDLT.contoso.com, which is an identifier for the Contoso on-premises organization trust in the Azure Authentication System. A request is then made to the Azure Authentication System for a delegation token which will be accepted by Contoso on-premises organization. 5. Once the delegation token is received back from the Azure Authentication System, the Exchange Online server in Contoso cloud sends an Autodiscover request to the on-premises Exchange Servers and upon receipt of a successful Autodiscover response it will then send an EWS request for Joe’s availability. 6. The Contoso on-premises Exchange server validates the token provided by the Azure Authentication System and once confirmed, returns the requested Free/Busy data for Joe’s mailbox. Hybrid Free/Busy lookup order To summarize, the following order would be used when processing for Free/Busy lookups from cloud to on-premises: Look for IntraOrganizationConnector (OAUTH) If there is no IntraOrganizationConnector or if it is disabled, then look for Organization Relationship (DAUTH) If neither of these are found or they’re disabled, then look for Availability Address Space. Availability Address Space is normally only used for Lotus Notes organizations. If Free/Busy lookup is getting all the way down to using the final Availability Address Space option for cloud to on-premises lookups in a hybrid deployment, then this would suggest there are configuration issues which must be repaired. The order for Free/Busy lookups from on-premises to cloud is almost the same with some exceptions: If we have Exchange 2007 servers in coexistence with Exchange 2010/2013, we use Availability Address Space from Exchange 2007 to Exchange 2010/2013 and then Exchange 2010/2013 will use Org Relationship (Ex2010) or IntraOrganization Connector (Ex2013) to Cloud. If we have Exchange 2010 with Exchange 2013/2016 and OAuth is enabled, Exchange 2010 will use Availability Address Space to Exchange 2013/2016 and then 2013/2016 will use the IntraOrganization Connector to Cloud. If we have Exchange 2010 with Exchange 2013/2016 and OAuth is not enabled, Exchange 2010 will not send the request to the Exchange 2013/2016. Instead Exchange 2010 will use Organization Relationship to Cloud for Free/Busy. Exchange 2013 Cu5+ and Exchange 2016 organizations without coexisting with legacy Exchange servers will use by default IntraOrganization Connectors from On-Premises to Cloud (normal process) This table summarizes which components (Get-IntraorganizationConnector / Get-OrganizationRelationship / Get-AvailabilityAddressSpace) are present (YES) or not (NO) in the Exchange Organization depending on the Exchange Server versions on-premises in a Hybrid deployment. Free/Busy component matrix Exchange Hybrid Environment Intra Organization Connector (IOC) Organization Relationship Availability Address Space Pure Exchange 2016 Hybrid YES (created automatically by HCW) YES YES Pure Exchange 2013 (CU5+) Hybrid YES (created automatically by HCW) YES YES Pure Exchange 2010 Hybrid NO YES YES Mixed Exchange 2007 + Exchange 2010 Hybrid NO - Ex2010 NO - Ex2007 YES - Ex2010 NO - Ex2007 YES - Ex2010 YES - Ex2007 Mixed Exchange 2007 + Exchange 2013 Hybrid YES - Ex2013 (if created manually) NO - Ex2007 YES - Ex2013 NO - Ex2007 YES - Ex2013 YES - Ex2007 Mixed Exchange 2010 + Exchange 2013 Hybrid YES - Ex2013 (if created manually) NO - Ex2010 YES - Ex2013 YES - Ex2010 YES - Ex2013 YES - Ex2010 Mixed Exchange 2010 + Exchange 2016 Hybrid YES - Ex2016 (if created manually) NO - Ex2010 YES - Ex2016 YES - Ex2010 YES - Ex2016 YES - Ex2010 Mixed Exchange 2013 + Exchange 2016 Hybrid YES - Ex2016 (created automatically by HCW) YES - Ex2013 YES - Ex2016 YES - Ex2013 YES - Ex2016 YES - Ex2013 Note: OAuth is by default configured by HCW in Exchange 2013 CU5 and above (including Exchange 2016) organizations. If Exchange 2013 / Exchange 2016 servers coexist with Exchange 2010 or older then OAUTH is not configured by default by the HCW but can be manually configured. Note: Exchange 2013 CU5 is considered old and unlike wine, the finest CU is always the most recent one. You probably have heard about our n and n-1 supported version statement in Hybrid Deployments and can read more about it here and here. Which Free/Busy lookup method are you using? I won't go into all the details about the two types of authentication (OAUTH vs DAUTH) in this post, but I recommend reading this blog post at bedtime: Deep Dive: How Hybrid Authentication Really Works. As explained there, OAuth is an open-standards based model which is more preferred to a customized model. You can quickly determine if you are using OAuth or not by running: Get-IntraOrganizationConnector cmdlet in Exchange Management Shell (for F/B direction On-Premises to Cloud) and in Exchange Online PowerShell (for F/B direction Cloud to On-Premises). If you see Enabled = True, then you are using OAuth or the system should at least be trying to. Note: We also need to make sure that the Target Domain is present on the Intra Organization Connector or Organization Relationship when deciding if using OAUTH or DAUTH. I have created two flowcharts to show you the logic of using OAUTH vs DAUTH in a F/B lookup process for each F/B direction. The user’s setup remains still the same as shown in previous examples and I am reposting the diagram here so that you don’t need to scroll up. 1. Joe’s Exchange server deciding whether to use Oauth or Dauth : 2. Jane’s Exchange Online server deciding whether to use Oauth or Dauth Now that you know which method is being used (or at least which should be attempted) and we know the direction Free/Busy is failing, we can see if you have everything configured correctly. In most cases these configurations are handled by the HCW and should be accurate and you can re-run the HCW to get things back to a good configuration. If Exchange On-Premises > Exchange Online Free/Busy is failing for all users, you would first check Intra Organization Connector or the Organization Relationship from on-premises. If Exchange Online > Exchange On-Premises Free/Busy is failing for all users, you would first check Intra Organization Connector or the Organization Relationship from Exchange Online. Below, you will see a sample of the expected configuration for Intra Organization Connectors and Org Relationships from both sides (on-premises and cloud). This Baseline configuration was documented by my co-worker Ray Fong (Support Escalation Engineer) and I am very happy to have it when I troubleshoot Free/Busy issues. Exchange Online side Use this article to connect to Exchange Online PowerShell. INTRA ORGANIZATION CONNECTOR IN CLOUD Get-IntraOrganizationConnector | fl TargetAddressDomains,DiscoveryEndpoint,Enabled TargetAddressDomains : {contoso.com} DiscoveryEndpoint : https://autodiscover.contoso.com/autodiscover/autodiscover.svc * Enabled : True Note: In practice, the On-Premises Discovery Endpoint (Autodiscover) is more likely to be found in the format https://mail.contoso.com/autodiscover/autodiscover.svc because of the EWS External URL, so pay attention to this Autodiscover URL and replace it if needed with the autodiscover.yourdomain.tld on the IOC present in the Cloud Side (Reference Set-IntraOrganizationConnector) Get-IntraOrganizationConfiguration | fl OnPremiseTargetAddresses OnPremiseTargetAddresses : {contoso.com} TargetAddressDomains - This should be your federated domains. Example: contoso.com You can find the domains name by cross-checking Exchange Online's (Get-IntraOrganizationConfiguration).OnPremiseTargetAddresses TargetDiscoveryEndpoint - This should be the address of the On-Premises Autodiscover Endpoint. Example: https://autodiscover.contoso.com/autodiscover/autodiscover.svc/.If you paste the URL in IE, you should receive a regular windows authentication security prompt Enabled - This must be True. Exchange On-Premises side (use Exchange Management Shell) INTRA ORGANIZATION CONNECTOR IN ON-PREM Get-IntraOrganizationConnector | fl Name,TargetAddressDomains,DiscoveryEndpoint,Enabled Name : ExchangeHybridOnPremisesToOnline TargetAddressDomains : {contoso.mail.onmicrosoft.com} DiscoveryEndpoint : https://outlook.office365.com/autodiscover/autodiscover.svc Enabled : True TargetAddressDomains - This should be your tenant.mail.onmicroosft.com name. Example: 'contoso.mail.onmicrosoft.com' TargetDiscoveryEndpoint - This should be the address of EXO Autodiscover endpoint. Example: https://outlook.office365.com/autodiscover/autodiscover.svc Enabled - This must be True. Exchange Online side Use this article to connect to Exchange Online PowerShell. ORGANIZATION RELATIONSHIP IN CLOUD Get-OrganizationRelationship "Exchange Online to on premises Organization Relationship" | fl DomainNames,FreeBusy*,Target*,Enabled DomainNames : {contoso.com} FreeBusyAccessEnabled : True FreeBusyAccessLevel : LimitedDetails FreeBusyAccessScope : TargetApplicationUri : FYDIBOHF25SPDLT.contoso.com TargetSharingEpr : TargetOwaURL : TargetAutodiscoverEpr : https://autodiscover.contoso.com/autodiscover/autodiscover.svc/WSSecurity Enabled : True DomainNames - This should be your federated domains. Example: contoso.com You can find the domains name by cross-checking On-Premises' (Get-FederatedOrganizationIdentifier).Domains TargetAutodiscoverEPR - This should be the address of the On-Premises Autodiscover Endpoint. Example: https://autodiscover.contoso.com/autodiscover/autodiscover.svc/WSSecurity. If you paste the URL in IE, you should receive a regular windows authentication security prompt TargetSharingEPR - Ideally this is blank. If it is set, it should be the On-Premises Exchange servers EWS ExternalUrl endpoint. Example: https://server.contoso.com/EWS/Exchange.asmx You can find the URL by cross-checking On-Prem's Get-WebServicesVirtualDirectory ExternaUrl. If you paste the URL in IE with /WSSecurity at the end (https://server.contoso.com/EWS/Exchange.asmx/WSSecurity), you should receive a regular windows authentication security prompt TargetApplicationURI - This must match the ApplicationUrI from On-Prem. Example: FYDIBOHF25SPDLT.contoso.com You can find the value by cross-checking On-Prem's (Get-FederationTrust).ApplicationUri FreeBusyAccessEnabled - This must be True. FreeBusyAccessLevel - This should be either AvailabilityOnly or LimitedDetails. AvailabilityOnly: Free/Busy access with time only LimitedDetails: Free/Busy access with time, subject, and location FreeBusyAccessScope - This is typically blank. The FreeBusyAccessScope parameter specifies a security distribution group in the internal organization that contains users that can have their Free/Busy information accessed by an external organization. Enabled - This must be True. Exchange On-Premises side (use Exchange Management Shell) ORGANIZATION RELATIONSHIP IN ON-PREM Get-OrganizationRelationship "On Premises to Exchange Online Organization Relationship" | fl DomainNames,FreeBusy*,Target*,Enabled DomainNames : {contoso.mail.onmicrosoft.com} FreeBusyAccessEnabled : True FreeBusyAccessLevel : LimitedDetails FreeBusyAccessScope : TargetApplicationUri : Outlook.com TargetSharingEpr : TargetOwaURL : https://outlook.com/owa/contoso.onmicrosoft.com TargetAutodiscoverEpr : https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc/WSSecurity Enabled : True DomainNames - This should be your tenant.mail.onmicrosoft.com name. Example: contoso.mail.onmicrosoft.com TargetAutodiscoverEpr - This should be a valid Exchange Online Autodiscover endpoint. Example: https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc/WSSecurity or https://pod51038.outlook.com/autodiscover/autodiscover.svc/WSSecurity [<-- A pod address] You can find the value from TargetAutodiscoverEpr of Get-FederationInformaiton -DomainName tenantname.mail.onmicrosoft.com -BypassAdditionalDomainValidation | fl TargetSharingEPR - Ideally this is blank. If it is set, it should be Office 365 EWS endpoint. Example: https://outlook.office365.com/EWS/Exchange.asmx TargetApplicationURI - This must be outlook.com if this is for organization relationship of the cloud tenant. For non-cloud organization relationship, this must match (Get-FederationTrust).ApplicationUri of the On-Prem trust of the other organization. FreeBusyAccessEnabled - This must be True. FreeBusyAccessLevel - This should be either AvailabilityOnly or LimitedDetails. AvailabilityOnly: Free/Busy access with time only LimitedDetails: Free/Busy access with time, subject, and location FreeBusyAccessScope - This is typically blank. The FreeBusyAccessScope parameter specifies a security distribution group in the internal organization that contains users that can have their Free/Busy information accessed by an external organization. Enabled - This must be True. Both Autodiscover and EWS virtual directories must be enabled for WSSecurity authentication and/or OAuth. For example, if using OAuth, you should have OAuth listed as Authentication Methods, whereas if using only DAuth (Exchange 2010 for example), you would see only WSSecurity. Example of virtual directories authentication methods for an Exchange 2013 Hybrid Organization: Get-AutodiscoverVirtualDirectory -Server SERVER01 | fl Name,AdminDisplayVersion,*authentication* Name : Autodiscover (Default Web Site) AdminDisplayVersion : Version 15.0 (Build 1044.25) InternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated, WSSecurity, OAuth} ExternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated, WSSecurity, OAuth} Get-WebServicesVirtualDirectory -Server server01| fl Name,AdminDisplayVersion,*Authentication* Name : EWS (Default Web Site) AdminDisplayVersion : Version 15.0 (Build 1044.25) InternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity, OAuth} ExternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity, OAuth} As explained earlier, there are some situations where Free/Busy from on-premises to cloud is going via Availability Address Space. This would be the expected configuration for Availability Address Space in Exchange on-premises (Exchange Management Shell): Get-AvailabilityAddressSpace contoso.mail.onmicrosoft.com | fl ForestName, UserName, UseServiceAccount, AccessMethod, ProxyUrl, Name ForestName : contoso.mail.onmicrosoft.com UserName : UseServiceAccount : True AccessMethod : InternalProxy ProxyUrl : https://server01.contoso.com/EWS/Exchange.asmx Name : contoso.mail.onmicrosoft.com ForestName - The should be the tenant.mail.onmicrosoft.com domain name. This should also match the domain name of RemoteRoutingAddress of remote mailboxes. Example: contoso.mail.onmicrosoft.com UserName - This should be blank. UserServiceAccount - This must be True. AccessMethod - This should be InternalProxy. ProxyUrl - This should be the Exchange 2013/2016 Exchange Web Services Virtual Directory URL. The address could be the internal FQDN or load balancing EWS URL. Example: https://server01.contoso.com/EWS/Exchange.asmx We recommend you check the following requirements for inbound/outbound connectivity to and from Exchange servers in a Hybrid Deployment: Understanding Federated Delegation Office 365 URLs and IP address ranges If you have read this far – thank you! This concludes the Part 1 of this blog series. Onto troubleshooting next! Huge thanks to all that contributed to this blog post: Ray Fong, Nino Bilic, Tim Heeney, Greg Taylor and Brian Day. Mirela BuruianaFailure extending the schema in the Active Directory?
It is possible while installing the Active Directory Connector (ADC), or while running an Exchange setup Forestprep action, to be greeted with this incredibly generic error message: Extending the schema in the Active Directory failed. Please consult the error log LDIF.ERR in your TEMP directory. When presented with this error message, you might dutifully go off to find yourr LDIF.ERR log, as directed, and find nothing of the sort present in your temp directory. So now what? Well, one possible explanation is that you’ve got spaces in your temp dir path. I know, I know – what up? Here’s the scoop: Exchange setup and ADC setup both need to import a bunch of schema into Active Directory on initial installation. To do this, they call on a little Windows utility called ldifde.exe. Ldifde.exe takes as one of its parameters a directory path for the location of its log file. Our setups pass as this parameter the path to the current user’s TEMP directory. The code calls a Win32 API (GetTempDir()), which returns the value of the TMP/TEMP variables, and appends a trailing backslash to the string (presumably so a filename can be just tacked on the end). If our code gets back the temp dir string and finds a space anywhere in it, we’ll enclose the whole string in quotation marks (perfectly reasonable string handling practice). However, when we pass a quoted string to ldifde.exe, it barfs. In fact, it barfs so early that it isn’t even able to write an LDIF.ERR log to the TEMP dir. Why does ldifde.exe panic when it gets our string? That’s where the bug is — I’m guessing ldifde.exe reads the trailing backslash and quotation mark at the end of the string (\”)as an escape sequence, and so doesn’t think the string is actually formatted correctly. The owners of ldifde are currently investigating this. How is it that you’ve got a space in your temp dir path? Well, by default the temp dir path is something like D:\Documents and Settings\Alex\Local Settings\Temp. “But wait,” you say, “there are spaces all over the place in that path.” Right you are. However, by happy accident your temp dir variable actually gets constructed by default using 8.3 short file names. So the path becomes D:\DOCUME~1\Alex\LOCALS~1\Temp …no spaces there, which is why most users don’t experience this problem. A user would have to manually change their temp dir path to get spaces in it… or would they? Another (this time not-so-happy) accident: you can disable 8.3 file name creation. There is, in fact, a registry key that will allow you to do this (intuitively named NtfsDisable8dot3NameCreation, found in HKLM\System\CurrentControlSet\Control\FileSystem). Set this value to 1, and 8.3 filename creation will be disabled. If a user logs on to a box where this setting is in effect, the userprofile for that user will include TMP/TEMP variables that look like this: D:\DOCUME~1\Alex\Local Settings\Temp. Uh-oh… Forestprep and ADC install fail. This registry setting is actually specified in many of the security templates found up on TechNet. The workaround is to remove the spaces in your temp dir path variables (SET TEMP=D:\SOMETHING\WITHOUT\SPACES; SET TMP=D:\SOMETHING\WITHOUT\SPACES), then run Forestprep or ADC install, then set your variables back the way you had them. We'll have a KB article about this created in the next few weeks. Alex MacLeodPublic Folder Replication Troubleshooting – Part 1: Troubleshooting the Replication of New Changes
Note: this blog post is the first in a Public Folder Troubleshooting series. Please also make sure to check out part 2, part 3 and part 4 of the series. Introduction The purpose of this blog post series is to serve as a guide to troubleshooting public folder replication problems. It will not tell you exactly how to fix every possible replication problem. However, it will show you how to isolate every possible replication problem so that you focus your troubleshooting on the point of failure. Put another way, this post is intended to take you from a problem description like “The content on my old server isn’t replicating to my new server” to a much narrower problem description like “My old server isn’t responding to the status requests from my new server, therefore the new server doesn’t know it’s missing data and isn’t trying to backfill. This means the problem is actually with the old server.” This post will also describe how to identify a few of the most common replication problems. Before I get into the details of troubleshooting, I want to give an overview of my general approach to these issues. The best troubleshooting tool for public folder replication is the application log. In order to isolate a replication problem, you must be able to follow the replication events in the log to see exactly where the process is breaking down. Typically, you should turn up Diagnostics Logging on Replication Incoming and Replication Outgoing to maximum as a starting point for troubleshooting. Each time a store sends or receives a replication message, it will log an event to that effect (assuming logging is turned up). The various kinds of replication messages can be differentiated by the 'Type' shown in the description of the event. I prefer to focus on the Type rather than the event ID for several reasons: - Event IDs change between versions of Exchange. For instance, in Exchange 5.5 an outbound backfill request was a 3014. In Exchange 2000 and 2003 it's a 3016. - Incoming and outgoing event IDs are different for each type. An outgoing hierarchy message is a 3018, while an incoming hierarchy message is 3028. - Status requests and status messages use the same event IDs, even though they are two different types of messages. Thus, you can't distinguish between them from event ID alone. Focusing on the Type is a little easier. You can easily correlate these with Event IDs by examining your application log. There are only 7 types, and you can see if the message is outgoing or incoming by looking at the Category of the event. If you focus on types instead of event IDs, all you need to remember is: Hierarchy - 0x2 Content - 0x4 Backfill Request - 0x8 Backfill Response - 0x80000002 (for hierarchy) or 0x80000004 (for content) Status - 0x10 Status Request - 0x20 Also note that Replication Errors logging is rarely helpful. Even when replication is working normally, most servers will generate lots of replication error events, such as event ID 3093 indicating there was an error reading a property. In most cases the property has no effect on replication and the error can be ignored. I recommend leaving Replication Errors logging at None unless you're looking for something specific, such as the problem described in this previous blog post. Troubleshooting the Replication of New Changes To troubleshoot public folder replication, you must first be familiar with the normal message flow that is expected when replication is working. Based on that knowledge, you can identify the point of failure when a problem arises. Before I discuss how to troubleshoot the replication of new changes, let's describe what the expected behavior is. Hierarchy Replication The replication of new hierarchy changes takes place whenever a folder is created or deleted, or a change is made to the properties of a public folder, such as the replica list, client permissions, description, administrative note, or storage limits. Note that this does not include the email addresses on a mail-enabled folder. The email addresses are stored on the directory object in the Active Directory, so changing them does not result in hierarchy replication. Only changing the properties stored in the public store itself will cause hierarchy replication Every 15 minutes (by default), the store broadcasts any changes that were made to the folder properties in one or more hierarchy replication messages. With logging turned up to Maximum on Replication Outgoing, you'll see an event like this on the server where the hierarchy was modified: Event Type: Information Event Source: MSExchangeIS Public Store Event Category: Replication Outgoing Messages Event ID: 3018 Description: An outgoing replication message was issued. Type: 0x2 Message ID: <91ACCD0758385549A56A10971798985572D5@bilongexch1.bilong.test> Database "First Storage Group\Public Folder Store (BILONGEXCH1)" CN min: 1-72CF, CN max: 1-72D3 RFIs: 1 1) FID: 1-6994, PFID: 1-1, Offset: 28 IPM_SUBTREE\NewFolder Notice the "Type: 0x2" at the beginning of the description, identifying this as a hierarchy replication message. A hierarchy replication message is sent from the originating server directly to all other public stores. There's no concept of topology for the replication of new changes. All changes to the hierarchy are sent directly from the server on which the changes were made to all other servers that have a public store associated with the same hierarchy. Every other server should log an incoming event showing a type 0x2 when they successfully process the incoming replication message. Content Replication Content replication takes place whenever a message is created or deleted, or the properties of a message are changed. The times at which the store broadcasts content changes for a folder can be modified by changing the replication schedule on that folder, but by default the changes will be broadcast every 15 minutes just like the hierarchy. A content replication message is identified by the type 0x4 in the event description. Once again, there is no concept of topology for the broadcast of new changes. When the content of a folder is modified on any given replica, that replica will send a 0x4 message directly to all other replicas of the folder. And again, every receiving server should log an incoming 0x4 event when they successfully process the incoming message. Troubleshooting Steps Those are the two most basic scenarios for replication. When new hierarchy changes or new content changes are not making it from one server to another, troubleshooting is very straightforward. 1. Did the server generate an outbound replication message? So you made a change to a folder or you added content to a folder on particular server, and that content didn't make it to some other server. The first question to answer is did the target server broadcast the changes? When troubleshooting, it's important to keep track of what server you're working against when you make changes. There are several ways to accomplish this. In ESM, you can right-click on the Public Folders node and choose "Connect To" to point to a particular store. For the most part ESM will make any changes on the specified store, but be aware of one exception - Client Permissions. Prior to Exchange 2003 Sp2, when you changed Client Permissions through ESM, ESM would attempt to make the change against a store that houses a replica of the folder, even though this isn't necessary since the permissions are stored as a property of the folder in the hierarchy. With the 2003 Sp2 version of ESM, this was changed so that it now does make the change on the hierarchy you’ve pointed it to. When you're testing hierarchy replication by making changes through ESM, you should avoid using the permissions for testing since it may be hard to predict which store the change will be made against, unless you’re running ESM from 2003 Sp2. If you're using Outlook and you want to verify which replica of a folder you are hitting, you can use MFCMAPI or a similar tool to view the PR_REPLICA_SERVER property of the folder. This will show you the name of the server that Outlook is using to access the content of that folder. If the replication schedule is Always for the folder in question (which is always true for the hierarchy), and you don't see an outbound 0x2 or 0x4 within 15 minutes, then you know something is wrong on the originating server. If the server is not generating any outbound hierarchy or content broadcasts, the replication agent may have failed to start. One of the common scenarios is described in KB272999. The important thing to note here is the 3079 event: Event ID: 3079 Source: MSExchangeIS Public Type: Error Category: Replication Errors Description: Unexpected replication thread error 0x3f0. EcGetReplMsg EcReplStartup FReplAgent This will be logged even with no additional logging turned up when you mount the public store. If the 3079 includes the "EcReplStartup" function, this means the replication agent failed to start and no new changes will be broadcast until the problem is corrected and the store is mounted again. Hierarchy replication is also vulnerable to certain permissions problems when Exchange 5.5 public stores are present in the organization. When an Exchange 2000 or 2003 server sends a hierarchy replication message to an Exchange 5.5 server, it must produce a ptagACLData property (the 5.5-style permissions based on legacyExchangeDN) from the ptagNTSD property (the 2000-style permissions based on SID). This means each SID must be converted to a legacyExchangeDN. This SID to legacyExchangeDN conversion can fail for several reasons. For instance, if a SID resolves to more than one user, an event like this may be generated: Event ID: 9528 Category: General Source: MSExchangeIS Type: Error Description: The SID S-1-5-21-408248388-469072634-37170099-1391 was found on 2 users in the DS, so the store cannot map this SID to a unique user. The users involved are: /DC=com/DC=company/CN=Users/CN=User1 /DC=com/DC=company/CN=Users/CN=User2 Since the SID can't be converted to a legacyExchangeDN, the store will fail to generate an outbound hierarchy broadcast message. 2. Was the message addressed to the server that didn't receive the change? If the originating server generated the outbound message, the next step is to be sure it was addressed to the server that didn't receive the data. The easiest way to verify this is to track the message. In message tracking, you can just track the message ID that was reported in the outbound replication event. In the Message History window, the To: line will be visible. If the store that didn't receive the changes is not listed here, then again the focus should be on the originating server. Does it see that server in the organization? Does that server have email addresses on its public store object? Does the originating server see that store in the replica list on the folder in question? 3. Did the message make it to the destination server? Once you've verified that the message was addressed to the destination server, the next question to answer is - did the message make it that far? You can determine this through message tracking. If message tracking says the message was delivered to the store, but you don't see an incoming replication event acknowledging the message, see the "Common Problems" section in the last post of this series. In next blog post: Troubleshooting the Replication of Existing Data. - Bill Long
