Why can't I modify the objects on my ADC Tools created Recipient CA?
If you create a Recipient Connection Agreement (RCA) with the ADC tools wizard and then go in to view the settings of that RCA, you may notice that the “Select the objects that you want to replicate” options on the From Windows tab are grayed out and you cannot make changes… Example: The reason for this is because of the custom filter created for the RCA by the ADC tools. If you look at the values for the attribute msExchServer1SearchFilter on the agreement itself, you will see a filter specified that is similar to: (&(|(objectclass=user)(objectclass=contact)(objectclass=group))(|(legacyExchangeDN=/o=ORG1/ou=SITE1/cn=*)(legacyExchangeDN=ADCDisabledMail*)(isDeleted=TRUE))); This is a custom search filter that (in this case) searches for objects in Active Directory that are either of class user, contact, or group AND that also belong to the Administrative Group called SITE1 (or that have been deleted and need that deletion to replicate across to the 5.5 Directory) Since this is a “custom” filter, the objects can’t be edited via the normal GUI interface and that is why the check boxes are not modifiable. Once again you can see from the filter that the connection agreement is in fact already set to replicate users, contacts or groups despite what the GUI seems to indicate at first glance. If you have multiple Administrative groups that have users in the same Active Directory location you would have multiple auto-created RCAs as these will ONLY replicate objects that match the Administrative Group listed in the filter. For example, a second Administrative Group (called AG2) with objects in the same Active Directory location would have a filter like the following: (&(|(objectclass=user)(objectclass=contact)(objectclass=group))(|(legacyExchangeDN=/o=ORG1/ou=AG2/cn=*)(legacyExchangeDN=ADCDisabledMail*)(isDeleted=TRUE))); Ultimately, if you want or need more flexibility with connection agreements (such as limiting or modifying the objects controlled by the RCA) you should create the connection agreements manually using the ADC Services Snap-in instead of having the ADC tools create them automatically. - Kyle LewallenMail Flow Troubleshooter in Exchange Troubleshooting Assistant (ExTRA) - A closer look
Have you tried the "Exchange Mail Flow Troubleshooter" in Exchange Troubleshooting Assistant v1.0 (ExTRA)? This is a new troubleshooter that you can use to troubleshoot common mail flow problems you see on Exchange servers. It is designed to identify the root cause of detected symptoms so that you can take a right course of corrective actions quickly. For example, the mail flow troubleshooter can identify common root causes such as: Messages backing up in remote delivery queues due to bad DNS configuration or unintentional third party software settings Messages backing up in the Messages awaiting directory lookup queue due to heavy distribution group expansions or permissions inheritance blocks Messages can not be received from the Internet due to metabase corruption For this blog post, I would like to introduce some of the features of this troubleshooter for Exchange 2000 Server and Exchange Server 2003. When you select "Mail Flow Troubleshooter" at the Task Selection screen, you will be presented with the selections of symptoms you would like to troubleshoot. 1. Users are receiving unexpected non-delivery reports when sending messages When you or your users receive a non-delivery report, this is the one you would like to select. The tool will further prompt you what DSN code the non-delivery report contains and provide you with guidance on what the DSN code generally means and what actions are suggested. For example, for the DSN code 5.1.6, you will see the following message in the output report: "The user directory attributes, such as homeMDB or msExchHomeServerName, may be missing or corrupted. Troubleshooting: Verify the integrity of the user directory attributes, and then run the Recipient Update Service again to make sure that the attributes that are required for transport are valid." For some DSN codes, the tool checks whether the records in DNS are consistent. In this v1.0, not many troubleshooting functions for non-delivery reports have been automated but we expect that more will be in future releases. 2. Expected messages from senders are delayed or are not received by some recipients You would like to select this when you or your users see symptoms like the following: - Your organization is not receiving any messages from the Internet - Some users can receive mails from the Internet but some users can not The possible root causes of this kind of issues (or mail flow issues in general) can vary from a transient network condition to a suboptimal SMTP configuration. Thus the mail flow troubleshooter executes a wide variety of troubleshooting steps. Area Troubleshooting Steps Connectivity - 'Ping' the designated gateway/bridgehead server to check general network health - Test connectivity over port 25 and other designated SMTP ports to the designated gateway/bridgehead server - Check SMTP service / SMTP virtual server status - Check filtering configuration (Sender ID, IMF, Recipient, Connection, Sender) Mail Acceptance - Send a test mail from the designated gateway/bridgehead to the designated address - Check for known SMTP proxies that may be blocking SMTP conversations Message Tracking - Scan message tracking logs beginning from the sending server to the destination server to see how far the test message has traveled (start the queue troubleshooter if a backup is detected) Domain Routing - Verify that local domains are correctly registered in the metabase (e.g. an error will be logged if a local domain is registered as an external domain in the metabase) Here are a few examples of the result output for this symptom. 3. Messages destined to recipients are delayed or are not received by some recipients This is the opposite direction of #2 in terms of the mail flow. You may want to select this when: - No messages are going out to the Internet - You can not send messages to a specific domain - You can not send messages to a specific external address Here are the major steps the mail flow troubleshooter executes to troubleshoot this symptom. Area Troubleshooting Steps Message Tracking - Locate the most recent message submitted by the specified sender to the specified recipient - Scan message tracking logs beginning from the sending server to see how far the message traveled (start queue troubleshooting if a backup detected) Exchange Gateway - Check SMTP service / SMTP virtual server status - Check SMTP connector configuration Domain Routing - Check address space (remote domain) settings in SMTP connector and metabase 4. Messages are backing up in one or more queues on a server We call this the "Queue" troubleshooter. It can be launched directly from the Task selection screen as well as it may be triggered by selecting #2 and #3 above. In many cases when we troubleshoot issues like "mails are not coming from the Internet" or "mails are not going out to the Internet," we find out that messages are actually backing up in a queue on an internal Exchange server. In these cases, the mail flow troubleshooter triggers the "Queue" troubleshooter automatically. For this first release, we mainly focused on developing the troubleshooting steps for the following SMTP queues. - Remote delivery - Messages awaiting directory lookup - Local delivery That said, we have general troubleshooting steps for other queues such as "Messages pending submission" and "Messages waiting to be routed" and expect adding specific troubleshooting steps for those queues in future releases. Area Troubleshooting Steps Queue Status - Detect any retry/frozen queues and also queues with a large number of messages Remote Delivery - Test whether DNS servers can be accessed from the "problem" server - Check whether DNS returns valid records for the remote hosts - Test connectivity to the remote hosts (ping, port 25) - Check remote SMTP virtual server configuration/status - Throw various SMTP commands (e.g. BDAT) - Check metabase for event sink registrations - SMTP Proxy existence - Detect antivirus software blocking SMTP ports - Link state check Messages awaiting directory lookup - Categorizer performance - DL expansion checks - Check for common destinations in messages - Check file version for known bugs - Check journaling configurations - Check whether permissions inheritances are correct - Investigate disk performance - DC/GC availability Local Delivery - Check for any dismounted databases - Check for missing SMTP system mailboxes - Event 326/327 occurrence - Common issues (PF hierarchy, Missing working directory, etc.) Here are some examples of the result output for this symptom. Thanks for your interest in the new tool. Please feel free to post comments to this blog post. Also visit microsoft.public.exchange.tools newsgroup for more discussions. - Haruya ShidaIntroducing the Microsoft Office 365 Hybrid Configuration Wizard
For up-to-date information on Hybrid Configuration Wizard steps, please see Microsoft Docs. The Exchange hybrid team has been working hard over the past year getting the 3 rd version of the Hybrid Configuration Wizard (HCW) ready. This new version is called the Microsoft Office 365 Hybrid Configuration Wizard. This article tells you what’s new and shows you how to run the wizard. We also explain the various issues that have been addressed with the new Wizard, and touch on some of the telemetry we pull with every run of the wizard. We think this new wizard has enough of the old to reduce the learning curve while adding plenty of enhancements to make your hybrid deployment as friction free as possible. Microsoft Office 365 Hybrid Configuration Wizard Stand-Alone Application This version of the HCW is a standalone application that is downloaded from the service. This is an important change because one of the bigger limitations of the previous versions of the HCW was that it was included with the on-premises product. This led to the following issues: Up-To-Date hybrid experience: When you ran the HCW you got the experience consistent with your on-premises version of Exchange Server. This meant that if you are running Exchange 2013 CU7 you got the CU7 experience. If you ran Exchange 2013 CU9 you got the CU9 experience in HCW. Each customer would have a different HCW experience. Solution: The new HCW will download the latest version every time it is run, therefore providing the latest and improved experience. As soon as we make changes to, or fix any issues in the HCW, customers will see the benefits immediately. HCW not tied to Cumulative Updates: Since the previous versions of the HCW were part of the on-premises product they were updated per the regular Exchange Serviceability model. This means that the hybrid team had to wait for a new Cumulative Update (Rollup for Exchange 2010) every three months to deliver any enhancements or changes. For a component like hybrid that is a problem, we have to be agile enough to handle changes not just to on-premises, but also in the service. Solution: Again, every time you attempt to run the HCW we will ensure you have the latest version. This version will of course go through its rounds of validation, but it is in no way tied to the releases of a CU. No more waiting months for fixes! Piloting Changes: As we move forward with this new HCW we will be making some aggressive changes. In the months ahead we want to add more capabilities to HCW. One of the most important changes in HCW will be the ability to roll out feature changes slowly and in a controlled manner. Solution: We have built in the capability to allow customers who are on “first Release” and any other customers we specify (for example TAP customers) to see the latest version of the HCW. Often the latest release and the production release will be the same version, but we do have the ability to pilot versions of the HCW as needed. Improvements to error handling The HCW has a lot of dependencies and relies on various prerequisites for a successful completion. For example, you have to add an external TXT record for the HCW to create the Federation Trust, you have to have your certificates properly installed on your Exchange servers, and you have to have Internet access from your Exchange servers to name a few. I am not trying to scare you away from hybrid, in fact the wizard does walk you through most of the prerequisites. I am instead trying to point out that there are many failure points for the HCW to contend with. Up till now the solution was to provide you an error message that included a stack trace. These error messages are extremely difficult to decipher and often the first reaction after a couple of failed Internet searches was to call into support. Figure 1 shows the old error Experience for those that may not be familiar with it. Figure 1: Old error messages Our goal is to allow you to successfully configure without an error, but we also want to make sure that we give you the information needed to get past any hurdles you may face. Figure 2 shows a sample of the new (much more informative) error experience. In the sample you can see the following major improvements to the error experience: Improved Title: We have added the ability to see what Phase and Task were being completed at the time of the failure. For instance, you can know if we failed at the prerequisite check or configuration phase. You will also immediately know if you failed to create the Organization Relationship or Outbound Connectors. Error code: We have added a new error code for all the possible error messages in the new wizard. You will now see all errors prepended with a code HCW8***. This change allows for our errors to be easily searched and it allows them to remain searchable even if we change the context of the errors. Humans can read the errors: One of the previous challenges was that we provided a stack trace as the error message instead of just a friendly actionable string. We now keep the stack trace in the logs for anyone who may want that information. New “More info” feature: We added the “More info…” option under the error message. We have recently associated a KB or TechNet article as the most likely solution to EVERY error message the HCW throws. Simply click the “More Info…” link and you will be taken to that solution Access to log Files: You can easily access the HCW log file by clicking on the link that says “Open Log File”. In addition, you will find the log file on the system were you ran the new wizard from by going to “%appdata%\Microsoft\Exchange Hybrid Configuration”. Keep in mind the old location for the logs in the Exchange install directory is not used. Coolest addition: When you run the HCW you will more than likely have the Exchange Admin Center already open, but there is a chance that if you run into an issue you will need to use either your on-premises or Exchange Online PowerShell. The new HCW error experience includes a link that will open the on-premises and/or Exchange Online PowerShell. We already have the credentials you entered into the wizard, so you can seamlessly open PowerShell by using those credentials. In addition, we open the Exchange Online PowerShell with a blue background and the Exchange on-premises PowerShell with a black background so you can easily differentiate the two. Figure 2: Awesome error experience Top issues solved by the new HCW About a year ago we came out with a toolto assist you to troubleshoot your hybrid experience. This tool collects and parses the HCW log and provided a link to an article that gave a solution to your issue. The tool has been run thousands of times and has given us great insights into what the top failure points are for the HCW. This telemetry tells us what we need to focus on and allows us to see any failure trends, but in the end we were limited to the information gathered from folks that ran the HCW troubleshooter. Because we want to be as helpful as possible, we now by default upload the HCW logs to the service when you run the new wizard. Gathering this data will allow us to serve you better by limiting the amount of time it takes for someone in support to find out more about your environment and it allows us to see any trending issues and failure points that we need to address. Even with the limited amount of logs we have collected from the troubleshooter, we have been able to identify the following issues and are addressing them in the new HCW. I think you will see why the log collection is so important to the hybrid team. Note: If you want to opt out of uploading the Hybrid logs you can do that by using the registry key below on the machine were you are running the HCW from: 1. Navigate to the following location in the registry, create the path if needed: Exchange 2016: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v16\Update-HybridConfiguration Exchange 2013: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Update-HybridConfiguration 2. Create REG_DWORD “DisableUploadLogs” with value 1 TXT proof string issues Any time you are required to add an DNS entry you are dealing with a potential for failure. HCW includes a step were you need to add a record to an external DNS to prove to the Azure Authorization Service (known as Microsoft Federation Gateway) that you own the domain. This step may seem trivial but it accounts for ~15% of our HCW failures. Usually the TXT proof string get messed up in one of two ways: Incorrect string entered: when creating the DNS record to provide domain ownership we often see that the incorrect value was provided. This is in large part due to the way the HCW copied the value. In the previous version of the HCW, when you copied the TXT string, we prepended the words “Domain Proof” so it looked similar to “Domain Proof = t4jnhkjdesy78hrn…”. Solution: While simple, moving forward we are only going to copy the part of the string that is needed from the “copy link” option in HCW, which should lead to less issues with incorrect TXT strings. Domain name lockouts: The point of providing this TXT string to the external DNS is so the service can validate that you own the domain and federation certificate. After a few failed attempts to validate a domain we lock you out from federating that domain for a few hours. The purpose of this lockout is to prevent a denial of service attack. Often this issue occurs because someone put the wrong value in DNS (see the first bullet), someone created the record and did not wait for replication of the record, or someone created the record in internal (not external) DNS. Solution: To resolve this we created a new external endpoint in the service that will perform the DNS lookup for the TXT record and only try to federate the domain if the record is correct or if that new service endpoint cannot be found. The logic for this is as follows: First we try to hit the new external service endpoint and see if the TXT record is resolvable externally and is correct in DNS. If so, we move forward with federating the domain. If the record is either wrong or not resolvable, we inform you that you need to verify the record and wait for replication. If the new external TXT validation service is not reachable, we will warn you that we could not verify the TXT record but allow you to continue anyway. Figure 3 show the new TXT experience you will be getting with the Microsoft Office 365 Hybrid Configuration Wizard. Figure 3: TXT records Missing Certificate in Wizard The HCW has a screen that asks you for the “Transport Certificate”. The HCW looks to ensure this certificate is installed on every server that you designated to be part of the Send and Receive Connector Configuration, as shown on the pages in Figure 4. Figure 4: Send and Receive Connector In order for the certificate to properly display you need to ensure that the following has been completed on all of the servers designated in the wizard pages shown in figure 4: The Certificate must be a third party trusted certificate. The proper names must be on the certificate such as mail.consoto.com or *.contoso.com. The SMTP service must be assigned to the certificate on each of the sending and receiving servers. The certificates must have a private key. These requirements are nothing new, but if you have a large environment, getting all of this correct on a large number of servers can be a tough task. If even one server was missing any of the requirements, we would fail to show you the certificate. In previous versions of the HCW you were left with a blank screen (see figure 5) which offered no direction or solution. Figure 5: Blank certificate The Microsoft Office 365 Exchange Hybrid Configuration Wizard experience will not remove the certificate requirements, but it will help you solve the issue. The HCW will now show you a list of certificates that meet the requirements, and it will show you the servers that do not have a proper certificate installed (see figure 6). This will allow you to either remove those servers from the HCW receive and send connector pages, or you can properly install the certificate on those servers. Figure 6: Better certificate error A more efficient Hybrid experience One of the things we tried to do with the HCW is ensure that we are performing the various configurations in the most efficient way possible (this is our on-going green effort). A good example of an inefficient task that the HCW previously performed was the Mailbox Replication Service (MRS) enablement process. In the HCW logs collected from the troubleshooter, we could see that this cmdlet was often taking an extremely long time to complete. What we do now, is enable the Migration endpoint on the servers in your environment so that you can start moving mailboxes when the HCW is complete without having to enable the endpoint. One of the cmdlets that we used in the previous version of HCW was get-WebServicesVirtualDirectory. In a larger often geographically dispersed environment this cmdlet could take over eight hours to run. In many cases you would end up getting the following error: ERROR: Updating hybrid configuration failed with error 'Subtask Configure execution failed: Configuring organization relationship settings. Execution of the Set-WebServicesVirtualDirectory cmdlet had thrown an exception. This may indicate invalid parameters in your Hybrid Configuration settings. Unable to access the configuration system on the remote server. Make sure that the remote server allows remote configuration Solution: We have resolved this issue in the new HCW using the -ADPropertiesOnly option with Get-WebServicesVirtualDirectory. This changes things so the HCW reads the MRS settings using a local directory call instead of waiting for a response from every server in the environment. This change along with a few others in this area, makes the process take around 15 minutes instead of 8 hours (your deployment times will vary) in these large environments. This is just one example of the type of cleanups we did in the HCW to improve the reliability and speed of the configuration tasks. Autodiscover issues in HCW The single most common failure point for the HCW is the inability to retrieve the Federation Information via the Autodiscover call initiated by the Get-FederationInformation cmdlet. The output of this cmdlet is needed in order to create the Organization Relationships so you can do things like free busy sharing. This accounts for nearly 30% of all HCW failures based on the logs collected from the troubleshooter (are you starting to see the importance of these log files?). When looking at the issue there are certain things the wizard cannot directly address. For instance, at times the issues are related to an improperly configured firewall, or someone doesn’t have a third-party certificate for IIS on the Exchange servers. However, a good portion of you have had things configured correctly and still we failed to complete the Get-FederationInformationcmdlet. One of the things this cmdlet does is use DNS settings from the server you are connected to in order to resolve the Autodiscover endpoint and retrieve the federation information. Many customers do not have a DNS record created for Autodiscover internally since there is often no need for this. The internal Outlook client will use the Service Connection Point to find the Autodiscover endpoint so there is no need for this from an outlook standpoint, however the Get-FederationInformation cmdlet does not use the Service Connection Point. Therefore, if there is no forwarding configured for this zone in DNS the Get-FederationInformationcmdlet will be unable to resolve the autodiscover endpoint and the HCW will fail. Solution: To resolve this issue, we have added a new method for checking for the federation information. We still try to use local DNS first and if it fails we then will try to hit an external service to see if we can get the federation information externally. This will ensure that if you have Autodiscover published properly externally the HCW will complete as expected. See figure 7 for details: Figure 7: Get-FedInfo OAuth Integration Another common failure point is the OAuth portion of the HCW. The HCW today shows you an option to configure OAuth if you are Exchange 2013 native, but not if you coexist with previous versions of the Exchange. OAuth is required for some featurestoday, such as cross premises discovery and automatic archive retention. Because of that, we want to ensure that OAuth is by default configured so all of the Hybrid features work when you complete the HCW. One downside to this is that the current OAuth configuration experience previously had a high rate of failure. We have gone through and fixed a good portion of the experience and we have also added logic to the new HCW so that if the OAuth portion fails we will disable the OAUTH configuration by disabling the IntraOrganizationConnector and let you know we disabled it and give you remediation steps. This will ensure that a failed OAuth configuration does not prevent other hybrid features such as cross premises Free Busy from working. Many more… The above are just a few of the issues that have been addressed with the latest version of the HCW. There are many example that we could have used such as a couple of issues we addressed with mail flow, Multi-Forest deployments, and many more. In this latest version we strived for feature parity, while improvement the failure rate, and allowing for future innovation. We think we have hit the mark. Running the HCW Now that we have covered some of the new features and benefits of running the Microsoft Office 365 Exchange Hybrid Configuration Wizard, let’s take a guided tour. We are not going to go through each option in depth as most of them have not changed from Exchange 2013. How to find the new HCW We have not moved the location of the HCW in the Exchange Admin Center, the entry point look and feel is consistent with previous version of the Exchange 2013 HCW. The only difference is that instead of calling local code when you click “configure” or “modify” in the hybrid node of EAC, we now initiate the click once application. Figure 8 shows the entry point. Figure 8: Entry Point HCW Landing Page The next screen you will see is the HCW landing page, which is a page that serves two purposes. The first and most important purpose is that we can redirect a small subset of customers (based on pre-defined criteria) to an alternate HCW experience. As discussed previously in this blog, this allows us to pilot new features without affecting the production HCW experience. The second benefit of this landing page is that it allows us to provide a proper error message if the browser version, popup blockers, etc. are not configured in a way that would support the HCW. When you are on the landing page you will select the “click here” option to download the HCW. See figure 9 for a view of the landing page. Figure 9: Landing page Welcome Screen The Welcome screen (see figure 10) will provide you with a link that will inform you about what a Hybrid configuration is along with an additional link at the bottom that explains what the HCW application is going to do. The Second link is at the bottom-left of the screen and says What does this application do?On this screen you will simply click next to continue. Figure 10: Welcome screen Server Detection Page The next screen allows you to choose which server you will use to perform your hybrid configuration. This is the machine that the HCW will remote PowerShell into in order to perform all of the hybrid configuration tasks. The selected server must be running a version of Exchange that is within two releases of our currently released Cumulative update. This means at launch the new HCW will work if you are connecting to an Exchange 2013 CU8 or newer version of Exchange. However, when Exchange 2013 CU11 releases you will see that we will no longer allow you to run the new HCW from Exchange 2013 CU8 and will require a minimum of CU9. Keep in mind that even though the HCW will allow you to proceed if you are two versions older than the current release (n-2), we actually only support going one version back for Hybrid (n-1). If for you were to select a server that is running an unsupported version, the HCW will provide you with an error stating that you are not running a supported version. In addition, the HCW will provide you with a list of servers that are running a supported version (if any exist). Figure 11: Unsupported version The HCW will try to select the best server to perform the configuration tasks from using the following logic: First we look to see if the server we are on is running the latest supported version of Exchange in the organization. Next we look to see if there is an existing Exchange server in the site running the latest supported version of Exchange. Finally, we attempt to connect to an out of site Exchange server (typically in a different geographical location) running the latest supported version. If you do not like the server selection the HCW made via the above mentioned detection logic you can manually specify the server name that you want to connect to. You can use the short name (ServerName) or the long name (ServerName.Contoso.com) in the provided box to select the appropriate server running the supported version of Exchange. The last option on this page allows you to select the tenant location. For most the tenant location is simply “Microsoft Office 365” but if your Office 365 is operated by 21 Vianet, you can also use the “21 Vianet” option. Figure 12: Server detection Credentials page The main improvement on this page is the fact that we do not force you to type in your on-premises credentials. However, if you are not signed in as the user with the Organization Management Role you can manually override this behavior and provide separate credentials. Figure 13: Credentials page Connection Status page We will then show you the connection status window, which will let you know if improper credentials were provided on the previous step. Usually this is a pretty uneventful window and you just click next. Figure 14: Connection status Mail flow options page The rest of the questions in the HCW from this page on are related to the mail flow options. The experience and windows you see from this point forward may vary depending on the options selected. For more information on the mail flow options you have please review this article. Figure 15: Mail flow options Receive and Send Connector Configuration This page of the wizard allows you to select the Exchange 2013 and/or Exchange 2016 servers that you intend on configuring for sending and receiving mail for your on-premises environment. You can have a mix of 2013 and 2016 servers selected. We do not allow you to choose Exchange 2010 servers from these menus. Figure 16: Receive Connector Figure 17: Send Connector Certificate selection page We described the enhancements to this certificate selection page previously in the blog, we covered the experience you will get if a valid certificate cannot be found on any one of the Sending and Receiving servers selected on the previous page (figure 16 and figure 17). This certificate page is what you should expect to see when the certificates are installed properly on all servers. In this case you will get a list of certificates that are meeting all of the requirements and installed on all of the selected servers. In most cases the list includes only one certificate that meets the list of requirements. Figure 18: Certificate FQDN for Mail Flow The final question in the wizard will allow the HCW to properly configure the smart host settings on the outbound connector in Exchange Online. You will usually provide the FQDN that matches your MX record in this window. Figure 19: FQDN Update page Up to this point in the HCW there has been no modification made to your on-premises or Exchange Online environment. When you select the update option on this page we will start making the modification based on the answers to the questions you provided on the previous screen. Similar to the old version of the HCW we will store those answer in the local Active Directory in a configuration object known as your desired state. We will then read from that configuration object to make the modification. Figure 20: Update Wrapping this up The Exchange hybrid configuration process is something that has evolved rapidly over the past few years. We have done a lot over that time to simplify these complex configurations. With this latest version we have continued that trend by adding flexibility for innovation, more HCW stability, better HCW performance, a cleaner configuration experience, and (if needed) a proper error experience. However, our tools and services are built for you so let us know what you think, when you try out the wizard send us feedback through the feedback widget in the HCW. Just look for the “give feedback” link on the bottom of the page in the wizard and please rate the experience. The Exchange Hybrid TeamIntroducing the Microsoft Exchange Server Profile Analyzer
Included in this month’s Exchange Tools Web release (direct link) is a new analysis tool called the Microsoft Exchange Server Profile Analyzer Tool (or EPA). We created this tool to help server administrators understand their user profile to help with the process of server sizing or capacity planning. A user profile describes how many actions an average user performs in an average day. When we refer to actions, we are talking about things that a user would do within an e-mail client application like send mail, delete mail, browse folder, etc. The EPA tool attempts to use whatever information is available in user mailboxes to generate an estimated profile, and depending on the characteristics of your users, the estimated profile may not be very accurate (for example, if messages are frequently deleted from the Sent Items folder, EPA will not generate accurate sent message statistics). For those of you who have worked with older versions of Exchange and Outlook, the EPA tool is building a profile in a similar way to the storstat.exe tool. This package includes three separate tools: EPA (available as EPAWin.exe and EPACmd.exe) is designed to generate a user profile based on the contents of user mailboxes. EPAOWA is designed to generate a user profile by analyzing OWA log files EPASummarizer is an assistant tool for EPA that can combine output files from EPA to summarize statistics across multiple data collections. This article will focus specifically on the EPA tool. We will explain how it gathers data, what data it gathers, and specific requirements for running the tool successfully. The major steps that EPA takes to collect data are: Step 1: Load topology from Active Directory. EPA reads Exchange configuration data such as organization, server, storage group, private mailbox store, and http virtual server from AD. This means that the account that runs EPA is required to have the ability to read Exchange configuration data from AD. In the GUI version of EPA (EPAWin.exe), you will see a tree view of the topology once this step is complete. Step 2: Scan HTTP virtual server and virtual directory configuration for each server. For each http virtual server found on the server, EPA reads configuration information from the IIS metabase for two attributes: running state and SSL accessibility. For each virtual server, EPA maintains a list of mailbox virtual directory access links. The format of the link looks like this: http(s)://servername(:portnumber)/virtualdirectoryname/. For a particular virtual server: If EPA detects its state as not running, EPA will ignore it. If EPA fails to read the running state, EPA assumes the virtual server is running. If EPA reads that SSL is not enabled, for each mailbox virtual directory found on the virtual server, EPA adds a corresponding “http://” link to the list If EPA reads that SSL is enabled, for each mailbox virtual directory found on the virtual server, EPA adds a corresponding “https://” link to the list If EPA fails to read the SSL attribute, EPA adds two entries of this virtual server to the list. One assumes the SSL is enabled (https://), and the other does not (http://). At the end, EPA sorts the list to put “https://” links on the top so that we will always attempt to use encrypted connections if they are available. When EPA moves on to Step 4 to collect data from each user mailbox, it will attempt to use each link in the list until it finds one that works. EPA will attempt to match the configured SMTP domain of a virtual server with each user’s SMTP domains to facilitate hosted Exchange scenarios. You may notice that this step would require the account that runs EPA to have access to IIS metabase. If you don’t have access to the IIS metabase, EPA will make some assumptions about your configuration that should allow it to run under most circumstances, however you may see some error messages reported in the log if metabase access is not available. Step 3: For each server and mailbox store we found in the topology, EPA determines whether or not they will be included in the data collection stage based on configuration input, Step 4: Data collection. Based on user’s configuration of the ServerThread and MailboxThreadPerServer attributes in the configuration file, EPA creates single/multiple working threads for data collection. If ServerThread is configured to 1 (which is the default setting), EPA will collect data sequentially on servers. Otherwise, it will create multiple threads for servers, and starts to collect data from multiple servers at the same time. Similar logic applies to MailboxThreadPerServer. The mailbox thread is the actual thread that is doing the data collection from a single mailbox (see the section below for the details of this function). After each mailbox is processed, the statistics generated from the collected data will be summarized to the parent mailbox store. The collection process for a mailbox store is done when all the mailboxes in the store are processed. The statistics for the server level are calculated when the collections for all the mailbox stores on the server are done. Similarly, the statistics at the organization level will be calculated when the collections on all the servers are finished. How EPA collects data from single mailbox EPA accesses items in user’s mailbox by using HTTP/Web Distributed Authoring and Versioning (WebDAV). This requires that the account that runs EPA has to have Full Mailbox Access. Note that EPA only supports Integrated Windows Authentication (NTLM) authentication at this time. If your server does not have Integrated Windows Authentication enabled for HTTP virtual servers, EPA will fail to collect data. EPA will also fail if HTTP virtual servers on back-end mailbox servers have OWA FBA (Forms-Based Authentication) enabled. The primary WebDAV methods that EPA uses are SEARCH, PROPFIND, BPROPFIND and X-MS-ENUMATTS. The SEARCH method is used to get the hierarchy table and content table from a folder. To get the actual properties of a folder or of a message item, EPA uses the PROPFIND and BPROPFIND methods. X-MS-ENUMATTS is used to get the attachment table from a message item. Table 1 shows the sets of properties that EPA reads for different types of store objects such as folder, message, attachment and appointment using WebDAV. Table 2 shows a list of collectors EPA calculates based on the properties listed in Table 1. Table 1: Properties EPA reads by using WebDAV Property Name Namespace Root Mailbox Folder contacts urn:schemas:httpmail: calendar urn:schemas:httpmail: deleteditems urn:schemas:httpmail: drafts urn:schemas:httpmail: inbox urn:schemas:httpmail: outbox urn:schemas:httpmail: sentitems urn:schemas:httpmail: tasks urn:schemas:httpmail: notes urn:schemas:httpmail: journal urn:schemas:httpmail: MsgFolderRootURI urn:schemas:httpmail: Folder Properties href DAV: displayname DAV: childcount DAV: objectcount DAV: haschildren DAV: hassubs DAV: creationdate DAV: foldersize http://schemas.microsoft.com/exchange/ 0x0FFF0102 (EntryID) http://schemas.microsoft.com/mapi/proptag/ 0x0e080003 (Size) http://schemas.microsoft.com/mapi/proptag/ 0x36010003 (FolderType) http://schemas.microsoft.com/mapi/proptag/ 0x663A000B (HasRules) http://schemas.microsoft.com/mapi/proptag/ Message Properties href DAV: creationdate DAV: contentclass DAV: subject urn:schemas:mailheader: to urn:schemas:mailheader: cc urn:schemas:mailheader: bcc urn:schemas:mailheader: hasattachment urn:schemas:httpmail: read urn:schemas:httpmail: htmldescription urn:schemas:httpmail outlookmessageclass http://schemas.microsoft.com/exchange/ 0x0e060040 (DeliveryTime) http://schemas.microsoft.com/mapi/proptag/ 0x0e080003 (Size) http://schemas.microsoft.com/mapi/proptag/ Attachment Properties 0x3703001f (Extension) http://schemas.microsoft.com/mapi/proptag/ 0x0e200003 (AttachmentSize) http://schemas.microsoft.com/mapi/proptag/ 0x3704001f (AttachmentName) http://schemas.microsoft.com/mapi/proptag/ Appointment Properties dtstart urn:schemas:calendar: Note: Please refer to http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wss/wss/_webdav_x-ms-enumatts.asp for a complete list of supported WebDAV properties and detailed descriptions of the data containined within the properties listed above. Table 2: Exchange Server Profile Analyzer Data Collectors Collector Name Description MailboxTotalSize The total size, in bytes, of the mailbox RulesTotalCount Total number of rules defined in the mailbox FolderTotalCount Total number of visible folders in the mailbox FolderMaxMessageCount Maximum number of messages in any one folder FolderTopLevelCount Number of visible folders in the mailbox FolderUserCreatedTopLevelCount Number of user created folders that are direct children of the root of the mailbox SearchFolderCount Number of search folders in the current mailbox FolderHierarchyHeight The height of the folder tree FolderTopLevelAverageSubfolders Average number of children each child of the folder tree root has. FolderTopLevelAverageHeight Average height of each of the root folder’s children's subtrees. FolderSize Size in bytes of all the messages in a folder (takes list of folders to measure, e.g. "Inbox, Deleted Items, Sent Items") FolderSizeAggregates Folder size statistics (across all folders in mailbox) - provides Avg, Min, Max. MessageMailboxCount Number of messages in the mailbox MessageFolderCount Number of messages in a folder (takes a list of folders to measure, e.g. ("Inbox","Deleted Items","Sent Items") MessageUnreadCount Number of messages that are unread MessageDAMCount Number of Deferred Action Messages in the mailbox MessageReplyCount Number of messages where the subject prefix is "RE:" or equivalent subject prefix for the given culture MessageForwardCount Number of messages where the subject prefix is "FW:" or equivalent subject prefix for the given culture MessageContainsAtLeastOneDLCount Number of messages containing at least 1 distribution list in the recipients table MessageContainsAtLeastOneAttachmentCount Number of messages containing at least 1 attachment MessageSizeDistribution Counts messages in a size range (takes a list of ranges, e.g. "2,10,100,1024" would provide counts of messages from 0-2,2-10,10-100,100-1024,1024-beyond) MessageSizeAggregates Message size statistics across all messages (provides Avg,Min,Max) MessageReceivedPerDayAggregates Average number of messages received per day (provides Avg,Min,Max, and can restrict to the last N days) MessageSentPerDayAggregates Average number of rows in each sub table when the sent items folder is categorized by date (provides Avg,Min,Max and can restrict to last N days) MessageRepliesSentPerDayAggregate Average number of messages prefixed with "RE:" sent per day (provides Avg,Min,Max and can restrict to last N days) MessageForwardsSentPerDayAggregates Average number of messages prefixed with "FW:" sent per day (provides Avg,Min,Max and can restrict to last N days) MessageBodyTypesCount Number of messages in each body type requested (takes list of body types as input, e.g. "RTF,HTML,Other") RecipientsPerMessageSentAggregates Average number of recipients of each message in the Sent Items folder RecipientsDLPerMessageSentAggregates Average number of distribution list recipients of each message in the Sent Items folder AttachmentSizeAggregates Attachment size statistics across all attachments (provides Avg,Min,Max) AttachmentSizeDistribution Counts attachments in specified size ranges (takes a range list of inputs, e.g. "2,10,100,1024" provides counts for ranges 0-2,2-10,10-100,100-1024,1024-up) AttachmentPerMessageAggregates Statistics on number of attachments per message (provides Avg,Min,Max) ContactCount Number of contacts in the mailbox ContactCreatedPerDay Number of contacts created per day (provides Avg,Min,Max, can be restricted to last N days) AppointmentCount Number of appointments in the calendar AppointmentCreatedPerDay Number of appointments created per day (provides Avg,Min,Max, can be restricted to last N days) MeetingRequestCount Number of meeting requests in the calendar. MeetingRequestReceivedPerDayAggregates Statistics on number of meeting requestes received per day (provides Avg,Min,Max, can be restricted to last N days) We want to hear from you! Feel free to send your feedback on this tool directly to epafb AT microsoft DOT com and we will use your input to help make future versions of this tool even better. Frequently Asked Questions Question: EPA reports CompletedWithException. Where can I find what exceptions occurred during the data collection process? Answer: EPA reports exception information exceptions in a log file. The default log file will be located at the user's application data path. For example, C:\Documents and Settings\<username>\Application Data\Microsoft\Epa\epalogYYYYMMDD_HHMMSS.log Question: Error: Unable to connect to Active Directory. Please make sure that the user account has enough permission. Error: Unable to check the state of HTTP virtual server 1 on ServerName. Error: Unknown error (0x80005000) Answer: The errors are reported because the account that runs EPA does not have permission to read IIS metabase. The error code may vary. EPA will continue to attempt to collect data from user mailboxes by making some assumptions about the configuration of your Exchange topology. See "Step 2" described above for details. Question: Error: Unable to find an available URI for user ServerName\MDBName\Mailbox x. Answer: The error is reported for two reasons normally. 1. No HTTP virtual server is running. 2. For each HTTP virtual server on the exchange server, SMTP domain of the virtual server does not match with user's SMTP domains. Question: Error: User ServerName\MDBName\Mailbox x cannot access any of the following links:http://ServerName/Exchange/. Answer: The error will come up in the following five cases. 1. The account that runs EPA does not have full mailbox access on the user's mailbox. This can be verified through Exchange System Manager. Please see the EPA documentation for details on how to configure and verify permissions. If the account does have full mailbox access rights, then try using Internet Explorer to access the user's mailbox with OWA. If you are not able to access the mailbox with OWA, EPA will not be able to access it either. 2. The authentication methods configured on the virtual directory do not include Integrated Windows Authentication which is the only method EPA supports. 3. There are restrictions set up on the Exchange Server that restrict TCP/IP traffic in some way so that HTTP or HTTPS traffic from the machine you are running EPA on is blocked. You can work around this by running EPA on a machine that has the ability to connect with the back-end Exchange Server (such as an Exchange front-end server). 4. When SSL is required on the server and the name on the SSL certificate does not match the name that EPA is using to access the server, EPA will fail due to SSL certificate validation errors. We will have a fix for this in next Web Release. 5. When OWA Forms-Based Authentication is enabled on the HTTP virtual server, EPA will fail since the DAV requests submitted by EPA will get "440 Login Timeout" error. This is similar to http://support.microsoft.com/default.aspx?scid=kb;en-us;817379 . We plan to provide better feedback for this case in the next web release version. If these troubleshooting suggestions don't solve your EPA problems, please contact epafb AT microsoft DOT com for further investigation. - Jessie ZhuIntroducing the Exchange Online Fiddler Extension
Working in support at Microsoft, you get to deal with a lot of organizations who are migrating to Exchange Online. This type of migration dramatically changes how web traffic flows in the organization as the traffic moves away from Exchange on the Local Area Network to servers hosted by Microsoft in a Microsoft Data Center, essentially a remote internet location. We spend plenty of time troubleshooting Outlook when a customer has difficulties connecting to their Office 365 mailbox. We troubleshoot other connectivity issues too of course, and we continually use tools such as The Support and Recovery Assistant (SARA), (top support tip: particularly useful to detect client based issues is the Collect Information about my Outlook configuration option in SARA) Outlook ETL files (which can only be viewed by Microsoft support), and Fiddler traces to see the conversation between the Outlook client and the service – whether that’s Exchange Online, some other Office 365 service, or an Authentication or Federation provider. Since we spent so much time working with Fiddler, we wanted to put together an extension which brings our knowledge of the issues we see on a daily basis directly into Fiddler to assist with troubleshooting Outlook and Exchange Online. Whether you are in contact with Microsoft support or otherwise, we’re putting it out there so you can use it too! What does the extension do, or add? It colourizes sessions in the session list according to any known error condition. It adds additional columns added to highlight particularly useful information: Session total Elapsed time. Type of Response server that responded to the session request. Exchange type a calculated field based on session characteristics. Authentication giving you information about the session Authentication. Host IP address of the server which responded to the session request. It adds an Exchange Online inspector tab. It adds an Office365 Auth inspector tab. It adds a menu to control extension features. Don’t want to see one or more of the columns? Simply turn them off and restart Fiddler to see the changes. We added an update checking feature. Making sure anyone using the extension gets any available extension updates as we update the session logic rule set. What kind of scenarios can the extension help me troubleshoot? Authentication See if the Outlook client is Basic or Modern Authentication capable. See if Exchange Online Basic or Modern Authentication is enabled. SAML response parser to validate UserPrincipalName, ImmutableID and signing certificate used for Authentication in an Outlook session. Autodiscover See if Exchange on your network redirects to Exchange Online. See any Autodiscover responses from a server not running IIS; Apache, for example. Network Connectivity Web proxy or other device blocking Outlook traffic from reaching the internet and Office 365 / Exchange Online. Outlook cannot connect to an Exchange Online mailbox. Outlook cannot communicate with an Exchange server - either on your network or in Exchange Online. Performance information Check the length of time sessions take to complete. Check the length of time the Exchange Online server took to service the request; 'Server Think Time'. Check the length of time it took to send the response back to Fiddler and for Fiddler to relay the response back to the client app (Outlook). Other OWA cannot connect to an Exchange Online mailbox. Outlook or OWA cannot perform a specific function with the mailbox. Exchange Free/Busy and other EWS calls. What does it look like? Here’s an example of the session coloring: Exchange Online inspector tab: Exchange Online inspector tab with a HTTP 503 Federation service unavailable scenario: Office 365 Auth tab with authentication summary: Office 365 Authentication tab with SAML response parser: Extension menu to control features: Where to get it and how to provide feedback? Want to try this out for yourself? If so, you can download the extension from https://aka.ms/EXOFiddlerExtension To install it the following pre-requisites need to be met: Fiddler must first be installed. See https://aka.ms/FiddlerDownload .NetFramework 4.6 or newer See this. Start the Fiddler application at least once before installing any extension to initialize Fiddler properly. If you have any issues, feedback or questions with the extension please raise them at https://aka.ms/EXOFiddlerExtensionIssues - we’re listening and want to help if we can. For additional information on the extension see the extension wiki at https://aka.ms/EXOFiddlerExtensionWiki. Thanks for using the extension and let us know how it works for you. We’d love to hear if it helps you solve any particularly difficult problems. All the best. Jeremy KnightMail flow insights are available in Security & Compliance center
Update 6/22/18: for more updates on the subject, please see this post. We're excited to announce that Mail flow Insights will soon be available in the Office 365 Security & Compliance Center. We'll continue to create and refine mail flow insights to help improve productivity for admins, and we'll announce them as they become available. The first wave of capabilities is being released in May 2018 and we're currently in the deployment stage; features shall start to show up for you on the week of May 14. Admins can use mail flow dashboard in the Office 365 Security & Compliance Center to discover trends, insights and take actions to fix issues related to mail flow in their Office 365 organization. You can see the details in this doc. But a quick summary is listed below. Where to find the mail flow insights Go to the Office 365 Security & Compliance Center at https://protection.office.com. Expand Mail flow and then select Dashboard. Capabilities in mail flow dashboard Here are the first wave of capabilities that are being released in May 2018. Queue alerts When messages can't be sent from your Office 365 organization to your on-premises or partner email servers using connectors, the messages are queued in Office 365. Office 365 will continue to retry to delivery for 48 hours. After 48 hours, the messages will expire and will be returned to the senders in non-delivery reports (also known as a NDRs or bounce messages). If the queued email volume exceeds the pre-defined threshold (the default value is 2000 messages), the alerts will be available in the mail flow dashboard at Recent alerts, and admins will receive an email notification (to their alternative email address). Queues Even if the queued message volume hasn't exceeded the threshold, you can still use the Queues area of the mail flow dashboard to see messages that have been queued for more than one hour. You can use the Queues area to monitor the number of queued messages (the value 0 indicates mail flow is OK) and take action before the number of queued messages becomes too large. Forwarding report The Forwarding Overview Report in the mail flow dashboard displays information on messages that are automatically forwarded from your Office 365 organization to recipients in external domains. TLS report The TLS Overview Report displays the TLS encryption that's used for the connection when messages are delivered to and from your Office 365 organization. The connections that are established with other email services are encrypted by TLS when TLS is offered by both sides. Connector report The Connector Report displays information about messages that are delivered to and from your Office 365 organization using connectors. You use connectors between your own email servers and Office 365 or your partner's servers and Office 365. The message volume for each connector and the TLS encryption for the connection is available. Mail loop insight A mail loop is bad because it wastes system resources, consumes your organization's mail volume quota, and sends confusing non-delivery reports (also known as NDRs or bounce messages) to the original senders. This insight reports when a mail loop is found in your organization, the email domains that are involved in the loop, and the number of messages from the previous day that were in the loop. Slow mail flow rules insight Inefficient mail flow rules (also known as transport rules) can lead to mail flow delays for your organization. This insight reports mail flow rules that have an impact on your organization's mail flow. The insight will help you to identify and fine-tune mail flow rules to help reduce mail flow delays. Let us know how you like it! Carolyn LiuNow Available: Office 365 Mail Flow Troubleshooter
Can't send or receive email? The Mail Flow Guided Walkthrough (GWT) can help. Problems with mail delivery can be a very frustrating for both the user and the administrator. Unfortunately, mail flow issues aren't uncommon and will likely occur at some point in time whether your mailboxes are hosted on-premises, in Office 365, or a combination of both. To help you troubleshoot mailflow issues, we've just released the Mail Flow Guided Walkthrough (GWT). You can use the walkthrough for troubleshooting issues sending and receiving mail or when mail is delayed in the following scenarios: The sender’s mailbox is hosted on Office 365 and the recipient’s mailbox is outside your organization. The sender’s mailbox is outside your organization and the recipient’s mailbox is hosted in Office 365. The sender’s and recipient’s mailboxes are both located in Office 365. The sender’s mailbox is hosted in Office 365 and the recipient’s mailbox is located on an on-premise Exchange server and both users are in the same organization. The sender’s mailbox is hosted on an on-premise Exchange server and the recipient’s mailbox is hosted in Office 365 and both users are in the same organization. The goal of this walkthrough is to assist you in resolving these issues in a timely manner by focusing on the scoping and steps used to isolate and resolve problems. I’d like to thank everyone who contributed to the development of this troubleshooter with special recognition going to the following folks: Tom Kern, Nitin Shukla, Sainath Vijayaraghavan, Jim Martin, Bruce Wilson, Stephen Gilbert, Scott Landry, Charlotte Raymundo, Mary Browning, Star Li, Chen Jiang, Victor Zhang. Joe Turick
