Exchange TLS and SSL Best Practices
Note: for more up to date guidance on TLS, please see this post. Whether you are running Exchange on-premises, in the cloud, or somewhere in between, we know that security is a top priority. Microsoft is committed to giving you the information needed to make informed decisions on how to properly secure your environment. It has been suggested by some external parties that customers need to disable TLS 1.0 support. One piece of guidance we are aware of suggests taking steps to prepare to disable TLS 1.0 in summer of 2016. Another piece of guidance suggests that TLS 1.0 should not be used with internal-only applications (we do not believe that Exchange is typically used in this manner, as it connects to the outside world via SMTP). While we believe the intentions of both proposals are good and will promote adoption of TLS 1.1 & 1.2, at this time, we do not yet recommend disabling TLS 1.0 on your Exchange Server(s). Additionally, while TLS 1.1 & 1.2 are superior to TLS 1.0, the real world risks may be somewhat overstated at this point due to mitigations that have been taken across the industry. Of course, security is rarely a binary decision: disabling TLS 1.0 doesn’t suddenly turn something insecure into something secure. That said, we will continue to work towards the goal of making TLS 1.1 & 1.2 work fully with Exchange and a broad array of clients. More importantly, many customers may not have taken initial steps towards following current best practices. We believe that the first step towards a more secure environment is to have a TLS organizational awareness. While disabling TLS 1.0 on Exchange is not advised at this time, there are definite steps which can be taken today. TLS 1.0 is not widely viewed as insecure when SSL 3.0 is disabled, machines are properly updated, and proper ciphers are used. The current recommendations, which will continue evolving, are as follows: Deploy supported operating systems, clients, browsers, and Exchange versions Test everything by disabling SSL 3.0 on Internet Explorer Disable support for SSL 3.0 on the client Disable support for SSL 3.0 on the server Prioritize TLS 1.2 ciphers, and AES/3DES above others Strongly consider disabling RC4 ciphers Do NOT use MD5/MD2 certificate hashing anywhere in the chain Use RSA-2048 when creating new certificate keys When renewing or creating new requests, request SHA 256-bit or better Know what your version of Exchange supports Use tools to test and verify Do NOT get confused by explicit TLS vs. implicit TLS (For now) Wait to disable TLS 1.0 on the Exchange server Let’s get started down the list! Deploy supported operating systems, clients, browsers, and Exchange versions Perhaps it goes without saying, but the first step to securing any environment is to make sure that all servers, devices, clients, applications, etc. are updated. Most issues that support sees after following recommendations on Exchange are easily fixed with updates already available from the vendor of the incompatible device (printers, firewalls, load balancers) or software (mailers, etc.). For Exchange, this means test & apply your Windows & Exchange updates regularly. Two reasons for this – first, an environment is only as secure as the weakest link; second, older software typically won’t let you take advantage of the latest TLS versions and ciphers. Make sure firewalls, old Linux MTAs, load balancers, and mass mailer software are all updated. Make sure the multifunction printers have the latest firmware. Test everything by disabling SSL 3.0 on Internet Explorer Disabling SSL 3.0 in the browser is a good first step, because it insures that all your users remain safe, no matter where they may browse. Additionally, it easily allows you to test to make sure that websites and applications will continue to work or not. There’s still a small bit of the Internet that is still relying on SSL 3.0, but the time is overdue for it to be retired. To test your environment with Internet Explorer, follow KB3009008. Disable support for SSL 3.0 on the client After testing, you may also consider disabling it at the SCHANNEL layer for all clients. While you are viewing these settings, make sure that your clients have TLS 1.1 & 1.2 enabled. In most cases, the most recent version supported by both the client & server will be used. This is a good way to start moving towards a more secure environment. All supported versions of Windows have TLS 1.1 & 1.2 capabilities, but the older ones may not have them enabled by default. Note that registry changes under SCHANNEL are only good for applications that use the SCHANNEL API. Some applications could utilize 3 rd party or open source security APIs (like OpenSSL) which may not look at these registry keys. Also, note that changes do not take effect until reboot. Disable support for SSL 3.0 on the server The next recommendation is to disable SSL 3.0 on all servers, Exchange included. Do this by following all recommendations in the original security bulletin. Since servers can be both clients and servers, it is recommended to follow all applicable steps. As before, while you are viewing these settings, make sure that your servers have TLS 1.1 & 1.2 enabled. Note: Any of these registry changes require a reboot to take effect! You can do this with confidence because TLS 1.0 will be the minimum which you support. Exchange and Windows have both supported TLS 1.0 for over a decade. TLS 1.0 itself is not considered vulnerable when SSL 3.0 is disabled on clients and servers. In fact, most Exchange sessions already have been using TLS 1.0 or even later, for years. You are simply disabling the ability for the session to be downgraded to SSL 3.0. Disabling SSL 3.0 is typically not too impactful except for clients and devices that are older than (roughly) 10 years old. These recommendations should have already been carried out in your organization with haste. Even so, the POODLE vulnerability itself does require someone to intercept the traffic and sit between the client and server during the initial session negotiation. While this is not super difficult to accomplish, it is also not trivial. It is a much more severe problem for users who travel and for mobile devices which use hotspots. As many customers do support remote access to email, this is something for Exchange administrators to worry about. Since some mobile device vendors have not released ways to disable SSL 3.0, you can at least keep your Exchange resources safe by disabling SSL 3.0 on the server side. In addition, enabling support for TLS v1.1 and v1.2 are highly recommended. But leaving TLS 1.0 enabled is a good thing for now. Clients and applications should always prefer the most secure option, provided that Windows, the application, and the client all support it. Note: If you terminate SSL at load balancers, you’ll want to disable SSL 3.0 there as well (and perform subsequent steps there in addition). Check with your vendor to get their guidance. Also, be sure to check all Exchange servers which may be sharing a single VIP or DNS record. Office 365 completed these changes, and you will find that SSL 3.0 is not possible for any protocol. Prioritize TLS 1.2 ciphers, and AES/3DES above others The next step we recommend is based on a step we took in Office 365 to prioritize the latest ciphers which are considered much more resilient to brute force attack. The thing with ciphers is that it isn’t just about enabling the most secure one and disabling the rest. You want to offer several choices for clients to allow maximum compatibility. You typically want to disable the ones which are the least secure, but leave others to provide choice. The negotiation of a particular cipher depends on: The client passes an ordered list of ciphers which it supports The server replies with the best cipher which it has selected (server gets final say) Changing the order on the server can minimize the use of a less secure cipher, but you may want to go further and disable it completely. Cipher changes are made through this registry key, explained here. Strongly consider disabling RC4 ciphers Of course, there is risk of some clients not continuing to work if you disable too many ciphers. That said, Microsoft has been recommending that disabling RC4-suite of ciphers is a good best practice. It is considered to be a weak cipher. Disabling RC4 should be done with some care as it can introduce incompatibilities with older servers and clients, though problems should be minimal as supported versions of Windows have supported 3DES and AES alternatives for years. The rollout of this in Office 365 is in progress and should be completed shortly. Do NOT use MD5/MD2 certificate hashing anywhere in the chain Ciphers depend on the certificate chain being used - you can introduce problems when connecting to a host which has an insecure signature algorithm used in their chain. For example, we have seen that Office 365 SMTP transport is no longer able to connect to hosts with MD5 and MD2 hashing because they do not support modern ciphers. This applies to the certificate and any certificates in the chain. We see this with SMTP because Exchange is acting as a client, and because there are many older SMTP systems and firewalls still out there. Use RSA-2048 when creating new certificate keys Some things to watch out for when you renew or reissue certificates. First is that when creating your requests, use 2048-bit RSA. Anything less is not considered secure anymore. When renewing or creating new requests, request SHA 256-bit or better Second, when you renew, you should consider moving the signature algorithm from SHA1 to SHA2 if you haven’t already done so. This isn’t considered something that you need to worry about until renewal time, unless your certificate happens to be good for another couple of years – in which case, go ahead and take care of it now. You can check your Exchange certificates with a browser (or in Certificate Manager MMC): This example certificate was generated with Exchange 2013 on Windows 2012 R2. It has an RSA 2048-bit key and has an RSA SHA256 (SHA-2) signature algorithm. Know what your version of Exchange supports Some applications sometimes need to be re-compiled and tested to take advantage of these new protocols. So, every part of Exchange and Windows-based clients need to be examined and tested thoroughly. Currently, for Exchange Server, we are aware of the following limitations: SMTP – key piece of Exchange server infrastructure – support for TLS 1.1 and 1.2 were added in Exchange Server 2013 CU8 and Exchange Server 2010 SP3 RU9. This means if you want to add support for the latest ciphers and TLS versions, you may need to apply an update. IMPORTANT: SMTP is the main protocol used when communicating outside of your organization, something which is a key purpose of email. If you disable TLS 1.0, SMTP would no longer be able to use Opportunistic TLS with any external party which doesn’t support TLS 1.1 or 1.2. Emails will then be sent/received in the clear, which is certainly significantly less secure than TLS 1.0. That said, we have enabled new logging in the Exchange SMTP protocol logs to allow you to audit the impact of future changes on SMTP. Additional Note: SMTP is notably a protocol where Exchange acts as both a client and a server. Some older server implementations have been observed to incorrectly implement version negotiation. In these cases, the remote servers terminate the connection when Exchange (acting as a client) offers a version newer than TLS 1.0. This results in a complete stoppage of email to these systems. Fortunately, these situations are becoming rare as time passes, but this is pointed out because the effects often are more impactful than a mail client which cannot connect. POP/IMAP – not used as frequently in all environments, but if you do, beware that we only currently support TLS 1.1 and 1.2 on-premises in the Exchange Server 2016 Preview. We hope to make this available in a future CU, or you can make a request for it via proper channels so we can prioritize it. Office 365 already has this support. HTTPS (OWA, Outlook, EWS, Remote PS, etc.) – The support for TLS 1.1 and 1.2 is based on the support in IIS itself. Windows 2008 R2 or later supports both TLS 1.1 and 1.2, though the specific version of Windows may have these disabled or enabled by default. There is another important caveat here: the HTTPS proxy between CAS and Mailbox requires TLS 1.0 in current versions of Exchange Server – so disabling TLS 1.0 between CAS and Mailbox causes the proxy to fail. This is also something we have addressed in the Exchange 2016 Preview. We hope to make this available in a future CU, or you can make a request for it via Support. If you have dedicated roles, you can technically disable TLS 1.0 between the client & CAS, but we still are not recommending this. Office 365 already supports TLS 1.1 & 1.2, if the client supports them. Clients – TLS 1.0 is universal, with near 100% support. Though TLS 1.1 and 1.2 are growing more common, many Exchange clients still do not work with anything but TLS 1.0. For example, at this time, we are tracking multiple issues with Outlook running on Windows 8.0 or older. We are hoping to address these issues soon, but with Windows 7 commonly running in most customer environments, this is a really good reason to not disable TLS 1.0 yet. Comprehensive testing of other clients running without TLS 1.0 has not been completed by Microsoft at this time. Note: Windows Remote Desktop may also have challenges, depending on your version of Windows. For servers which are managed remotely, be sure to test this first. Use tools to test and verify There are several tools and websites you can go to for testing your server(s) and clients. It is highly recommended to do so. Some offer a grading/scoring system. Others offer pass/fail. We’re inclined to recommend one with a scoring system, since security is about risks and tradeoffs. Don’t be surprised if one or more of these tools doesn’t fully test for POODLE and just thinks TLS 1.0 is bad. Use your newfound knowledge to read the results for what they are. We prefer tools that let you check specific things (like cipher order, or individual TLS/SSL versions) in addition to the blanket “vulnerability tests”. There is also one fantastic (non-Microsoft) website called SSLLabs which simulates multiple clients and can warn you of compatibility issues with the clients which it knows about. For example, here we see that disabling TLS 1.0 would likely cause issues with older versions of Android clients: In addition, you can see how you compare with the rest of the Internet. This is great for HTTPS. Most certificate vendors have test tools available as well, though they have differing coverage of what is tested. Other tools are available which test additional protocols. Here is a test being run against IMAP on port 993 (referred to as the “SSL binding”; see below for explanation): As you can see, even on port 993, TLS 1.0 is used with AES256. Do NOT get confused by explicit TLS vs. implicit TLS In the course of human events, shortcuts are taken. One unfortunate shortcut occurred when TLS 1.0 added optional support for a per-protocol implementation of STARTTLS, also known as “explicit TLS”. Prior to “explicit TLS”, if a server application level protocol wanted to implement SSL/TLS in addition to a non-secure option, it had to take up a separate port on the machine for each. This is “implicit TLS”. See the following chart: Protocol IANA port (Explicit TLS) Protocol IANA Port (Implicit TLS) E-SMTP 25 SMTPS 465** POP3 110 POPS 995 IMAP4 143 IMAPS 993 HTTP 80* HTTPS 443 * HTTP doesn’t implement explicit TLS, because it is stateless and the overhead would not be worth it. ** Exchange specifically does not support SMTPS (implicit TLS). The first protocol which implemented this verb was ESMTP. By doing so, SMTP could support clients & servers on the same port, and could also easily implement “opportunistic” TLS/SSL. In fact, Exchange has never supported SMTPS (465), although we do reuse that port by default in Exchange 2013 for one of the three transport roles. For POP and IMAP, Exchange supports both the explicit option and the implicit option. What can be confusing is that because STARTTLS didn’t come about until TLS 1.0 – some people started confusing explicit TLS with “TLS” and some mail applications started using the terminology interchangeably. So, disabling port 995 & 993 does not turn off SSL 3.0 (you are disabling implicit POPS & IMAPS, but not SSL) – nor is enabling port 110 & 143 (explicit TLS) required for TLS 1.x. The terminology is confusing, but the concepts are mostly unrelated. This unfortunate optimization was brought into Exchange: However, tinkering with ports and implicit/explicit should not be necessary as you are NOT disabling SSL 3.0 by doing so. Securing Exchange Server shouldn’t mean changing any of these settings – just the SCHANNEL registry settings discussed above. (For now) Wait to disable TLS 1.0 on the Exchange server In summary, as of July 2015, Exchange currently supports TLS 1.0, but can also support TLS 1.1 & 1.2 with the following minimum requirements met: Protocol TLS v1.1/1.2 Minimum Requirements SMTP Exchange 2013 CU8 or Exchange 2010 SP3 RU9 POP/IMAP Exchange 2016 Preview HTTP (server) Windows 2008 R2; MAPI clients must run Windows 8.1 or later HTTP (proxy to MBX) Exchange 2016 Preview As you can see, since Exchange Server 2016 isn’t released yet as an in-market product (it is for lab use only at this time), and since Windows 7 is still the most prevalent Windows version, it is quite impractical to fully disable TLS 1.0. Not only will POP/IMAP break (for lack of TLS 1.1 and 1.2 support), but you cannot disable TLS 1.0 on any Exchange server running the mailbox server role. Most importantly, disabling TLS 1.0 will result in compatibility issues with some common mobile devices, clients, and possibly interrupt some Internet email. Don’t panic – if you have disabled SSL 3.0 and decided on a cipher order that your organization can agree on, you are likely quite secure, and you are not vulnerable to the POODLE attack. Microsoft is committed to adding full support for TLS 1.1 and 1.2. TLS v1.3 is still in draft, but stay tuned for more on that. In the meantime, don’t panic. On a test Exchange lab with Exchange 2013 on Windows Server 2012 R2, we were able to achieve a top rating by simply disabling SSL 3.0 and removing RC4 ciphers. This is nearly as good as one can achieve at the time of this posting on released versions of Exchange without impacting common clients. Additionally, this configuration should be highly compatible with nearly all clients and devices from the past decade or more, while utilizing the latest security with clients which do support it. Of course, security requires a watchful eye as new threats and vulnerabilities are discovered from time to time. As always, stay tuned to Security Bulletins and updates. Scott Landry Senior Program Manager, Exchange SupportabilityExchange Online - Modern Authentication and Conditional Access Updates
We’re constantly improving the security of Office 365 products and services. Modern Authentication and Conditional Access are two of the best ways of ensuring that your clients can take advantage of authentication features like multi-factor authentication (MFA), third-party SAML identity providers, and are implementing automated access control decisions for accessing your cloud apps based on conditions. Firstly, here’s some news about Modern Authentication. As you might already know, all new Office 365 tenants created on or after August 1, 2017 have Modern Authentication enabled by default in Exchange Online for all clients. Today, we’re announcing that Modern Authentication will soon be enabled for the Windows Outlook client and Skype for Business client in all managed (non-federated) tenants that were created before to August 1, 2017. Those tenants already have Modern Authentication enabled for Outlook mobile, Outlook for Mac and Outlook on the Web, so there are no changes to any of those clients. What does it mean to be a ‘managed tenant’? If you use Password Hash Sync, Pass-Through Authentication, or you create, manage and authenticate your user identities directly in the cloud, your tenant is considered a ‘managed tenant’ – and this change affects you. If your still create, manage and authenticate your identities in your on-premises Active Directory, and you use ADFS or some other 3 rd party iDP to authenticate your users – your tenant will not be affected by this change. Will my user experience be different? This change affects the dialog users will see when requesting their credentials. They used to see the following prompt (the exact dialog depends upon the OS of the client, but this should be similar enough to help you identify it): Now they will see the following prompt: How does this change authentication? From the user’s perspective, it’s just a dialog change. From a security perspective, the client is now using OAuth (not Basic Auth) to authenticate. What’s better about that? Why do I care? Switching to Modern Authentication (even if it’s used just for username and password) is more secure than using Basic Auth. Modern Authentication is not subject to credential capture and re-use, credentials are not stored on the client device, it ensures users re-authenticate when something about their connection or state changes, and it makes adding MFA simple. What do I need to do as an Admin? Nothing. Nothing at all, well except perhaps one thing: help your users understand that this new dialog means their connection to Office 365 is even more secure than it was before. Feel free to take the credit for that; tell them you changed it to increase their security; we don’t mind. The next thing to do is to start thinking about enabling MFA and Conditional Access, to make those connections even more secure. Here’s a great place to start finding out more. Speaking of Conditional Access, that leads us to the next thing we wanted to announce: we’re making some changes there too, specifically related to Exchange ActiveSync (EAS). We’re making a change to ensure that EAS connections will be evaluated against previously unsupported conditions within Conditional Access (CA). As you might know, many conditions that are available in CA policies have not been supported for EAS. These include country, named locations, sign-in risk, and device platform. Currently, if you include any of these conditions in a policy that targets EAS, that condition is always enforced. For example, a policy to require a compliant device outside of the corporate network would always apply (independent of the user’s location). The below shows how the admin would enable the client app condition used to target CA policy to EAS clients. The change we have made ensures that CA policy applied to EAS correctly honors previously configured conditions. You may see some cases where EAS may begin to work where it was previously blocked. So, if you have CA policies today that block EAS traffic because a condition is not supported, we advise you inspect and remove any of the unsupported conditions from policy. For example, suppose you previously configured the following policy: “Block all EAS traffic from French Guyana”. Today all EAS traffic is blocked. If you are relying on a rule like that to block all EAS traffic, you need to re-think your strategy. With the change we are making, only the EAS traffic from French Guyana will be blocked. We’re sure that you find this behavior more logical, but we wanted to make sure you were aware of the change. So, it’s worth checking your existing CA policies to make sure you don’t have rules that might be affected by this change. Other than this, we don’t expect any other change in behavior: EAS clients should still receive quarantine email when they don’t meet the CA policy requirements; otherwise they will get email access just as they do today. We really do treat the security of our service and the protection of your data as our primary concern. Please leave any comments or feedback, and thanks for reading! The Exchange TeamTooManyBadItemsPermanentException error when migrating to Exchange Online?
Some of you may have noticed that more migrations might be failing due to encountering 'too many bad items'. Upon closer review, you may notice that the migration report contains entries referencing corrupted items and being unable to translate principals. I wanted to take a few minutes and provide more information to help understand what this means, why these are now occurring, and what can be done about them. Ready to geek out? During a mailbox migration, there are several stages we go through. We start off with copying the folder hierarchy (including any views associated with those folders), then perform an initial copy of the data (what we call the Initial Sync). Once the initial data copy process is complete, we then copy rules and security descriptors. Reviewing a move report shows entries similar to these. Stage: CreatingFolderHierarchy. Percent complete: 10 Initializing folder hierarchy from mailbox <guid>: X folders total Folder hierarchy initialized for mailbox <guid>: X folders created Stage: LoadingMessages Copying messages is complete. Copying rules and security descriptors. For our discussion today, we are interested in the stage of “Copying rules and security descriptors”. Security descriptors are Access Control Lists (ACLs), which are then comprised of Access Control Entries (ACEs, or the individual permissions entries) and stored in SDDL format. In the context of a mailbox, we include both the Mailbox security descriptor (Mailbox permissions) as well as Folder security descriptors (permissions on individual folders). When we look at the Mailbox Security descriptor, it should be noted that only Explicit mailbox permissions are copied. These would include permissions granted by using the Add-MailboxPermission cmdlet, by using the Exchange Management Console (2010) or Exchange Admin Center (2013 and 2016) to add Full Access rights. Any Inherited permissions are not evaluated during the copy process. For example, granting the Receive-As permission on a database object in Active Directory results in an Inherited Allow for Full Access for all mailboxes on that database. When mailboxes on that database are migrated to Exchange Online, those Inherited permissions will not get copied. Now that we have briefly covered security descriptors, let’s look at the issue. About midway through 2016, a change was introduced to Exchange Online whereby if a security principal could not be successfully validated/mapped to an Exchange Online object, it would be marked as a bad item. Previously, the behavior was that invalid permissions would simply be ignored, and administrators were then left to wonder why some permissions no longer worked after the migration. With this new behavior, corrupt/invalid permissions are now logged so that administrators will know that there are problems with permissions. From my perspective as a Support Engineer, this is a change for the better because as Administrators, you are now able to see when there are issues with permissions. It is possible that this behavior will continue to evolve over time, but I would advise to become familiar with this new behavior so that you understand what is happening. Now how does this affect you? Since we are now incrementing the bad item count for each corrupt/invalid permission, this means that if we encounter more corrupt/invalid permissions than your current bad item limit is set to (default is 10 for a migration batch), the migration will fail. Depending on the state of permissions, you could potentially see a LOT of bad entries being logged. If you are looking at the migration report text file (downloadable from the Exchange Online Portal), you may see entries similar to the following: 11/12/2016 8:44:43 AM [EXO MRS Server] A corrupted item was encountered: Unable to translate principals for folder "Folder Name"/"FolderNTSD": Failed to find a principal from the source forest. 5/19/2016 6:33:50 PM [EXO MRS Server] A corrupted item was encountered: Unable to translate principals to the target mailbox: Failed to find a principal in the target forest that corresponds to the following source forest principal values: Alias: <alias>; DisplayName: <Display Name>; MailboxGuid: <mailbox guid>; SID: <SID of User>; ObjectGuid: <Object GUID>; LegDN: <legacyExchangeDN>; Proxies: [X500:<legacyExchagneDN format>; SMTP:user@contoso.com;];. 5/19/2016 6:33:50 PM [EXO MRS Server] A corrupted item was encountered: Unable to translate principals to the target mailbox: Failed to find a principal in the target forest that corresponds to the following source forest principal values: SID: <SID of User>; ObjectGuid: <Object GUID>;. So, what is the logic used to validate permissions? I’m glad you asked! Here is the process spelled out. There are four basic steps to this process, broken out as follows. Exchange Online - I need to resolve this SID which is present in the security descriptor (Folder or Mailbox) Exchange Online - Make a request to the On-Premises MRS Proxy, passing the SID to resolve On-Premises MRS Proxy - Look up the SID against Active Directory and return a set of attributes (including primary SID and legacyExchangeDN) Exchange Online – Take the legExchangeDN value provided, and attempt to match it up with a user account in the cloud which has that stamped as an X500 proxy address. Normally, Directory Synchronization will take care of stamping the legacyExchangeDN from each side as an X500 proxy address, but this does mean that the On-Premises legacyExchangeDN must match a Mail-enabled recipient (i.e. Mailbox, MailUser, Mail-enabled Security Group) in the cloud by an X500 Proxy. If it does not, then resolving that permission entry will fail. I do want to differentiate between the different types of permissions errors you may see. SourcePrincipalMappingException – these mean that when MRS Proxy tried to look up the SID against On-Premises Active Directory, it couldn’t be resolved. This is a common scenario when users leave the company and their accounts are deleted. You could also encounter these issues if the SID in question is part of the SIDHistory of an On-Premises account. When MRS Proxy attempts to look up the SID, we only search by ObjectSID or msExchMasterAccountSID. MRS Proxy does not evaluate against SIDHistory, so the SID failing to be resolved would be expected behavior. SIDHistory being populated won’t be a common scenario, but it is nonetheless something to be aware of. Note: Exchange Online has a special built-in bad item limit of 1000 for these Source Principal Mapping errors, so these moves will not fail unless you encounter more than 1000 of these types of bad items. TargetPrincipalMappingException – these mean that we can’t map the permission to a user account in the Target forest (Exchange Online). A common scenario here would be if a user or group was given permissions on a mailbox, but that user or group is not in your dirsync scope. After trying to move that mailbox via MRS, that user or group is not going to be present in Exchange Online, so this error would be expected. Another scenario is if a security group (not mail-enabled!) was used to assign permissions. Non mail-enabled security groups are not synchronized to Exchange Online, so they won’t exist in the Target forest. To resolve this issue, there are really two options. Increase the bad item limit to account for permissions errors. In complex legacy environments where multiple Exchange versions have been in place, and there has been a lot of user turnover, I’ve seen where permissions errors can number into the thousands. Be prepared that you may need to increase the bad item limit to a number higher than you expect. The good news here is that with improvement to Exchange over the years, the odds of encountering actual bad messages is relatively slim, so odds are good that the vast majority of bad items are bad permissions. The second bit of good news here is that we log the type of bad item that is encountered and make this information available in the move report. I’ll show you how to dig into a move report and look at the bad items later on in this blog post. Cancel the move, fix the bad permissions from the folder or mailbox by either removing them or fixing the issue causing the user/group to not be resolved in Exchange Online, and then submit the move again. But – you may ask – what if I want to fix the permissions on the current move and then resume it? Well, I’m not going to stop you from fixing bad permissions. But I will tell you that it won’t make any difference for the current move. We only evaluate permissions once, at the end of the initial data copy. If the move fails due to bad items (permissions), even if you fix the bad permissions we won’t re-evaluate the now fixed-up permissions and allow the move to complete successfully. You either have to up the bad item limit, or remove the move and fix the permissions and submit a new move. Now, I promised earlier that I would go through how to review the permissions errors. You can do this by using PowerShell and saving the move report into a variable where it is stored in memory. I typically have the move report exported out to an XML file because I don’t have direct access to customer tenant information. If you are reviewing failed moves within your own tenant, there is no need to do that if you don’t want. I’ll provide the context to do both just in case you want to know both methods. To save the move report to a variable, you would run the following from PowerShell connected to Exchange Online. $movereport = Get-MoveRequestStatistics <move request identity> -IncludeReport To save the move report to an XML file, then import the XML file into PowerShell, you would run the following from PowerShell connected to Exchange Online. Get-MoveRequestStatistics <move request identity> -IncludeReport | Export-CliXml c:\temp\movereport.xml Once the file is saved, then you import it into PowerShell. Note that this PowerShell instance does not have to be connected to Exchange Online. It can be just a regular PowerShell instance. $movereport = Import-CliXml c:\temp\movereport.xml If you never dug into a move report, let me just say that there are all sorts of golden nuggets of information buried inside (which won’t show in the text file from the Portal, by the way!) Now that you have the move report imported as a variable, you can access all the rich information within the report. We specified our variable earlier as $movereport, so we just need to call that variable, and access the information stored inside it. $movereport.report.baditems – this gives you a list of all the bad items encountered. A cool tip is that you can use the Out-GridView PowerShell function to open another window with the list. $movereport.report.baditems | Out-GridView What is nice about the Grid View is that you can then filter the output. For example, to validate that all of your bad items are permissions errors, you can simply choose “Add criteria”, check the “Kind” box, and click “Add”. Change “Contains” to “Does not contain”, and type Security. This will quickly show you if there are any other types of bad items. Now that we have identified the behavior change, and gone over how to address it, let’s end by talking about what approach should be taken for migrating mailboxes. The recommended approach to this new change in behavior would be to continue to migrate using low bad item counts, and then manually remediate those that fail. We recommend this approach because migrations that fail would indicate either a LOT of bad source permissions (more than 1000), or it indicates there are valid, working permissions On-Premises that are failing to be correctly mapped to objects in Exchange Online. Both of these conditions should not be common, so investigation would be warranted to ensure that you are in fact dealing with bad permissions. Special thanks to Brad Hughes and the rest of the MRS team for their assistance and review of this content. Ben WinzenzNew Message Trace in Office 365 Security & Compliance Center
Overview Message Trace is a key tool for email admins to troubleshoot and track the health of their organization's mail flow. Message Trace in the Exchange Admin Center (EAC) is a useful tool for tracing messages, but it's visually cluttered and confusing, and it lacks a number of more sophisticated capabilities. To make Message Trace more effective and easier for both professional and part-time email admins, we've redesigned Message Trace and placed it in the Office 365 Security & Compliance Center. The new Message Trace sports a streamlined, modern interface, and adds a number of new features that will help you get your job done more quickly and with greater effectiveness. What's New Message Trace Start Page All your queries, extended message trace requests, and downloadable reports on a single page. Save custom queries, for fast and error-free querying in the future. Recent queries are automatically saved – pick up where you left off, no need to start from scratch again. New Message Trace Panel A more modern, streamlined and intuitive interface that's simpler to understand and use, helping you get your job done more quickly. New inline address picker to select and validate senders and recipients with fewer clicks. New time range slider to easily and visually choose common query start and end times. You can now view up to the last 10 days of data in real-time (previously limited to 7 days). Downloadable extended message trace reports are now available for any date range, not just those starting more than 7 days ago. Updated Results Panel View up to 10,000 records in the Results panel (previous maximum was 1,000). Results are displayed in your default or custom time zone, instead of UTC. Filter results by Sender, Recipient, Subject or Status. Export the results you want to a CSV file (all, filtered, loaded, or selected). Find related message records: Select an item and find all related items that share the same message ID. Color-coded delivery status makes it easier to quickly spot possible issues. View message details more easily with a single click, or from the more info (. . .) fly-out menu. Message Details Panel While the key updates in the new Message Trace are the new start page and improvements to the search and results panels, the Message Trace details panel was also updated. The details panel continues to provide the same rich, diagnostics and recovery guidance as before, but it's been slightly streamlined, now using expandable sections to tuck the message events and more information details out of the way until you need them. The new Message Trace in the Security & Compliance Center is deploying now and should be available for all Office 365 customers by the middle of May. Frequently Asked Questions Where can I find the new Message Trace? You can access the new Message Trace from the Security & Compliance Center navigation menu Mail Flow > Message Trace. Will the new Message Trace replace Message Trace in Exchange Admin Center (EAC)? No. The new Message Trace is only available in the Security & Compliance Center (SCC). The legacy Message Trace in the EAC and the new Message Trace in SCC will co-exist for the foreseeable future. Why is the new Message Trace located in the Security & Compliance Center instead of the Exchange Admin Center? A significant part of an email admin's day-to-day job is keeping their organization's email secure, reducing or eliminating spam, protecting users from phishing and malware attacks, and preventing sensitive data from leaving the organization (data loss prevention or DLP). We've placed the new Message Trace in the SCC alongside the email antispam, antimalware, and DLP tools that already exist in the SCC, to give email admins a single location that hosts most of the tools they regularly work with. What roles are required to access Message Trace in the SCC? The new Message Trace in the SCC supports the same roles as the legacy Message Trace in the EAC, including Global administrator, Exchange administrator, and others. Will this new Message Trace ship to on-premises Exchange? At this time, we have no plans to ship this on-premises. The Security & Compliance Center is a purely online portal and does not exist in our on-premises releases. If my organization is in Hybrid mode – what happens? You will still be able to use this, but only for email traffic that goes through our cloud infrastructure. You will not be able to use this to track messages that are sent purely on-premises. Kevin ShaughnessyIntroducing the Exchange Online Fiddler Extension
Working in support at Microsoft, you get to deal with a lot of organizations who are migrating to Exchange Online. This type of migration dramatically changes how web traffic flows in the organization as the traffic moves away from Exchange on the Local Area Network to servers hosted by Microsoft in a Microsoft Data Center, essentially a remote internet location. We spend plenty of time troubleshooting Outlook when a customer has difficulties connecting to their Office 365 mailbox. We troubleshoot other connectivity issues too of course, and we continually use tools such as The Support and Recovery Assistant (SARA), (top support tip: particularly useful to detect client based issues is the Collect Information about my Outlook configuration option in SARA) Outlook ETL files (which can only be viewed by Microsoft support), and Fiddler traces to see the conversation between the Outlook client and the service – whether that’s Exchange Online, some other Office 365 service, or an Authentication or Federation provider. Since we spent so much time working with Fiddler, we wanted to put together an extension which brings our knowledge of the issues we see on a daily basis directly into Fiddler to assist with troubleshooting Outlook and Exchange Online. Whether you are in contact with Microsoft support or otherwise, we’re putting it out there so you can use it too! What does the extension do, or add? It colourizes sessions in the session list according to any known error condition. It adds additional columns added to highlight particularly useful information: Session total Elapsed time. Type of Response server that responded to the session request. Exchange type a calculated field based on session characteristics. Authentication giving you information about the session Authentication. Host IP address of the server which responded to the session request. It adds an Exchange Online inspector tab. It adds an Office365 Auth inspector tab. It adds a menu to control extension features. Don’t want to see one or more of the columns? Simply turn them off and restart Fiddler to see the changes. We added an update checking feature. Making sure anyone using the extension gets any available extension updates as we update the session logic rule set. What kind of scenarios can the extension help me troubleshoot? Authentication See if the Outlook client is Basic or Modern Authentication capable. See if Exchange Online Basic or Modern Authentication is enabled. SAML response parser to validate UserPrincipalName, ImmutableID and signing certificate used for Authentication in an Outlook session. Autodiscover See if Exchange on your network redirects to Exchange Online. See any Autodiscover responses from a server not running IIS; Apache, for example. Network Connectivity Web proxy or other device blocking Outlook traffic from reaching the internet and Office 365 / Exchange Online. Outlook cannot connect to an Exchange Online mailbox. Outlook cannot communicate with an Exchange server - either on your network or in Exchange Online. Performance information Check the length of time sessions take to complete. Check the length of time the Exchange Online server took to service the request; 'Server Think Time'. Check the length of time it took to send the response back to Fiddler and for Fiddler to relay the response back to the client app (Outlook). Other OWA cannot connect to an Exchange Online mailbox. Outlook or OWA cannot perform a specific function with the mailbox. Exchange Free/Busy and other EWS calls. What does it look like? Here’s an example of the session coloring: Exchange Online inspector tab: Exchange Online inspector tab with a HTTP 503 Federation service unavailable scenario: Office 365 Auth tab with authentication summary: Office 365 Authentication tab with SAML response parser: Extension menu to control features: Where to get it and how to provide feedback? Want to try this out for yourself? If so, you can download the extension from https://aka.ms/EXOFiddlerExtension To install it the following pre-requisites need to be met: Fiddler must first be installed. See https://aka.ms/FiddlerDownload .NetFramework 4.6 or newer See this. Start the Fiddler application at least once before installing any extension to initialize Fiddler properly. If you have any issues, feedback or questions with the extension please raise them at https://aka.ms/EXOFiddlerExtensionIssues - we’re listening and want to help if we can. For additional information on the extension see the extension wiki at https://aka.ms/EXOFiddlerExtensionWiki. Thanks for using the extension and let us know how it works for you. We’d love to hear if it helps you solve any particularly difficult problems. All the best. Jeremy KnightAutomatic Processing of Recurring Meeting Requests with Conflicting Instances
The behavior of the Resource Booking Attendant may seem confusing when it comes to processing a recurring meeting request where individual recurrences conflict with existing calendar items. Some administrators have tried working with calendar processing settings in the hopes of allowing double booking rooms with recurring meetings, as can be done with non-recurring meeting requests. As we'll see, double-booking a room with recurring meetings and the Resource Booking Attendant is impossible. If anyone's still confused hopefully this post will clear up what's going on. Recurring meeting request processing is handled according to the settings from Set-MailboxCalendarSettings in Exchange 2007 and Set-CalendarProcessing in Exchange 2010. Since each version uses the same options to deal with recurring meeting requests, let's use the Exchange 2010 SP1 documentation for Set-CalendarProcessing: Parameter Description ConflictPercentageAllowed The ConflictPercentageAllowed parameter specifies the maximum percentage of meeting conflicts for new recurring meeting requests. Valid input for this parameter is an integer from 0 through 100. If a new recurring meeting request conflicts with existing reservations for the resource more than the percentage specified by the ConflictPercentAllowed value, the recurring meeting request is automatically declined. When the value is 0, no conflicts are permitted for new recurring meeting requests. MaximumConflictInstances The MaximumConflictInstances parameter specifies the maximum number of conflicts for new recurring meeting requests when the AllowRecurringMeetings parameter is set to $true. Valid input for this parameter is an integer from 0 through 2147483647. If a new recurring meeting request conflicts with existing reservations for the resource more than the number of times specified by the MaximumConflictInstances value, the recurring meeting request is automatically declined. When the value is 0, no conflicts are permitted for new recurring meeting requests. When active, these two settings apply to whether Exchange accepts a recurring meeting request in its entirety, the whole series, but not whether individual instances are accepted. Individual recurring meeting instances are always declined. Examples So, for example, say a room mailbox is configured with the following settings: AutomateProcessing : AutoAccept AllowConflicts : False AllowRecurringMeetings : True ConflictPercentageAllowed : 40 MaximumConflictInstances : 6 A recurring meeting request which repeats 10 times (10 instances) is sent to the mailbox. Now let's suppose 5 out of 10 (or half) of the instances for that recurring meeting occur at the same time other meetings are scheduled in the calendar. In this case, the 50% conflict ratio exceeds the ConflictPercentageAllowed setting of 40. The result: The entire meeting request is rejected and the room sends a message back to the meeting organizer with the subject Declined All. The five conflicting instances are declined, as are the other five instances that didn't conflict with another meeting. Another meeting request is sent where there are 100 instances of a recurring meeting and 10 of them conflict with existing bookings. In this case the ConflictPercentageAllowed is not exceeded by the 10% conflict rate. However, we exceed the MaximumConflictInstances setting of 6. The result: Once again, the entire meeting request is rejected and all 100 instances are declined. For our last example, a recurring meeting request comes to this room where there are 10 instances and only 2 of them conflict with existing bookings. The result: Since we don't exceed either MaximumConflictInstances or ConflictPercentageAllowed, the room mailbox accepts the recurring meeting request series but declines to book the two instances that conflict. The meeting instances that don't conflict are accepted and booked and the room sends three emails back to the organizer, one with Accepted in the subject stating the recurring meeting request was accepted, and two with Declined in the subject for the two conflicting instances. So by these examples we see how a recurring meeting request with conflicting instances can be accepted or declined based on the calendar processing settings. However, it's important to repeat that a recurring meeting request instance that conflicts with another booking is ALWAYS declined. How all the settings work together Now I'd like to touch lightly on how all the settings that affect recurring meeting request processing work together, including the above discussed ConflictPercentageAllowed and MaximumConflictInstances, along with the AutomateProcessing, AllowConflicts, and AllowRecurringMeetings settings. AutomateProcessing: Must be set to AutoAccept to allow automatic booking of any meeting requests, recurring or not. Note, the default is AutoUpdate. AllowRecurringMeetings: Set to True by default. If it's changed to False, all recurring meeting requests are rejected. AllowConflicts: The parameter determines whether conflicting meeting requests are allowed. If set to True, sets the calendar to ignore the ConflictPercentageAllowed and MaximumConflictInstances settings. In this case, the calendar accepts all recurring meeting requests. Conflicting instances are always declined, but the number or percentage of conflicting instances don't have any impact on the entire series. If set to False (the default), we must turn to ConflictPercentageAllowed and MaximumConflictInstances to decide whether the entire recurring meeting request series is declined or not. This flowchart presents how a recurring meeting request is processed based on these settings. Figure 1: How recurring meeting requests are processed So what if an administrator wants the ability to double book a room? What alternatives are there? Use non-recurring meetings for instances that conflict. Set the mailbox to be managed by a delegate instead of the ResourceBookingAttendant alone. A delegate could then allow double-booking of conflicting recurring meeting instances. To summarize, a recurring meeting request with conflicting instances can be accepted or declined based on the calendar processing settings for the mailbox, which are configured using the Set-CalendarProcessing (Exchange 2010) or Set-MailboxCalendarSettings (Exchange 2007) cmdlets. However, individual instances of the recurring meeting request that conflict with another calendar item are ALWAYS declined. Thanks to Kristi Starr, Chris Korff, and Tom Kern for assistance with this material. Jesse Tedoff
