The New Cluster Clean Upgrade Method
Prior to releasing Exchange Server 2003 we didn’t test this scenario, and since we only support scenarios that we have tested and have confidence in this was considered unsupported. However, with so many customers interested in this method of upgrading we decided to test it and document the process. A KB article will be released describing the process, but for now here are the steps you need to know. Summary: In some cases it may be preferable to do a clean install of the operating system or Exchange Server rather than upgrading. On a cluster, your existing Exchange 2000 virtual server can be preserved and upgraded even if you need to clean install Windows 2003/Exchange 2003. Some custom configuration may be lost using this method, we recommend performing a standard upgrade if at all possible. Please read and understand all the steps and issues thoroughly before beginning this upgrade process. This kb article covers clusters that meet the following requirements: Windows 2000 SP3 Exchange 2000 SP3 Two nodes Active/Passive configuration Upgrading to: Windows 2003 Exchange 2003 Steps: For this example I’ll run the EVS on node2 while I upgrade node1, but it doesn’t matter which node is upgraded first. 1. Evict the passive node and rebuild Make sure the EVS and all cluster resources are all running on node2 (active node). Use cluster admin from node2 to stop the cluster service on node1 and then evict it from the cluster. On node1, format the disk and clean install W2k3. Rejoin node1 to the cluster (Important! This must happen before running Exchange setup) On node1, clean install E2k3. 2. Configure services On node 1, run services.msc and set Distributed Transaction Service startup type to Manual If you are using POP3 or IMAP set the start up type to manual on each service. (If you are not using these services consider deleting the resources and setting the services to disabled). If you are using full text indexes please see the note below for special instructions. On node2, take the Microsoft Exchange System Attendant resource offline (this will take all Exchange resources offline) 3. Upgrade the Exchange Virtual Server (EVS) Move the offline EVS to rebuilt node (node1) On node1, right click the System Attendant resource and select upgrade EVS The upgraded EVS cannot be failed over to run on the older version node (node2). To prevent this from happening change the System Attendant resource possible owner list to include only the upgraded node. Right click the System Attendant resource and select properties. On the “general” tab next to “possible owners” click “modify”. Under Possible owners select the name of the node that hasn’t been upgraded, and click the ? arrow button to remove it from the list. Click ok. Bring the EVS online 4. Repeat 1-2 (with node2 as the passive node) Running Exchange setup on node2 should add it back to the possible owner list for the System Attendant resource After both existing nodes are upgraded to Windows Server 2003 additional nodes can be added to the cluster. Issues: Full Text Indexes begin a Full Population after upgrading the Exchange Virtual Server After the cluster node has been upgraded to Exchange 2003, the Exchange Virtual Server has been upgraded, and the cluster resources come on-line, a full population will begin on all full-text indexes and the indexes will be disabled for searching. To avoid having the full population begin automatically, manually start and then pause a full or incremental population on all the full-text indexes on the server prior to upgrading the Exchange Virtual Server. If the indexes are already paused when you upgrade, they will remain paused and the upgrade will finish normally. After you have finished upgrading, you can manually resume building the indexes when it is convenient to do so. Once the indexes have been built, you can enable searching on the newly built indexes. One Way Upgrade Once an EVS is upgraded it cannot be run on a node with older versions of Windows and/or Exchange. Other Applications Must Be Installed Again After Upgrade Workflows, virus scanning, event sinks, and any other application installed on your existing server will need to be reinstalled and configured. Make sure all applications you are planning to bring forward are compatible with the newer versions of Windows and Exchange. Customized Registry Keys Lost Any feature that is enabled by setting a registry key (such as journaling) will not be carried forward with this upgrade. Any customization, performance tuning, manually set or altered registry key will not be persisted using this upgrade method. These will need to be manually reset. No Active/Active support Using this method of upgrading Active/Active clusters is not supported. Risk of Downtime Increased by time between upgrading first and second nodes Once the Exchange Virtual Server has been upgraded it cannot be run on the older version node. If anything happens to the upgraded node during this time, it has no passive node to fail to. To reduce the risk of downtime, upgrade the second node soon as possible after upgrading the first. The risk of downtime is increased the longer the EVS has only one node available to run on. - Carol SwalesExchange TLS and SSL Best Practices
Note: for more up to date guidance on TLS, please see this post. Whether you are running Exchange on-premises, in the cloud, or somewhere in between, we know that security is a top priority. Microsoft is committed to giving you the information needed to make informed decisions on how to properly secure your environment. It has been suggested by some external parties that customers need to disable TLS 1.0 support. One piece of guidance we are aware of suggests taking steps to prepare to disable TLS 1.0 in summer of 2016. Another piece of guidance suggests that TLS 1.0 should not be used with internal-only applications (we do not believe that Exchange is typically used in this manner, as it connects to the outside world via SMTP). While we believe the intentions of both proposals are good and will promote adoption of TLS 1.1 & 1.2, at this time, we do not yet recommend disabling TLS 1.0 on your Exchange Server(s). Additionally, while TLS 1.1 & 1.2 are superior to TLS 1.0, the real world risks may be somewhat overstated at this point due to mitigations that have been taken across the industry. Of course, security is rarely a binary decision: disabling TLS 1.0 doesn’t suddenly turn something insecure into something secure. That said, we will continue to work towards the goal of making TLS 1.1 & 1.2 work fully with Exchange and a broad array of clients. More importantly, many customers may not have taken initial steps towards following current best practices. We believe that the first step towards a more secure environment is to have a TLS organizational awareness. While disabling TLS 1.0 on Exchange is not advised at this time, there are definite steps which can be taken today. TLS 1.0 is not widely viewed as insecure when SSL 3.0 is disabled, machines are properly updated, and proper ciphers are used. The current recommendations, which will continue evolving, are as follows: Deploy supported operating systems, clients, browsers, and Exchange versions Test everything by disabling SSL 3.0 on Internet Explorer Disable support for SSL 3.0 on the client Disable support for SSL 3.0 on the server Prioritize TLS 1.2 ciphers, and AES/3DES above others Strongly consider disabling RC4 ciphers Do NOT use MD5/MD2 certificate hashing anywhere in the chain Use RSA-2048 when creating new certificate keys When renewing or creating new requests, request SHA 256-bit or better Know what your version of Exchange supports Use tools to test and verify Do NOT get confused by explicit TLS vs. implicit TLS (For now) Wait to disable TLS 1.0 on the Exchange server Let’s get started down the list! Deploy supported operating systems, clients, browsers, and Exchange versions Perhaps it goes without saying, but the first step to securing any environment is to make sure that all servers, devices, clients, applications, etc. are updated. Most issues that support sees after following recommendations on Exchange are easily fixed with updates already available from the vendor of the incompatible device (printers, firewalls, load balancers) or software (mailers, etc.). For Exchange, this means test & apply your Windows & Exchange updates regularly. Two reasons for this – first, an environment is only as secure as the weakest link; second, older software typically won’t let you take advantage of the latest TLS versions and ciphers. Make sure firewalls, old Linux MTAs, load balancers, and mass mailer software are all updated. Make sure the multifunction printers have the latest firmware. Test everything by disabling SSL 3.0 on Internet Explorer Disabling SSL 3.0 in the browser is a good first step, because it insures that all your users remain safe, no matter where they may browse. Additionally, it easily allows you to test to make sure that websites and applications will continue to work or not. There’s still a small bit of the Internet that is still relying on SSL 3.0, but the time is overdue for it to be retired. To test your environment with Internet Explorer, follow KB3009008. Disable support for SSL 3.0 on the client After testing, you may also consider disabling it at the SCHANNEL layer for all clients. While you are viewing these settings, make sure that your clients have TLS 1.1 & 1.2 enabled. In most cases, the most recent version supported by both the client & server will be used. This is a good way to start moving towards a more secure environment. All supported versions of Windows have TLS 1.1 & 1.2 capabilities, but the older ones may not have them enabled by default. Note that registry changes under SCHANNEL are only good for applications that use the SCHANNEL API. Some applications could utilize 3 rd party or open source security APIs (like OpenSSL) which may not look at these registry keys. Also, note that changes do not take effect until reboot. Disable support for SSL 3.0 on the server The next recommendation is to disable SSL 3.0 on all servers, Exchange included. Do this by following all recommendations in the original security bulletin. Since servers can be both clients and servers, it is recommended to follow all applicable steps. As before, while you are viewing these settings, make sure that your servers have TLS 1.1 & 1.2 enabled. Note: Any of these registry changes require a reboot to take effect! You can do this with confidence because TLS 1.0 will be the minimum which you support. Exchange and Windows have both supported TLS 1.0 for over a decade. TLS 1.0 itself is not considered vulnerable when SSL 3.0 is disabled on clients and servers. In fact, most Exchange sessions already have been using TLS 1.0 or even later, for years. You are simply disabling the ability for the session to be downgraded to SSL 3.0. Disabling SSL 3.0 is typically not too impactful except for clients and devices that are older than (roughly) 10 years old. These recommendations should have already been carried out in your organization with haste. Even so, the POODLE vulnerability itself does require someone to intercept the traffic and sit between the client and server during the initial session negotiation. While this is not super difficult to accomplish, it is also not trivial. It is a much more severe problem for users who travel and for mobile devices which use hotspots. As many customers do support remote access to email, this is something for Exchange administrators to worry about. Since some mobile device vendors have not released ways to disable SSL 3.0, you can at least keep your Exchange resources safe by disabling SSL 3.0 on the server side. In addition, enabling support for TLS v1.1 and v1.2 are highly recommended. But leaving TLS 1.0 enabled is a good thing for now. Clients and applications should always prefer the most secure option, provided that Windows, the application, and the client all support it. Note: If you terminate SSL at load balancers, you’ll want to disable SSL 3.0 there as well (and perform subsequent steps there in addition). Check with your vendor to get their guidance. Also, be sure to check all Exchange servers which may be sharing a single VIP or DNS record. Office 365 completed these changes, and you will find that SSL 3.0 is not possible for any protocol. Prioritize TLS 1.2 ciphers, and AES/3DES above others The next step we recommend is based on a step we took in Office 365 to prioritize the latest ciphers which are considered much more resilient to brute force attack. The thing with ciphers is that it isn’t just about enabling the most secure one and disabling the rest. You want to offer several choices for clients to allow maximum compatibility. You typically want to disable the ones which are the least secure, but leave others to provide choice. The negotiation of a particular cipher depends on: The client passes an ordered list of ciphers which it supports The server replies with the best cipher which it has selected (server gets final say) Changing the order on the server can minimize the use of a less secure cipher, but you may want to go further and disable it completely. Cipher changes are made through this registry key, explained here. Strongly consider disabling RC4 ciphers Of course, there is risk of some clients not continuing to work if you disable too many ciphers. That said, Microsoft has been recommending that disabling RC4-suite of ciphers is a good best practice. It is considered to be a weak cipher. Disabling RC4 should be done with some care as it can introduce incompatibilities with older servers and clients, though problems should be minimal as supported versions of Windows have supported 3DES and AES alternatives for years. The rollout of this in Office 365 is in progress and should be completed shortly. Do NOT use MD5/MD2 certificate hashing anywhere in the chain Ciphers depend on the certificate chain being used - you can introduce problems when connecting to a host which has an insecure signature algorithm used in their chain. For example, we have seen that Office 365 SMTP transport is no longer able to connect to hosts with MD5 and MD2 hashing because they do not support modern ciphers. This applies to the certificate and any certificates in the chain. We see this with SMTP because Exchange is acting as a client, and because there are many older SMTP systems and firewalls still out there. Use RSA-2048 when creating new certificate keys Some things to watch out for when you renew or reissue certificates. First is that when creating your requests, use 2048-bit RSA. Anything less is not considered secure anymore. When renewing or creating new requests, request SHA 256-bit or better Second, when you renew, you should consider moving the signature algorithm from SHA1 to SHA2 if you haven’t already done so. This isn’t considered something that you need to worry about until renewal time, unless your certificate happens to be good for another couple of years – in which case, go ahead and take care of it now. You can check your Exchange certificates with a browser (or in Certificate Manager MMC): This example certificate was generated with Exchange 2013 on Windows 2012 R2. It has an RSA 2048-bit key and has an RSA SHA256 (SHA-2) signature algorithm. Know what your version of Exchange supports Some applications sometimes need to be re-compiled and tested to take advantage of these new protocols. So, every part of Exchange and Windows-based clients need to be examined and tested thoroughly. Currently, for Exchange Server, we are aware of the following limitations: SMTP – key piece of Exchange server infrastructure – support for TLS 1.1 and 1.2 were added in Exchange Server 2013 CU8 and Exchange Server 2010 SP3 RU9. This means if you want to add support for the latest ciphers and TLS versions, you may need to apply an update. IMPORTANT: SMTP is the main protocol used when communicating outside of your organization, something which is a key purpose of email. If you disable TLS 1.0, SMTP would no longer be able to use Opportunistic TLS with any external party which doesn’t support TLS 1.1 or 1.2. Emails will then be sent/received in the clear, which is certainly significantly less secure than TLS 1.0. That said, we have enabled new logging in the Exchange SMTP protocol logs to allow you to audit the impact of future changes on SMTP. Additional Note: SMTP is notably a protocol where Exchange acts as both a client and a server. Some older server implementations have been observed to incorrectly implement version negotiation. In these cases, the remote servers terminate the connection when Exchange (acting as a client) offers a version newer than TLS 1.0. This results in a complete stoppage of email to these systems. Fortunately, these situations are becoming rare as time passes, but this is pointed out because the effects often are more impactful than a mail client which cannot connect. POP/IMAP – not used as frequently in all environments, but if you do, beware that we only currently support TLS 1.1 and 1.2 on-premises in the Exchange Server 2016 Preview. We hope to make this available in a future CU, or you can make a request for it via proper channels so we can prioritize it. Office 365 already has this support. HTTPS (OWA, Outlook, EWS, Remote PS, etc.) – The support for TLS 1.1 and 1.2 is based on the support in IIS itself. Windows 2008 R2 or later supports both TLS 1.1 and 1.2, though the specific version of Windows may have these disabled or enabled by default. There is another important caveat here: the HTTPS proxy between CAS and Mailbox requires TLS 1.0 in current versions of Exchange Server – so disabling TLS 1.0 between CAS and Mailbox causes the proxy to fail. This is also something we have addressed in the Exchange 2016 Preview. We hope to make this available in a future CU, or you can make a request for it via Support. If you have dedicated roles, you can technically disable TLS 1.0 between the client & CAS, but we still are not recommending this. Office 365 already supports TLS 1.1 & 1.2, if the client supports them. Clients – TLS 1.0 is universal, with near 100% support. Though TLS 1.1 and 1.2 are growing more common, many Exchange clients still do not work with anything but TLS 1.0. For example, at this time, we are tracking multiple issues with Outlook running on Windows 8.0 or older. We are hoping to address these issues soon, but with Windows 7 commonly running in most customer environments, this is a really good reason to not disable TLS 1.0 yet. Comprehensive testing of other clients running without TLS 1.0 has not been completed by Microsoft at this time. Note: Windows Remote Desktop may also have challenges, depending on your version of Windows. For servers which are managed remotely, be sure to test this first. Use tools to test and verify There are several tools and websites you can go to for testing your server(s) and clients. It is highly recommended to do so. Some offer a grading/scoring system. Others offer pass/fail. We’re inclined to recommend one with a scoring system, since security is about risks and tradeoffs. Don’t be surprised if one or more of these tools doesn’t fully test for POODLE and just thinks TLS 1.0 is bad. Use your newfound knowledge to read the results for what they are. We prefer tools that let you check specific things (like cipher order, or individual TLS/SSL versions) in addition to the blanket “vulnerability tests”. There is also one fantastic (non-Microsoft) website called SSLLabs which simulates multiple clients and can warn you of compatibility issues with the clients which it knows about. For example, here we see that disabling TLS 1.0 would likely cause issues with older versions of Android clients: In addition, you can see how you compare with the rest of the Internet. This is great for HTTPS. Most certificate vendors have test tools available as well, though they have differing coverage of what is tested. Other tools are available which test additional protocols. Here is a test being run against IMAP on port 993 (referred to as the “SSL binding”; see below for explanation): As you can see, even on port 993, TLS 1.0 is used with AES256. Do NOT get confused by explicit TLS vs. implicit TLS In the course of human events, shortcuts are taken. One unfortunate shortcut occurred when TLS 1.0 added optional support for a per-protocol implementation of STARTTLS, also known as “explicit TLS”. Prior to “explicit TLS”, if a server application level protocol wanted to implement SSL/TLS in addition to a non-secure option, it had to take up a separate port on the machine for each. This is “implicit TLS”. See the following chart: Protocol IANA port (Explicit TLS) Protocol IANA Port (Implicit TLS) E-SMTP 25 SMTPS 465** POP3 110 POPS 995 IMAP4 143 IMAPS 993 HTTP 80* HTTPS 443 * HTTP doesn’t implement explicit TLS, because it is stateless and the overhead would not be worth it. ** Exchange specifically does not support SMTPS (implicit TLS). The first protocol which implemented this verb was ESMTP. By doing so, SMTP could support clients & servers on the same port, and could also easily implement “opportunistic” TLS/SSL. In fact, Exchange has never supported SMTPS (465), although we do reuse that port by default in Exchange 2013 for one of the three transport roles. For POP and IMAP, Exchange supports both the explicit option and the implicit option. What can be confusing is that because STARTTLS didn’t come about until TLS 1.0 – some people started confusing explicit TLS with “TLS” and some mail applications started using the terminology interchangeably. So, disabling port 995 & 993 does not turn off SSL 3.0 (you are disabling implicit POPS & IMAPS, but not SSL) – nor is enabling port 110 & 143 (explicit TLS) required for TLS 1.x. The terminology is confusing, but the concepts are mostly unrelated. This unfortunate optimization was brought into Exchange: However, tinkering with ports and implicit/explicit should not be necessary as you are NOT disabling SSL 3.0 by doing so. Securing Exchange Server shouldn’t mean changing any of these settings – just the SCHANNEL registry settings discussed above. (For now) Wait to disable TLS 1.0 on the Exchange server In summary, as of July 2015, Exchange currently supports TLS 1.0, but can also support TLS 1.1 & 1.2 with the following minimum requirements met: Protocol TLS v1.1/1.2 Minimum Requirements SMTP Exchange 2013 CU8 or Exchange 2010 SP3 RU9 POP/IMAP Exchange 2016 Preview HTTP (server) Windows 2008 R2; MAPI clients must run Windows 8.1 or later HTTP (proxy to MBX) Exchange 2016 Preview As you can see, since Exchange Server 2016 isn’t released yet as an in-market product (it is for lab use only at this time), and since Windows 7 is still the most prevalent Windows version, it is quite impractical to fully disable TLS 1.0. Not only will POP/IMAP break (for lack of TLS 1.1 and 1.2 support), but you cannot disable TLS 1.0 on any Exchange server running the mailbox server role. Most importantly, disabling TLS 1.0 will result in compatibility issues with some common mobile devices, clients, and possibly interrupt some Internet email. Don’t panic – if you have disabled SSL 3.0 and decided on a cipher order that your organization can agree on, you are likely quite secure, and you are not vulnerable to the POODLE attack. Microsoft is committed to adding full support for TLS 1.1 and 1.2. TLS v1.3 is still in draft, but stay tuned for more on that. In the meantime, don’t panic. On a test Exchange lab with Exchange 2013 on Windows Server 2012 R2, we were able to achieve a top rating by simply disabling SSL 3.0 and removing RC4 ciphers. This is nearly as good as one can achieve at the time of this posting on released versions of Exchange without impacting common clients. Additionally, this configuration should be highly compatible with nearly all clients and devices from the past decade or more, while utilizing the latest security with clients which do support it. Of course, security requires a watchful eye as new threats and vulnerabilities are discovered from time to time. As always, stay tuned to Security Bulletins and updates. Scott Landry Senior Program Manager, Exchange Supportability
