Support of DANE and DNSSEC in Office 365 Exchange Online
Microsoftis committedto providing world-classemailsecurity solutionsand the support forthe latestInternetstandards in order to provide advanced email protection for our customers.Today we areannouncing that Exchange Online will be adding support for two new Internet standards specific to SMTP traffic.Introducing: Log Parser Studio
To download the Log Parser Studio, please see the attachment on this blog post. Anyone who regularly uses Log Parser 2.2 knows just how useful and powerful it can be for obtaining valuable information from IIS (Internet Information Server) and other logs. In addition, adding the power of SQL allows explicit searching of gigabytes of logs returning only the data that is needed while filtering out the noise. The only thing missing is a great graphical user interface (GUI) to function as a front-end to Log Parser and a ‘Query Library’ in order to manage all those great queries and scripts that one builds up over time. Log Parser Studio was created to fulfill this need; by allowing those who use Log Parser 2.2 (and even those who don’t due to lack of an interface) to work faster and more efficiently to get to the data they need with less “fiddling” with scripts and folders full of queries. With Log Parser Studio (LPS for short) we can house all of our queries in a central location. We can edit and create new queries in the ‘Query Editor’ and save them for later. We can search for queries using free text search as well as export and import both libraries and queries in different formats allowing for easy collaboration as well as storing multiple types of separate libraries for different protocols. Processing Logs for Exchange Protocols We all know this very well: processing logs for different Exchange protocols is a time consuming task. In the absence of special purpose tools, it becomes a tedious task for an Exchange Administrator to sift thru those logs and process them using Log Parser (or some other tool), if output format is important. You also need expertise in writing those SQL queries. You can also use special purpose scripts that one can find on the web and then analyze the output to make some sense of out of those lengthy logs. Log Parser Studio is mainly designed for quick and easy processing of different logs for Exchange protocols. Once you launch it, you’ll notice tabs for different Exchange protocols, i.e. Microsoft Exchange ActiveSync (MAS), Exchange Web Services (EWS), Outlook Web App (OWA/HTTP) and others. Under those tabs there are tens of SQL queries written for specific purposes (description and other particulars of a query are also available in the main UI), which can be run by just one click! Let’s get into the specifics of some of the cool features of Log Parser Studio … Query Library and Management Upon launching LPS, the first thing you will see is the Query Library preloaded with queries. This is where we manage all of our queries. The library is always available by clicking on the Library tab. You can load a query for review or execution using several methods. The easiest method is to simply select the query in the list and double-click it. Upon doing so the query will auto-open in its own Query tab. The Query Library is home base for queries. All queries maintained by LPS are stored in this library. There are easy controls to quickly locate desired queries & mark them as favorites for quick access later. Library Recovery The initial library that ships with LPS is embedded in the application and created upon install. If you ever delete, corrupt or lose the library you can easily reset back to the original by using the recover library feature (Options | Recover Library). When recovering the library all existing queries will be deleted. If you have custom/modified queries that you do not want to lose, you should export those first, then after recovering the default set of queries, you can merge them back into LPS. Import/Export Depending on your need, the entire library or subsets of the library can be imported and exported either as the default LPS XML format or as SQL queries. For example, if you have a folder full of Log Parser SQL queries, you can import some or all of them into LPS’s library. Usually, the only thing you will need to do after the import is make a few adjustments. All LPS needs is the base SQL query and to swap out the filename references with ‘[LOGFILEPATH]’ and/or ‘[OUTFILEPATH]’ as discussed in detail in the PDF manual included with the tool (you can access it via LPS | Help | Documentation). Queries Remember that a well-written structured query makes all the difference between a successful query that returns the concise information you need vs. a subpar query which taxes your system, returns much more information than you actually need and in some cases crashes the application. The art of creating great SQL/Log Parser queries is outside the scope of this post, however all of the queries included with LPS have been written to achieve the most concise results while returning the fewest records. Knowing what you want and how to get it with the least number of rows returned is the key! Batch Jobs and Multithreading You’ll find that LPS in combination with Log Parser 2.2 is a very powerful tool. However, if all you could do was run a single query at a time and wait for the results, you probably wouldn’t be making near as much progress as you could be. In lieu of this LPS contains both batch jobs and multithreaded queries. A batch job is simply a collection of predefined queries that can all be executed with the press of a single button. From within the Batch Manager you can remove any single or all queries as well as execute them. You can also execute them by clicking the Run Multiple Queries button or the Execute button in the Batch Manager. Upon execution, LPS will prepare and execute each query in the batch. By default LPS will send ALL queries to Log Parser 2.2 as soon as each is prepared. This is where multithreading works in our favor. For example, if we have 50 queries setup as a batch job and execute the job, we’ll have 50 threads in the background all working with Log Parser simultaneously leaving the user free to work with other queries. As each job finishes the results are passed back to the grid or the CSV output based on the query type. Even in this scenario you can continue to work with other queries, search, modify and execute. As each query completes its thread is retired and its resources freed. These threads are managed very efficiently in the background so there should be no issue running multiple queries at once. Now what if we did want the queries in the batch to run concurrently for performance or other reasons? This functionality is already built-into LPS’s options. Just make the change in LPS | Options | Preferences by checking the ‘Process Batch Queries in Sequence’ checkbox. When checked, the first query in the batch is executed and the next query will not begin until the first one is complete. This process will continue until the last query in the batch has been executed. Automation In conjunction with batch jobs, automation allows unattended scheduled automation of batch jobs. For example we can create a scheduled task that will automatically run a chosen batch job which also operates on a separate set of custom folders. This process requires two components, a folder list file (.FLD) and a batch list file (.XML). We create these ahead of time from within LPS. For more details on how to do that, please refer to the manual. Charts Many queries that return data to the Result Grid can be charted using the built-in charting feature. The basic requirements for charts are the same as Log Parser 2.2, i.e. The first column in the grid may be any data type (string, number etc.) The second column must be some type of number (Integer, Double, Decimal), Strings are not allowed Keep the above requirements in mind when creating your own queries so that you will consciously write the query to include a number for column two. To generate a chart click the chart button after a query has completed. For #2 above, even if you forgot to do so, you can drag any numbered column and drop it in the second column after the fact. This way if you have multiple numbered columns, you can simply drag the one that you’re interested in, into second column and generate different charts from the same data. Again, for more details on charting feature, please refer to the manual. Keyboard Shortcuts/Commands There are multiple keyboard shortcuts built-in to LPS. You can view the list anytime while using LPS by clicking LPS | Help | Keyboard Shortcuts. The currently included shortcuts are as follows: Shortcut What it does CTRL+N Start a new query. CTRL+S Save active query in library or query tab depending on which has focus. CTRL+Q Open library window. CTRL+B Add selected query in library to batch. ALT+B Open Batch Manager. CTRL+B Add the selected queries to batch. CTRL+D Duplicates the current active query to a new tab. CTRL+ALT+E Open the error log if one exists. CTRL+E Export current selected query results to CSV. ALT+F Add selected query in library to the favorites list. CTRL+ALT+L Open the raw Library in the first available text editor. CTRL+F5 Reload the Library from disk. F5 Execute active query. F2 Edit name/description of currently selected query in the Library. F3 Display the list of IIS fields. Supported Input and Output types Log Parser 2.2 has the ability to query multiple types of logs. Since LPS is a work in progress, only the most used types are currently available. Additional input and output types will be added when possible in upcoming versions or updates. Supported Input Types Full support for W3SVC/IIS, CSV, HTTP Error and basic support for all built-in Log Parser 2.2 input formats. In addition, some custom written LPS formats such as Microsoft Exchange specific formats that are not available with the default Log Parser 2.2 install. Supported Output Types CSV and TXT are the currently supported output file types. Log Parser Studio - Quick Start Guide Want to skip all the details & just run some queries right now? Start here … The very first thing Log Parser Studio needs to know is where the log files are, and the default location that you would like any queries that export their results as CSV files to be saved. 1. Setup your default CSV output path: a. Go to LPS | Options | Preferences | Default Output Path. b. Browse to and select the folder you would like to use for exported results. c. Click Apply. d. Any queries that export CSV files will now be saved in this folder. NOTE: If you forget to set this path before you start the CSV files will be saved in %AppData%\Microsoft\Log Parser Studio by default but it is recommended that y ou move this to another location. 2. Tell LPS where the log files are by opening the Log File Manager. If you try to run a query before completing this step LPS will prompt and ask you to set the log path. Upon clicking OK on that prompt, you are presented with the Log File Manager. Click Add Folder to add a folder or Add File to add a single or multiple files. When adding a folder you still must select at least one file so LPS will know which type of log we are working with. When doing so, LPS will automatically turn this into a wildcard (*.xxx) Indicating that all matching logs in the folder will be searched. You can easily tell which folder or files are currently being searched by examining the status bar at the bottom-right of Log Parser Studio. To see the full path, roll your mouse over the status bar. NOTE: LPS and Log Parser handle multiple types of logs and objects that can be queried. It is important to remember that the type of log you are querying must match the query you are performing. In other words, when running a query that expects IIS logs, only IIS logs should be selected in the File Manager. Failure to do this (it’s easy to forget) will result errors or unexpected behavior will be returned when running the query. 3. Choose a query from the library and run it: a. Click the Library tab if it isn’t already selected. b. Choose a query in the list and double-click it. This will open the query in its own tab. c. Click the Run Single Query button to execute the query The query execution will begin in the background. Once the query has completed there are two possible outputs targets; the result grid in the top half of the query tab or a CSV file. Some queries return to the grid while other more memory intensive queries are saved to CSV. As a general rule queries that may return very large result sets are probably best served going to a CSV file for further processing in Excel. Once you have the results there are many features for working with those results. For more details, please refer to the manual. Have fun with Log Parser Studio! & always remember – There’s a query for that! Kary Wall Escalation Engineer Microsoft Exchange SupportNamespace Planning in Exchange 2016
If you are like the vast majority of our customers, you already have some version(s) of Exchange deployed in your environment. Depending on the version, you may have different namespace requirements today. Exchange 2010 Exchange 2010 leverages the Autodiscover service for enabling client profile changes, so that namespace exists. Exchange 2010 introduced additional namespace requirements, which resulted in additional complexity around namespace planning, especially for site resilient solutions: Primary datacenter Internet protocol namespace (mail.contoso.com) Secondary datacenter Internet protocol namespace (mail2.contoso.com) Primary datacenter Outlook Web App failback namespace (mailpri.contoso.com) Secondary datacenter Outlook Web App failback namespace (mailsec.contoso.com) Transport namespace (smtp.contoso.com) Primary datacenter RPC Client Access namespace (rpc.contoso.com) Secondary datacenter RPC Client Access namespace (rpc2.contoso.com) Out of these seven namespaces, five of them were required on certificates. The RPC Client Access namespaces were not required on the certificate because they were accessed via RPC connectivity and not via an Internet-based protocol, like HTTP. Exchange 2016 One of the benefits of the Exchange 2016 architecture (first introduced in Exchange 2013) is that the namespace model can be simplified, when compared to Exchange 2010. An example of how it can be simplified can be seen when thinking about a site resilience scenario. If you have two datacenters participating in a site resilient architecture, by replacing the Exchange 2010 infrastructure with Exchange 2016, five namespaces could potentially be removed: Secondary datacenter Internet protocol namespace (mail2.contoso.com) Primary datacenter Outlook Web App failback namespace (mailpri.contoso.com) Secondary datacenter Outlook Web App failback namespace (mailsec.contoso.com) Primary datacenter RPC Client Access namespace (rpc.contoso.com) Secondary datacenter RPC Client Access namespace (rpc2.contoso.com) There’s two reasons for this. First, Exchange 2016 no longer leverages an RPC Client Access namespace.This is due to the architectural changes within the product - for a given mailbox, the protocol that services the request is always going to be the protocol instance on the Mailbox server that hosts the active copy of the database for the user’s mailbox. In other words, the RPC Client Access service is no longer decoupled from the store, like it was in Exchange 2010. Second, as mentioned, the Client Access services proxies requests to the Mailbox server hosting the active database copy. Figure 1: Client Access services (on MBX Server 1) proxying traffic to the Mailbox server hosting the active database copy (on MBX Server 3) This proxy logic is not limited to the Active Directory site boundary. Unlike Exchange 2010, Exchange 2016 does not require the client namespaces to move with the DAG during an activation event – a Mailbox server in one Active Directory site can proxy a session to a Mailbox server that is located in another Active Directory site. This means that unique namespaces are no longer required for each datacenter (mail.contoso.com and mail2.contoso.com); instead, only a single namespace is needed for the datacenter pair – mail.contoso.com. This also means failback namespaces are also not required during DAG activation scenarios, so mailpri.contoso.com and mailsec.contoso.com are removed. Namespace Models Depending on your architecture and infrastructure you have two choices: Deploy a unified namespace for the site resilient datacenter pair (unbound model). Deploy a dedicated namespace for each datacenter in the site resilient pair (bound model). It’s also worth mentioning that these choices are also tied to the DAG architecture. Unbound Model In an unbound model, you have a single DAG deployed across the datacenter pair. This DAG has Mailbox servers in each datacenter – typically all Mailbox servers are active and host active database copies, however you could deploy all active copies in a single datacenter. Mailboxes for both datacenters are dispersed across the mailbox databases within this DAG. In this model, clients can connect to both datacenters in the event there is a WAN failure – neither datacenter’s connectivity is a boundary, hence the term unbound. It does not guarantee, however, the connectivity provides users an equal experience; meaning one connection may provide a better user experience because it has lower latency or more bandwidth. In an unbound model, a single namespace is preferred because either datacenter can service the user request. This means that from a load balancing perspective, the Exchange 2016 Mailbox servers in both datacenters participate in handling traffic, as seen in the following diagram, where VIP (virtual IP address) is the load balanced IP address associated with the namespace: Figure 2: Single Namespace used across Site Resilient Datacenter Pair (Unbound Model) As a result, for a given datacenter, the expectation is that 50% of the traffic will be proxied from the other datacenter. Bound Model As its name implies, in a bound model, users are associated (or bound) to a specific datacenter. In other words, there is preference to have the users operate out of one datacenter during normal operations and only have the users operate out of the second datacenter during failure events. There is also a possibility that users do not have equal connectivity to both datacenters. Typically, in a bound model, there are two DAGs deployed in the datacenter pair. Each DAG contains a set of mailbox databases for a particular datacenter; by controlling where the databases are mounted, you control connectivity. In a bound model, multiple namespaces are preferred, two per datacenter (primary and failback namespaces), to prevent clients trying to connect to the datacenter where they may have no connectivity. Switchover to the other datacenter is a controlled event. Figure 3: Multiple Namespaces used across Site Resilient Datacenter Pair (Bound Model) Autodiscover Namespace Exchange 2016 takes advantage of the Autodiscover service for client profile configuration; so the autodiscover.contoso.com namespace remains in place. Office Online Server Namespaces The document collaboration features included in Outlook on the web require Office Online Server. In site resilient deployments, you want to deploy an Office Online Server farm in each datacenter that participates in the site resilient datacenter pair.This ensures that there is a local instance that can service the document collaboration requests for the local mailboxes and avoids cross-site proxy scenarios. From a namespace perspective, this means that each datacenter in the site resilient datacenter pair requires a unique namespace for Office Online Server; in other words, the namespace model for Office Online Server is a bound model.The namespace model that is used by Office Online Server is mutually exclusive from the model used by Exchange, meaning that you can deploy Exchange using an unbound model, while utilizing a bound model for Office Online Server as seen in the following figure: Figure 4: Office Online Server Namespaces (Bound Model) with an Exchange Unbound Model Namespace As all the data serviced by Office Online Server is either stored in Exchange or SharePoint, during a datacenter outage, namespace manipulation steps are not required. For example, if we refer to the previous diagram – if the West datacenter fails, you don’t need to change the DNS record for the Office Online Server namespace in West and point it to the load balancer in East. This is due to the architecture of Exchange and Office Online Server. Any Exchange 2016 Mailbox server will always proxy the client’s request to the Mailbox server that hosts the user’s mailbox database. The Mailbox server hosting the user’s mailbox is responsible for generating the Office Online Server URL that is used by OWA. This URL is defined per-Mailbox server, thereby ensuring that any Office Online Server interactions are always local to the Mailbox server. Internal vs. External Namespaces Since the release of Exchange 2007, the recommendation is to deploy a split-brain DNS infrastructure for the Internet-based client namespaces.A split-brain DNS infrastructure enables different IP addresses to be returned for a given namespace based on where the client resides – if the client is within the internal network, the IP address of the internal load balancer is returned; if the client is external, the IP address of the external gateway/firewall is returned. This approach simplifies the end-user experience – users only have to know a single namespace (e.g., mail.contoso.com) to access their data, regardless of where they are connecting. A split-brain DNS infrastructure, also simplifies the configuration of the Exchange virtual directories, as the InternalURL and ExternalURL values within the environment can be the same value. In the event that you do not deploy a split-brain DNS (also known as split-DNS) infrastructure, Exchange 2016 does allow you to specify different namespaces for internal clients vs. external clients for all clients. Important: In the event that you are utilizing a split-brain DNS infrastructure, then you must utilize the same authentication value for both your internal and external Outlook Anywhere settings, or switch to use different names for Outlook Anywhere inside and out. Outlook gives priority to the internal settings over the external settings and since the same namespace is used for both, regardless of whether the client is internal or external, it will utilize only the internal authentication settings. Regional Namespace The concept of regional namespaces has existed since OWA debuted in 1997. A regional namespace is a way for clients to connect to the client access endpoint that is closest to the Mailbox servers hosting the data. Use of a regional namespace does not necessarily mean you are restricted to a bound model, either. This is because depending on your infrastructure and network capabilities, you may choose to have a dedicated namespace for each datacenter pair. For example, your company may have a set of datacenters in North America and in Europe, and due to a desire to reduce cross-region network traffic, you deploy a dedicated namespace for each region (notice that within a region, the unbound model is used): Figure 5: : Regional Namespaces coupled with Geo-DNS to Round-Robin between Datacenters within a Region Namespaces and Active Directory Site Topologies When planning your namespace architecture, it is important to understand that namespaces and authentication settings must be identical and/or consistent within an Active Directory site. For example, when Autodiscover generates a response to send to the client, it generates a list of internal URLs based on the virtual directory settings of the Mailbox servers located in the Active Directory where the mailbox is located. If you attempt to have multiple namespaces within a single Active Directory site, clients will be randomly directed to different namespaces. Likewise, setting different authentication settings within an Active Directory site will lead to different behaviors for the clients. In other words, you can only define different namespace and authentication settings between Active Directory sites, not within Active Directory sites. Conclusion Exchange 2016 introduces significant flexibility in your namespace architecture, enabling deployment of a single unified namespace for a site resilient datacenter pair (or worldwide), or deployment of multiple namespaces. As we delve into the intricacies surrounding load balancing principles and client connectivity, you will understand (hopefully) how to choose the best namespace model. Ross Smith IV Principal Program Manager Office 365 Customer ExperienceLoad Balancing in Exchange 2016
Like Exchange 2013, Exchange 2016 does not require session affinity at the load balancing layer. To understand this statement better, and see how this impacts your designs, we need to look at how MBX2016 functions. From a protocol perspective, the following will happen: A client resolves the namespace to a load balanced virtual IP address. The load balancer assigns the session to a MBX server in the load balanced pool. The Client Access services located on the MBX server authenticates the request and performs a service discovery by accessing Active Directory to retrieve the following information: Mailbox version (for this discussion, we will assume an Exchange 2016 mailbox) Mailbox location information (e.g., database information, ExternalURL values, etc.) The Client Access services located on the MBX server makes a decision on whether to proxy the request or redirect the request to another MBX infrastructure (within the same forest). The Client Access services located on the MBX server queries an Active Manager instance that is responsible for the database to determine which Mailbox server is hosting the active copy. The Client Access services located on the MBX server proxies the request to the Mailbox server hosting the active copy. Step 5 is the fundamental change that enables the removal of session affinity at the load balancer. For a given protocol session, the Client Access services located on the Mailbox server now maintains a 1:1 relationship with the Mailbox server hosting the user’s data. In the event that the active database copy is moved to a different Mailbox server, MBX closes the sessions to the previous server and establishes sessions to the new server. This means that all sessions, regardless of their origination point (i.e., MBX servers in the load balanced array), end up at the same place, the Mailbox server hosting the active database copy.This is vastly different in releases prior to Exchange 2013 – for example, in Exchange 2010, if all requests from a specific client did not go to the same endpoint, the user experience was negatively affected. The protocol used in step 6 depends on the protocol used to connect to MBX. If the client leverages the HTTP protocol, then the protocol used between Mailbox servers is HTTP (secured via SSL using a self-signed certificate). If the protocol leveraged by the client is IMAP or POP, then the protocol used between the Mailbox servers is IMAP or POP. Telephony requests are unique, however. Instead of proxying the request at step 6, MBX will redirect the request to the Mailbox server hosting the active copy of the user’s database, as the telephony devices support redirection and need to establish their SIP and RTP sessions directly with the Unified Messaging components on the Mailbox server. Figure 1: Exchange 2016 Client Access Protocol Architecture However, there is a concern with this architectural change. Since session affinity is not used by the load balancer, this means that the load balancer has no knowledge of the target URL or request content.All the load balancer uses is layer 4 information, the IP address and the protocol/port (TCP 443): Figure 2: Layer 4 Load Balancing The load balancer can use a variety of means to select the target server from the load balanced pool, such as, round-robin (each inbound connection goes to the next target server in the circular list) or least-connection (load balancer sends each new connection to the server that has the fewest established connections at that time). Health Probe Checking Unfortunately, this lack of knowledge around target URL (or the content of the request), introduces complexities around health probes. Exchange 2016 includes a built-in monitoring solution, known as Managed Availability. Managed Availability includes an offline responder. When the offline responder is invoked, the affected protocol (or server) is removed from service. To ensure that load balancers do not route traffic to a Mailbox server that Managed Availability has marked as offline, load balancer health probes must be configured to check <virtualdirectory>/healthcheck.htm (e.g., https://mail.contoso.com/owa/healthcheck.htm). Note that healthcheck.htmdoes not actually exist within the virtual directories; it is generated in-memory based on the component state of the protocol in question. If the load balancer health probe receives a 200 status response, then the protocol is up; if the load balancer receives a different status code, then Managed Availability has marked that protocol instance down on the Mailbox server. As a result, the load balancer should also consider that end point down and remove the Mailbox server from the applicable load balancing pool. Administrators can also manually take a protocol offline for maintenance, thereby removing it from the applicable load balancing pool. For example, to take the OWA proxy protocol on a Mailbox server out of rotation, you would execute the following command: Set-ServerComponentState <Mailbox Server> -Component OwaProxy –Requestor Maintenance –State Inactive For more information on server component states, see the article Server Component States in Exchange 2013. What if the load balancer health probe did not monitor healthcheck.htm? If the load balancer did not utilize the healthcheck.htm in its health probe, then the load balancer would have no knowledge of Managed Availability’s removal of (or adding back) a server from the applicable load balancing pool. The end result is that the load balancer would have one view of the world, while Managed Availability would have another view of the world. In this situation, the load balancer could direct requests to a Mailbox server that Managed Availability has marked down, which would result in a negative (or broken) user experience. This is why the recommendation exists to utilize healthcheck.htm in the load balancing health probes. Namespace and Affinity Scenarios Now that we understand how health checks are performed, let’s look at four scenarios: Single Namespace / Layer 4 (No Session Affinity) Single Namespace / Layer 7 (No Session Affinity) Single Namespace / Session Affinity Multiple Namespaces / No Session Affinity Single Namespace / Layer 4 (No Session Affinity) In this scenario, a single namespace is deployed for all the HTTP protocol clients (mail.contoso.com). The load balancer is operating at layer 4 and is not maintaining session affinity. The load balancer is also configured to check the health of the target Mailbox servers in the load balancing pool; however, because this is a layer 4 solution, the load balancer is configured to check the health of only a single virtual directory (as it cannot distinguish OWA requests from RPC requests). Administrators will have to choose which virtual directory they want to target for the health probe; you will want to choose a virtual directory that is heavily used.For example, if the majority of your users utilize OWA, then targeting the OWA virtual directory in the health probe is appropriate. Figure 3: Single Namespace with No Session Affinity As long as the OWA health probe response is healthy, the load balancer will keep the target MBX in the load balancing pool. However, if the OWA health probe fails for any reason, then the load balancer will remove the target MBX from the load balancing pool for all requests associated with that particular namespace. In other words, in this example, health from the perspective of the load balancer, is per-server, not per-protocol, for the given namespace.This means that if the health probe fails, all client requests for that namespace will have to be directed to another server, regardless of protocol. Figure 4: Single Namespace with No Session Affinity - Health Probe Failure Single Namespace / Layer 7 (No Session Affinity) In this scenario, a single namespace is deployed for all the HTTP protocol clients (mail.contoso.com). The load balancer is configured to utilize layer 7, meaning SSL termination occurs and the load balancer knows the target URL. The load balancer is also configured to check the health of the target Mailbox servers in the load balancing pool; in this MBXe, a health probe is configured on each virtual directory. As long as the OWA health probe response is healthy, the load balancer will keep the target MBX in the OWA load balancing pool. However, if the OWA health probe fails for any reason, then the load balancer will remove the target MBX from the load balancing pool for OWA requests. In other words, in this example, health is per-protocol; this means that if the health probe fails, only the affected client protocol will have to be directed to another server. Figure 5: Single Namespace with Layer 7 (No Session Affinity) - Health Probe Failure A single namespace utilizing layer 7 without session affinity is the recommended namespace and load balancer configuration for Exchange 2016. Single Namespace / Layer 7 with Session Affinity this scenario, a single namespace is deployed for all the HTTP protocol clients (mail.contoso.com). The load balancer is configured to maintain session affinity (layer 7), meaning SSL termination occurs and the load balancer knows the target URL. The load balancer is also configured to check the health of the target Mailbox servers in the load balancing pool; in this MBXe, the health probe is configured on each virtual directory. As long as the OWA health probe response is healthy, the load balancer will keep the target MBX in the OWA load balancing pool. However, if the OWA health probe fails for any reason, then the load balancer will remove the target MBX from the load balancing pool for OWA requests. In other words, in this example, health is per-protocol; this means that if the health probe fails, only the affected client protocol will have to be directed to another server. Figure 6: Single Namespace with Layer 7 with Session Affinity - Health Probe Failure Note: By having session affinity enabled, the load balancer’s capacity and utilization are decreased because processing is used to maintain more involved affinity options, such as cookie-based load balancing or Secure Sockets Layer (SSL) session-ID. Check with your vendor on the impacts session affinity will have in your load balancing scalability. Multiple Namespaces / No Session Affinity This scenario combines the best of both worlds – provides per-protocol health checking, while not requiring complex load balancing logic. In this scenario, a unique namespace is deployed for each HTTP protocol client; for example: Figure 7: Multiple Namespaces with No Session Affinity Note: As seen in the picture depicted above, ECP is provided its own namespace. ECP and OWA are intimately tied together, and thus, ECP does not necessarily require its own namespace. However, ECP does have its own application pool, is the endpoint for the Exchange Administration Center, and used by Outlook clients for certain configuration items. Therefore, you may want to provide a unique namespace for ECP. The load balancer is configured to not maintain session affinity (layer 4). The load balancer is also configured to check the health of the target Mailbox servers in the load balancing pool; in this case, the health probes are effectively configured to target the health of each virtual directory, as each virtual directory is defined with a unique namespace, and while the load balancer still has no idea what the URL is being accessed, the result is as if it does know. As long as the OWA health probe response is healthy, the load balancer will keep the target MBX in the OWA load balancing pool. However, if the OWA health probe fails for any reason, then the load balancer will remove the target MBX from the load balancing pool for OWA requests. In other words, in this example, health is per-protocol; this means that if the health probe fails, only the affected client protocol will have to be directed to another server. Figure 8: Multiple Namespaces with No Session Affinity - Health Probe Failure The downside to this approach is that it introduces additional namespaces, additional VIPs (one per namespace), and increases the number of names added as subject alternative names on the certificate, which can be costly depending on your certificate provider. But, this does not introduce extra complexity to the end user – the only URL the user needs to know is the OWA URL. ActiveSync, Outlook, and Exchange Web Services clients will utilize Autodiscover to determine the correct URL. Exchange Scenario Summary The following table identifies the benefits and concerns with each approach: Benefits Concerns Single Namespace / Layer 4 (No Session Affinity) Single namespace Reduced load balancer complexity Session affinity maintained at MBX Per-server health Recommended: Single Namespace / Layer 7 (No Session Affinity) Single namespace Per-protocol health SSL offloading which may impact load balancer scalability Single Namespace / Layer 7 (Session Affinity) Single namespace Per-protocol health Session affinity maintained at load balancer Increased load balancer complexity Reduced load balancer scalability Multiple Namespaces / No Session Affinity Per-protocol health Session affinity maintained at MBX Users only have to know OWA URL Multiple namespaces Additional names on certificate Increased rule set on load balancer Multiple VIPs Office Online Server Load Balancing Exchange Server 2016 leverages Office Online Server to provide the rich document preview and editing capabilities for OWA. While this was a necessary change to ensure a homogenous experience across the Office Server suite, this does introduce additional complexity for environments that don’t have Office Online Server. As discussed in the architecture and namespace planning articles, the Office Online Server infrastructure requires unique namespaces. The load balancer is configured to maintain layer 7 with session affinity (using cookie-based persistence) for each Office Online Server namespace, meaning SSL termination occurs and the load balancer knows the target URL.This ensures the client is always directed to the same Office Online Server while the user is utilizing the document collaboration capabilities within OWA. Conclusion Exchange 2016 introduces significant flexibility in your namespace and load balancing architecture. With load balancing, the decision ultimately comes down to balancing functionality vs. simplicity. The simplest solution lacks session affinity management and per-protocol health checking, but provides the capability to deploy a single namespace. At the other end of the spectrum, you can utilize session affinity management, per-protocol health checking with a single namespace, but at the cost of increased complexity. Or you could balance the functionality and simplicity spectrums, and deploy a load balancing solution that doesn’t leverage session affinity, but provides per-protocol health checking at the expense of requiring a unique namespace per protocol. Ross Smith IV Principal Program Manager Office 365 Customer ExperienceLife in a Post TMG World – Is It As Scary As You Think?
Let’s start this post about Exchange with a common question: Now that Microsoft has stopped selling TMG, should I rip it out and find something else to publish Exchange with? I have occasionally tried to answer this question with an analogy. Let’s try it. My car (let’s call it Threat Management Gateway, or TMG for short), isn’t actively developed or sold any more (like TMG). However, it (TMG) works fine right now, it does what I need (publishes Exchange securely) and I can get parts for it and have it serviced as needed (extended support for TMG ends 2020) and so I ‘m keeping it. When it eventually either doesn’t meet my requirements (I want to publish something it can’t do) or runs out of life (2020, but it could be later if I am ok to accept the risk of no support) then I’ll replace it. Now, it might seem odd to offer up a car analogy to explain why Microsoft no longer selling TMG is not a reason for Exchange customers to panic, but I hope you’ll agree, it works, and leads you to conclude that when something stops being sold, like your car, it doesn’t immediately mean you replace it, but instead think about the situation and decide what to do next. You might well decide to go ahead and replace TMG simply based on our decision to stop selling or updating it, that’s fine, but just make sure you are thinking the decision through. Of course, you might also decide not to buy another car. Your needs have changed. Think about that. Here are some interesting Exchange-related facts to help further cement the idea I’m eventually going to get to. We do not require traffic to be authenticated prior to hitting services in front of Exchange Online. We do not do any form of pre-authentication of services in front of our corporate, on-premises messaging deployments either. We have spent an awfully large amount of time as a company working on securing our code, writing secure code, testing our code for security, and understanding the threats that exist to our code. This is why we feel confident enough to do #1 and #2. We have come to learn that adding layers of security often adds little additional security, but certainly lots of complexity. We have invested in getting our policies right and monitoring our systems. This basically says we didn’t buy another car when ours didn’t meet our needs any more. We don’t use TMG to protect ourselves any more. Why did we decide that? To explain that, you have to cast your mind back to the days of Exchange and Windows 2000. The first thing to admit is that our code was less ‘optimal’ (that’s a polite way of putting it), and there were security issues caused by anonymous access. So, how did we (Exchange) tell you to guard against them? By using something called ISA (Internet Security and Acceleration – which is an odd name for what it was, a firewall). ISA, amongst other things, did pre-authentication of connections. It forced users to authenticate to it, so it could then allow only authenticated users access to Exchange. It essentially stopped anonymous users getting to Windows and Exchange. Which was good for Windows and Exchange, because there were all kinds of things that they could do if they got there anonymously. However once authenticated users got access, they too could still do those bad things if they chose to. And so of course could anyone not coming through ISA, such as internal users. So why would you use ISA? It was so that you would know who these external users were wouldn’t you? But do you really think that’s true? Do you think most customers a) noticed something bad was going on and b) trawled logs to find out who it was who did it? No, they didn’t. So it was a bit like an insurance policy. You bought it, you knew you had it, you didn’t really check to see if it covers what you were doing until you needed it, and by then, it was too late, you found out your policy didn’t cover that scenario and you were in the deep doo doo. Insurance alone is not enough. If you put any security device in front of anything, it doesn’t mean you can or should just walk away and call it secure. So at around the same time as we were telling customers to use ISA, back in the 2000 days, the whole millennium bug thing was over, and the proliferation of the PC, and the Internet was continuing to expand. This is a very nice write up on the Microsoft view of the world. Those industry changes ultimately resulted in something we called Trustworthy Computing. Which was all about changing the way we develop software – “The data our software and services store on behalf of our customers should be protected from harm and used or modified only in appropriate ways. Security models should be easy for developers to understand and build into their applications.” There was also the Secure Windows Initiative. And the Security Development Lifecycle. And many other three letter acronyms I’m sure, because whatever it was you did, it needed a good TLA. We made a lot of progress over those ten years since then. We delivered on the goal that the security of the application can be better managed inside the OS and the application rather than at the network layer. But of course most people still seem to think of security as being mainly at the network layer, so think for a moment about what your hardware/software/appliance based firewall does today. It allows connections from a destination, on some configurable protocol/port, to a configured destination protocol/port. If you have a load balancer, and you configure it to allow inbound connections to an IP on its external interface, to TCP 443 specifically, telling it to ignore everything else, and it takes those packets and forward them to your Exchange servers, is that not the same thing as a firewall? Your load balancer is a packet filtering firewall. Don’t tell your load balancing vendor that, they might want to charge you extra for it, but it is. And when you couple that packet level filtering firewall/load balancer with software behind it that has been hardened for 10 years against attacks, you have a pretty darn secure setup. And that is the point. If you hang one leg of your load balancer on the Internet, and one leg on your LAN, and you operate a secure and well managed Windows/Exchange Server – you have a more secure environment than you think. Adding pre-authentication and layers of networking complexity in front of that buys you very little extra, if anything. So let’s apply this directly to Exchange, and try and offer you some advice from all of this. What should YOU do? The first thing to realize is that you now have a CHOICE. And the real goal of this post is to help you make an INFORMED choice. If you understand the risks, and know what you can and cannot do to mitigate them, you can make better decisions. Do I think everyone should throw out that TMG box they have today and go firewall commando? No. not at all. I think they should evaluate what it does for them, and, if they need it going forward. If they do that, and decide they still want pre-auth, then find something that can do it, when the time to replace TMG comes. You could consider it a sliding scale, of choice. Something like this perhaps; So this illustrated that there are some options and choices; Just use a load balancer – as discussed previously, a load balancer only allowing in specified traffic, is a packet filtering firewall. You can’t just put it there and leave it though, you need to make sure you keep it up to date, your servers up to date and possibly employ some form of IDS solution to tell you if there’s a problem. This is what Office 365 does. TMG/UAG – at the other end of the scale are the old school ‘application level’ firewall products. Microsoft has stopped selling TMG, but as I said earlier, that doesn’t mean you can’t use it if you already have it, and it doesn’t stop you using it if you buy an appliance with it embedded. In the middle of these two extremes (though ARR is further to the left of the spectrum as shown in the diagram) are some other options. Some load balancing vendors offer pre-authentication modules, if you absolutely must have pre-auth (but again, really… you should question the reason), some use LDAP, some require domain joining the appliance and using Kerberos Constrained Delegation, and Microsoft has two options here too. The first, (and favored by pirates the world over) is Application Request Routing, or ARR! for short. ARR! (the ! is my own addition, marketing didn’t add that to the acronym but if marketing were run by pirates, they would have) “is a proxy based routing module that forwards HTTP requests to application servers based on HTTP headers and server variables, and load balance algorithms” – read about it here, and in the series of blog posts we’ll be posting here in the not too distant future. It is a reverse proxy. It does not do pre-authentication, but it does let you put a non-domain joined machine in front of Exchange to terminate the SSL, if your 1990’s style security policy absolutely requires it, ARR is an option. The second is WAP. Another TLA. Recently announced at TechEd 2013 in New Orleans is the upcoming Windows Server 2012 R2 feature – Web Application Proxy. A Windows 2012 feature that is focused on browser and device based access and with strong ADFS support and WAP is the direction the Windows team are investing in these days. It can currently offer pre-authentication for OWA access, but not for Outlook Anywhere or ActiveSync. See a video of the TechEd session here (the US session) and here (the Europe session). Of course all this does raise some tough questions. So let’s try and answer a few of those; Q: I hear what you are saying, but Windows is totally insecure, my security guy told me so. A: Yes, he’s right. Well he was right, in the yesteryear world in which he formed that opinion. But times have changed, and when was the last time he verified that belief? Is it still true? Do things change in this industry? Q: My security guy says Microsoft keeps releasing security patches and surely that’s a sign that their software is full of holes? A: Or is the opposite true? All software has the potential for bugs and exploits, and not telling customers about risks, or releasing patches for issues discovered is negligent. Microsoft takes the view that informed customers are safer customers, and making vulnerabilities and mitigations known is the best way of protecting against them. Q: My security guy says he can’t keep up with the patches and so he wants to make the server ‘secure’ and then leave it alone. Is that a good idea? A: No. It’s not (I hope) what he does with his routers and hardware based firewalls is it? Software is a point in time piece of code. Security software guards against exploits and attacks it knows of today. What about tomorrow? None of us are saying Windows, or any other vendor’s solution is secure forever, which is why a well-managed and secure network keeps machines monitored and patched. If he does not patch other devices in the chain, overall security is compromised. Patches are the reality of life today, and they are the way we keep up with the bad guys. Q: My security guy says his hardware based firewall appliance is much more secure than any Windows box. A: Sure. Right up to the point at which that device has a vulnerability exposed. Any security device is only as secure as the code that was written to counter the threats known at that time. After that, then it’s all the same, they can all be exploited. Q: My security guy says I can’t have traffic going all the way through his 2 layers of DMZ and multitude of devices, because it is policy. It is more secure if it gets terminated and inspected at every level. A: Policy. I love it when I hear that. Who made the policy? And when? Was it a few years back? Have the business requirements changed since then? Have the risks they saw back then changed any? Sure, they have, but rarely does the policy get updated. It’s very hard to change the entire architecture for Exchange, but I think it’s fair to question the policy. If they must have multiple layers, for whatever perceived benefit that gives (ask them what it really does, and how they know when a layer has been breached), there are ways to do that, but one could argue that more layers doesn’t necessarily make it better, it just makes it harder. Harder to monitor, and to manage. Q: My security guy says if I don’t allow access from outside except through a VPN, we are more secure. A: But every client who connects via a VPN adds one more gateway/endpoint to the network don’t they? And they have access to everything on the network rather than just to a single port/protocol. How is that necessarily more secure? Plus, how many users like VPN’s? Does making it harder to connect and get email, so people can do their job, make them more productive? No, it usually means they might do less work as they cannot bothered to input a little code, just so they can check email. Q: My security guy says if we allow users to authenticate from the Internet to Exchange then we will be exposed to an account lockout Denial of Service (DoS). A: Yes, he’s right. Well, he’s right only because account lockout policies are being used, something we’ve been advising against for years, as they invite account lockout DoS’s. These days, users typically have their SMTP address set to equal their User Principal Name (UPN) so they can log on with (what they think is) their email address. If you know someone’s email address, you know their account logon name. Is that a problem? Well, only if you use account lockout policies rather than using strong password/phrases and monitoring. That’s what we have been telling people for years. But many security people feel that account lockouts are their first line of defense against dictionary attacks trying to steal passwords. In fact, you could also argue that a bad guy trying out passwords and getting locked out now knows the account he’s trying is valid… Note the common theme in these questions is obviously – “the security g uy said…..”. And it’s not that I have it in for security guys generally speaking, but given they are the people who ask these questions, and in my experience some of them think their job is to secure access by preventing access. If you can’t get to it, it must be safe right? Wrong. Their job is to secure the business requirements. Or put another way, to allow their business to do their work, securely. After all, most businesses are not in the business of security. They make pencils. Or cupcakes. Or do something else. And is the job of the security folks working at those companies to help them make pencils, or cupcakes, securely, and not to stop them from doing those things? So there you go, you have choices. What should you choose? I’m fine with you choosing any of them, but only if you choose the one that meets your needs, based on your comfort with risk, based on your operational level of skill, and based on your budget. Greg Taylor Principal Program Manager Lead Exchange Customer Adoption Team
