Combining Web Farm publishing with Software or Hardware Based Load Balanced CAS arrays
Published Jul 20 2010 04:26 PM 10.4K Views

In my post just the other day I provided a link to the new guide covering Exchange publishing via Forefront TMG or UAG. One poster followed up by asking, “How do I set things up when I have both TMG/UAG with a web farm and a hardware load balancer?” It’s a great question, as these days hardware load balancers are becoming commonplace, and trying to get both Forefront TMG/UAG and the load balancer to work together is important to get right. As it happens, I had written something about this up too, and was saving it for a rainy day, so it looks like today, it’s raining. I hope this helps answer the question.

The introduction of the Client Access Server (CAS) role as the MAPI end point Outlook uses to connect to a mailbox has prompted many organizations to consider load balancing internal clients for the first time. The introduction of a load balancer to provide fault tolerance and sharing of load to client access, when combined with using a product such as Forefront TMG or UAG to publish Exchange, when those products can also provide load balancing, can be a source of confusion.

The most common question is whether Forefront TMG (Forefront TMG will be referred to throughout this section but the same is true of Forefront UAG in these scenarios) should be used to publish the Virtual IP address (VIP) created on the load balancer, as shown in the diagram below, or whether a farm of CAS should be configured on Forefront TMG, and that used as the destination for the publishing rule.

Figure 1 - All Connections through the Load Balancer

This approach of publishing the load balancer itself has both advantages and some disadvantages.

An obvious advantage is that a simple, common path now exists for both internal and external client connections, both via the load balancer. The disadvantage is that a single point of failure now exists for all client connections, though that will always be the case when concentrating connections to any form of hardware device and is usually mitigated by using redundancy in the configuration.

Another advantage is that a hardware load balancer usually has many more affinity methods available to it, and so that extra capability can be leveraged when balancing the load across the CAS.

One of the more subtle disadvantages is only clear when you consider how Forefront TMG views the health of the end point it is publishing – if the end point is a single load balancer, if there is an issue connecting to that load balancer the entire target is marked as down, whereas if Forefront TMG is treating the health of each member CAS on an individual basis, then any one member being down does not impact the entire service. This is similar to the previous case however, in that redundancy in the load balancer can help mitigate this risk.

A further issue that can cause problems in this scenario, though it is relatively easy to work around if the network configuration allows it, is that Forefront TMG typically uses its own IP address as the source IP in the TCP packets that reach the load balancer, effectively appearing to the load balancer as a single IP address, or client, which will impact the load balancers ability to distribute load based on source IP address. There are three mitigations to this problem;

  • Configuring Forefront TMG to not replace the IP address of the client with its own IP address (though this requires Forefront TMG to be set as the default gateway (or used as the ultimate exit route from the network) on the load balancing hardware (if it is decrypting SSL) to ensure the packets route back through Forefront TMG), or on CAS, if SSL is being decrypted there.
  • Configure the load balancer to use a form of affinity other than based on source IP – though this can be a problem for clients such as Outlook Anywhere where one client can create multiple SSL sessions, this can result in sessions from the same client being split across multiple CAS.
  • Configure Forefront TMG to use Bi-Directional Affinity (available only in the Enterprise version of Forefront TMG) which allows Forefront TMG to manage this complex networking scenario. There are however some caveats to this approach, which are discussed in this blog post: http://blogs.technet.com/b/isablog/archive/2008/03/12/bi-directional-affinity-in-isa-server.aspx.

One last disadvantage to this solution is that publishing the load balancer itself rather than each individual server is that certain scenarios, any that involve Kerberos Constrained Delegation (KCD) for example (certificate based authentication and NTLM Outlook Anywhere are two Exchange scenarios), cannot be configured. KCD requires that Forefront TMG utilize the Service Principal Name of the delegated service, and since SPN’s cannot be configured on more than one machine in a domain, there is no way to configure KCD from Forefront TMG to CAS in this scenario at this time. In these scenarios, publishing a single virtual IP address, that of the load balancer, would prevent KCD from working altogether.

Another potential solution is to not use the hardware load balancer and simply point all client traffic at Forefront TMG and allow it to load balance all the connections. This is shown in the diagram below, and shows all internal and external client requests being made via Forefront TMG.

Figure 2 - Use Forefront TMG as only Sole Load Balancer

The problem with this suggestion is that Forefront TMG is unable to use a farm for any protocol other than HTTP. Accessing a mailbox from an Outlook client when connected to the same network is done using RPC, POP3 or IMAP4. Neither Forefront TMG nor UAG can load balance these protocols across a farm of servers. Therefore you should not use a name that ultimately uses Forefront TMG or UAG as the MAPI end point for your Outlook clients. Whilst it is technically possible to configure Forefront TMG to make the appropriate ports available, they can only be used to publish a single IP address. This single IP could be a single server, or a load balanced IP address, though if you have load balancing available, but choose to concentrate all your connections to Forefront TMG, you are negating all the benefit of having the load balancer in the environment.

Another alternative would be to force all your internal users into Outlook Anywhere mode, so all traffic is HTTPS and can therefore utilize the Forefront TMG/UAG web farm. Some customers without hardware load balancers have done this to solve this problem, and whilst it is certainly possible, it is not necessary if you do happen to have a hardware load balancer, as we will discuss.

Knowing that Forefront TMG cannot effectively load balance RPC requests, but can load balance HTTP based traffic, you may be tempted to force all your internal Outlook clients to connect using Outlook Anywhere, using HTT P, and then allow Forefront TMG to load balance this traffic to the CAS in the web farm being published. Whilst this would work in most cases, uneven load balancing is often seen as the number of source IP addresses seen by Forefront TMG is low, particularly if NAT is being used in any part of the network, and so the connections from Forefront TMG to CAS tend to be uneven. For this reason a dedicated software or hardware load balancer is the recommended approach for internal Outlook to CAS connections.

The opposite approach is to not use Forefront TMG at all, and instead only use the load balancer at the network edge (assuming the device is designed for and supported in this scenario).

Figure 3 - Use Only a Hardware Load Balancer

In this scenario you benefit from being able to use a multitude of affinity options provided by your load balancing device, and can use the same device for internal and external load balancing if the network supports it, but you do lose the ability to pre-authenticate traffic at the perimeter of the network, and scenarios involving KCD will require that CAS be responsible for terminating the SSL stream from the client.

A better solution is using a web farm for all clients accessing via Forefront TMG, and pointing all internal clients at the hardware load balancer. The diagram below outlines this design.

Figure 4 - Use Forefront TMG to Publish Each CAS and Point Internal Client at the Hardware Load Balancer

In this configuration, a web farm of CAS is created in Forefront TMG, containing the individual CAS servers, and used as the target for all publishing rules. A virtual array is also configured on the hardware load balancer containing those same CAS servers. The internalURL and DNS settings, used by clients connecting when inside the network point to the load balancing device, and the external settings resolve to the external interface of Forefront TMG.

The advantage of this approach is that clients benefit from being able to use the hardware load balancer for all protocols, including RPC, and Forefront TMG provides the load balancing for clients accessing from the Internet, and fully support scenarios such as certificate based authentication, by being able to delegate to specific CAS within the farm.

If you require POP3 and/or IMAP4 access from the Internet, this would be the only scenario where using Forefront TMG to publish the internal Virtual IP of the load balancer would be recommended, as Forefront TMG is unable to publish those protocols to a web farm, and using the VIP as a target gives additional availability to the solution.

The final presented solution is to simply place the load balancer at the network edge (assuming the device is designed for and supported in this scenario), and use it to publish any Exchange resources that you do not wish to pre-authenticate or for which you require KCD.

Figure 5 - Split the Edge Connections between Devices as Needed

This solution allows Forefront TMG to provide pre-authentication to the Outlook Anywhere users and perform KCD back to CAS (and could easily allow certificate based authentication with KCD for ActiveSync users), and enables the load balancer itself to be used for OWA and EAS access. There is no perimeter pre-authentication for these clients, which is a trade off, but this allows the full range of load balancer affinity types to be used for these clients, and avoids the routing complexities previously discussed. It’s an unusual configuration, requiring pre-authentication for Outlook Anywhere, but not for OWA, but some customers may choose this route as they are using some kind of custom security software on their CAS to provide strong authentication, and that software can’t be installed on Forefront TMG.

The choices available to you are summarized in the table below.

 

Depicted in Figure

Network Edge

Internal Clients

Advantages

Disadvantages

Figure 1

Forefront TMG publishes Hardware load balancer VIP

Hardware load balancer VIP

  • Simple Configuration
  • Ability to leverage multiple affinity types
  • HW load balancer requires redundancy to avoid being a single point of failure and marked as down by Forefront TMG
  • Network routing can be a problem
  • Cannot be used if Certificate Based Authentication or NTLM Outlook Anywhere with pre-authentication is required

Figure 2

Forefront TMG balances load over each and every CAS

Route all traffic to TMG

  • Removes the need for the additional cost of the load balancer
  • Cannot provide resilient and load balanced RPC Client Access to internal Outlook clients
  • Likely poor load balancing for internal clients due to small source IP pool
  • Network configuration may make this difficult to implement

Figure 3

Load Balancer balances load over each and every CAS

Hardware load balancer VIP

  • Removes the need for the additional cost of Forefront TMG
  • Allows most affinity methods to be used
  • No ability to pre-auth traffic entering via the load balancer
  • Network configuration may make this difficult to implement
  • Scenarios involving KCD require SSL termination and certificate validation to be done on CAS

Figure 4

Forefront TMG balances load over each and every CAS

Hardware load balancer VIP

  • Certificate Based Authentication or NTLM Outlook Anywhe re with pre-authentication is possible
  • Hardware load balancer can balance Outlook RPC traffic effectively
  • Ability to leverage additional affinity types
  • Two load balancing pools to manage

Figure 5

Forefront TMG balances load over each and every CAS and

Load Balancer balances load over each and every CAS

Hardware load balancer VIP

  • Certificate Based Authentication or NTLM Outlook Anywhere with pre-authentication is possible
  • Ability to use multiple affinity types
  • Multiple external namespaces required
  • No ability to pre-auth traffic entering via the load balancer

Conclusion

The decision as to which of these solutions you should deploy will come as a result of understanding the scenarios you wish to support, and considering the network implications that can impact routing and load balancer effectiveness. It is important to understand that if you require pre-authentication of traffic in the perimeter network, then you need to deploy Forefront TMG, but if you don’t, you could simply use the load balancer to do load balancing for internal and external users. If you realize that you need to load balance RPC Client Access traffic, you need a hardware or software load balancer, as you cannot do that with Forefront TMG. If you ultimately want the best of both worlds, you may decide to deploy both, and use them for different purposes. As long as you carefully plan your requirements, you should be able to make the decision based on your needs, but always remember to keep one eye on the future. Things can change!

- Greg Taylor

21 Comments
Not applicable
I'm considering deploying the option described in figure 4. In the description you say that the TMG web farm and the load balancer's virtual array should contain the "individual CAS servers. Does this mean that I should not configure a CAS array?
Not applicable
Did that work on an article about Hardware Load balancing already in April. but it is in German: http://www.loadbalancerblog.de/loadbalancing-anwendungen/load-balancing-exchange-2010-berndkruczek/
Not applicable
Hi Mike, you would still need to create an RPC Client Access array, for Outlook users to connect to, and to enable you to set on each database. But from a TMG/UAG perspective, you would add each server individually as members of the web farm you create. Then TMG/UAG can pick one CAS for each connecting client to use when connecting over HTTPS.

The fact that the same CAS is also probably part of the user's RPC Client Access array (which it would be if the users mailbox database were in that same AD site and using those CAS for it's MAPI endpoint), means the https traffic hits the rpc proxy on the CAS, then effectively loops on the same box to the RPC Client Access array (on TCP ports 6001, 2 and 4).

So yes, still create an array, for MAPI usage, but add each server individually, to get the rpc traffic, inside the HTTPs traffic, to the CAS.
Not applicable
Having gone through your white paper this topic immediately sprang into my mind.  So what a pleasure to check the RSS feed this morning to find this great article.  Thanks very much!!!
Not applicable
Greg,

Thank you so much !!!
Your are the best Exchange guy on Earth ! who said after Ross ? ;))
Not applicable
Something that's always bothered me is that certificate-based authentication is available for ActiveSync, but there's no supported way to do similar two-factor authentication for Outlook Anywhere.  Am I missing something?

At the moment we have to restrict remote Outlook users to a VPN.  If Outlook Anywhere supported two-factor authentication, it would be preferable to a VPN, which is a bit more fragile over unreliable internet connections.
Not applicable
Hi Luke, yes, it's a challenge, and you are correct that at the current time Outlook Anywhere does not support two-factor authentication.

There are a few options to consider though:

Direct Access - Direct Access extends the internal network out to your client machines, allowing only managed machines to connect. It's a seamless experience.

SSL VPN's - there are some solutions that can make VPN access pretty invisible to the user.

Something else that I have planned for a future paper - I do have a solution that might help allay those security fears, it's not two factor, but does provide significently increased security around controlling which machines are able to access Exchange remotely. I'm working on it.
Not applicable
Can the ExchangeAB SPN be unregistered for Exchange 2010?  I have a server running Exchange 2010 with Update Rollup 4 applied and after I unregister the ExchangeAB SPN, once I reboot the server, the SPN comes back.  Is this supposed to work this way?  I'm trying to use:

http://support.microsoft.com/kb/927612

This worked on my Exchange 2007 servers.
Not applicable
I'm not sure, I can check, but I'm more interested to know why you feel you need to do that. Are you having address book problems? Or is it for some other reason?
Not applicable
It's for another reason.  I have Autodiscover setup in EXCH2007 as it's own website so that it can use it's own SSL cert since I use (free) SSL certs from IPSCA for EDU institutions.  We don't have enough $$ to pay for the SAN SSL certs.  In EXCH2007, none of my servers have the ExchangeAB SPN registered and this setup works great for Autodiscover as one server has basically 2 sites in IIS, Default and Autodiscover.  I've then added another IP address for that server and Autodiscover only listens on that IP.  Then I've published that IP in DNS so that all of the SCP's and URL's for OOF, AAS, and Autodiscover are able to contact it.  I'm hoping I can do this in EXCH2010 but want to know if that SPN should be re-registering itself everytime?
Not applicable
I understand you have second up a second web site for AutoDiscover becuase you want to use two certs with different names on them, but I don't see how the SPN comes in to play. The A record for AutoDiscover (or the pointer in the SCP) should point at that additional IP, but the FQDN's of the rpc client access array, the ews, oab vdirs, etc, should resolve to the first web site. Are you saying when you do this you have issues? And if you do, what exactly are they? Your solution is to try and remove SPN's, but what's the problem that lef you to this conclusion?
Not applicable
Yes, with Exch2007, I couldn't setup a new mail profile using Outlook and Autodiscover.  I worked with a Microsoft Tech rep and once we unregistered the SPN on my Exch2007 server, the Autodiscover piece of setting up a new mail profile completed successfully.  The KB article I used I listed above as basically, when I setup a new Outlook profile, I couldn't use "negotiate authentication" and had to use "NTLM authentication."

This is the same scenario I am seeing now with Exch2010.  With a mailbox that's on Exch2010 server, I can't setup a new Outlook profile and have Autodiscover successfully finish.  I am continually prompted for a username/password.  I also can't contact the OOF and AS service if I manually create a new Outlook profile.  This was exactly the same with Exch2007 and once I unregistered the SPN, these issues went away with Microsoft Tech rep.

With a mailbox that's still on Exch2007, the new Outlook profile and mailbox setup completes successfully using Autodiscover.  This is has been tested with Outlook 2007 and 2010 with the same results so I know it's an Exchange 2010 config issue.
Not applicable
Interesting problem, and the reason why you can't get to OOF adn AS when you configure the profile manually of course is because you can't get to AutoDiscover.

I do have one suggestion though, in IIS, on the site where you have added the new AutoDiscover VDir, go into IIS, on the AutoDiscover virtual directory, then go to Authentication. Select Windows Integrated, and click on the Providers link in the Actions pane. Remove Negotiate, add NTLM only. Then Outlook can only do NTLM against the service, and SPN's become a non-issue.
Not applicable
Well, I've tried that, but I don't think it really matters.  Since the user's mailbox is on the Exch2010 server and that server has the SPN registered, it never passes the credentials onto the Autodiscover site/server.  Since it has the ExchangeAB SPN, I'm thinking the Exch2010 server thinks it needs to respond to the authentication request and since the Autodiscover site isn't on the Exch2010 server...the authentication loop password prompt keeps happening over and over.  

Is there any way to validate that the SPN WILL get re-registered everytime at reboot, even if removed?
Not applicable
Thanks Greg.  Any clues about the not-two-factor method that could let us control which machines are able to access Exchange remotely?  We're in the middle of trying to solve it.  Are we just talking Win7 Direct Access?
Not applicable
@kylet - I'm going to try and set this up myself and have a look when I have a moment, as frankly, it shouldn't be this hard to get this to work.

@Luke - drop me a line directly, grtaylor @ and I'll see if I can help. And no, it's not Direct Access.
Not applicable
Thanks Greg...I appreciate it.  Let me know what you find out!
Not applicable
Any updates Greg?
Not applicable
Kylet, so I added a new web site and VDir for AutoDiscover and it works just fine for me. A few thoughts - are you removing the Vdir in the Default Site? You don't need to. Also, the SCP for internal users can point at the AutoDiscover Vdir in the default site, that's fine - only external users need to use the DNS name/cert on the second site - and when I tested that, mine worked fine. Feel free to drop me a line directly if you want and let's see if we can figure this out. grtaylor @.
Not applicable
Thanks!  So, on my 2010 Exchange server I changed the -AutoDiscoverServiceInternalUri to servername.externaldomain.edu/autodiscover/autodiscover.xml and now the OOF works for my Outlook client and the "Test Email Autoconfiguration" works!  It hasn't worked with Outlook 2010 and Exchange 2010 until now.

With Outlook 2010 or 2007 and Exchange 2007, I've had those -AutoDiscoverServiceInternalUri's set to look at that second site, autodiscover.externaldomain.edu/autodiscover/autodiscover.xml and the OOF and "Test Email Autoconfiguration" always worked.

That change made the difference in getting Outlook 2010 to work!  Your thoughts?
Not applicable
:D
Version history
Last update:
‎Jul 01 2019 03:52 PM
Updated by: