Update 4/26/11: This post has been updated to include additional steps to ensure Kerberos authentication can be used for OAB downloads by domain-connected Outlook clients.
With Exchange 2010, a major change was instituted in the way clients connect and access mailbox related data. Unlike previous versions, clients no longer connect directly to the Information Store on the Mailbox server role to access mailbox data. Instead, clients connect to a set of services on the Client Access server (CAS) role and services within the CAS role access mailbox data using MAPI/RPC from the Mailbox server on behalf of the connecting user.
This architectural change provides many benefits, including:
- A common code path for different client types for message body content conversion.
- Data validation logic. For example, calendar versioning which is used by the Mailbox Assistant, the Calendar Repair Assistant.
- Compliance features like Single Item Recovery or Legal Hold.
- Better client experience during switchovers/failovers by obfuscating the Mailbox server hosting the active mailbox database copy.
- A unified interface for Address Book-related data.
To facilitate high availability and fault tolerance, customers must deploy multiple Client Access servers within the Active Directory sites that host Mailbox servers. To achieve high availability and fault tolerance, you must deploy one or more unified namespaces that are mapped to a virtual IP address of a load balancer. The load balancer must be configured to accept requests over specific TCP ports and provide persistence, where required, for the client protocol being used.
Kerberos Authentication Explained
Typically, domain-joined clients/applications either leverage NTLM or Kerberos for authentication. Other clients, like web browsers, can also use Basic authentication secured with SSL encryption, regardless of the client’s location (inside or outside of the corporate network). The actual authentication mechanism used depends on the configuration of both the client and the server and they negotiate the authentication to be used during the establishment of the connection. MAPI supports Kerberos authentication and the default setting in Outlook 2007 and later is to negotiate the strongest authentication available when not running in Outlook Anywhere mode.
In Exchange 2010, MAPI clients connect to load-balanced array of servers, and not an individual server with its own unique network identity. This change in the messaging architecture presents a challenge, however. In previous Exchange versions, clients connected directly to the Mailbox server which was a single identity on the network. This meant that the client, if capable, could utilize Kerberos authentication for establishing the session with the Mailbox server.
To understand the legacy behavior and how this is a challenge in Exchange 2010 let’s review Kerberos authentication, as documented in How the Kerberos Version 5 Authentication Protocol Works:
- The client contacts the Key Distribution Center's authentication service for a short-lived ticket (a message containing the client's identity and —for Windows clients –SIDs) called a ticket-granting ticket (TGT).
- The authentication service (AS) constructs the TGT and creates a session key the client can use to encrypt communication with the ticket-granting service (TGS). The TGT has a limited lifetime. At the point that the client has received the TGT, the client has not been granted access to any resources, even to resources on the local computer.
- The client wants access to local and network resources. To gain access, the client sends a request to the TGS for a ticket for the local computer or some network server or service. This ticket is referred to as the service ticket. To get the ticket, the client presents the TGT, an authenticator, and the name of the target server (the Service Principal Name or SPN).
Important: SPNs are unique identifiers for services running on servers. Every service that uses Kerberos authentication needs to have a SPN set for it so that clients can identify the service on the network. If a SPN is not set for a service, then clients will have no way of locating that service. Without properly set SPNs, Kerberos authentication is not possible. Additionally, in Windows Server 2003, KDCs will not issue a service ticket for an account that does not have a SPN.
- The TGS examines the TGT and the authenticator. If these are acceptable, the TGS creates a service ticket. The client's identity is taken from the TGT and copied to the service ticket. Then the ticket is sent to the client.
- After the client has the service ticket, the client sends the ticket and a new authenticator to the target server, requesting access. The server will decrypt the ticket, validate the authenticator, and for Windows services, create an access token for the user based on the SIDs in the ticket.
- Optionally, the client might request that the target server verify its own identity. This is called mutual authentication. If mutual authentication is requested, the target server will take the client computer's timestamp from the authenticator, encrypt it with the session key the TGS provided for client-target server messages, and send it to the client.
With Exchange 2003 and Exchange 2007, each Mailbox server would have the following SPNs created and assigned to the Mailbox server’s computer account in Active Directory:
- exchangeMDB/<Mailbox server FQDN>
- exchangeRFR/<Mailbox server FQDN>
- exchangeAB/<Mailbox server FQDN>
These three SPNs enabled a client that was configured with either Negotiate or Kerberos authentication to successfully obtain a service ticket from the TGS. In turn, the Mailbox server could decrypt the service ticket and create an access token and provide access to the mailbox data.
Exchange 2010, however, leverages a load-balanced namespace for accessing mailbox data. The FQDN that a client uses to connect to Exchange resolves to a group of load-balanced Client Access servers. Given that an SPN can only be registered to one unique entity within the Active Directory environment, this means:
- During CAS install, Exchange Setup cannot create load-balanced namespace SPNs for the address book, referral and RPC Client Access SPN records.
- Administrators cannot manually register the same load-balanced namespace SPNs on each CAS computer account.
To understand this problem and why either of the above items would not work, let’s assume the Outlook profile is configured to use the load-balanced namespace outlook.contoso.com for its home server property and it is connecting to the CAS member cas1.contoso.com:
- Client contacts KDC and obtains a TGT.
- Client sends a request to the TGS based on the TGT, the authenticator, and the name of the target server. In this case, the name of the target server outlook.contoso.com, and not the FQDN of the Client Access server.
- Client retrieves service ticket and sends it to outlook.contoso.com, which happens to get directed by the load balancer to cas1.contoso.com.
- cas1.contoso.com fails to decrypt the service ticket because:
- Its name does not match outlook.contoso.com.
- The SPN is not associated with cas1.contoso.com or multiple objects within Active Directory have an SPN associated with the associated service and outlook.contoso.com.
- Client fails to obtain an access token using Kerberos authentication.
As a result, when a mailbox is moved to Exchange 2010, Outlook and other MAPI clients that are configured to use Negotiate will ultimately end up using NTLM authentication.
NTLM Authentication Concerns
Why is using NTLM authentication for Outlook and other MAPI clients a concern? Let’s take a look at the authentication process:
- The user launches Outlook.
- The user’s computer sends traffic to the server (load-balanced namespace) specified in the Outlook profile. This traffic includes the user’s integrated authentication information (in other words, NTLM authentication).
- The load balancer directs the traffic to a specific CAS member within the load-balanced array.
- The CAS member needs to verify the user’s credentials. It does this by sending traffic to a specific domain controller, the one which it has an associated secure channel binding, requesting a verification of the user's credentials.
- The domain controller responds to the CAS member with the information on the user’s credentials.
- The CAS member generates an access token and services the request.
The problem here is that each connection from the client must follow this same process. Therefore, as the number of connections increase, there is potential for a bottleneck in terms of handling the NTLM authentication. The bottleneck is a result of the following two issues:
- The CAS member only uses a single domain controller for all authentication requests. In addition, there is no load distribution mechanism across the CAS members in the load balanced array to ensure that each CAS is using a different domain controller for its secure channel binding.
- Windows limits the number of concurrent secure channel calls. There are a specific number of threads that can handle NTLM authentication (controlled by MaxConcurrentAPI value, with default equaling 2).
Each MaxConcurrentAPI thread can only deal with one authentication request at a time. However, attributes like network latency, the number of hops the authentication requests have to go through, and the volume of requests can introduce a bottleneck in the authentication process, resulting in some of these transactions waiting longer than a remote client can tolerate. If you have a heavy load, you are more than likely going to hit that limit when these clients authenticate via NTLM. You can monitor for NTLM failure events using the Netlogon performance counters (see KB 928576), as well as, the frequency of event ID 5783.
Windows Server does have a means to increase the number of concurrent secure channel calls up to 150 when you deploy KB 975363. However, increasing the number of concurrent secure channel calls up as far as 150 is also not recommended, as it can have adverse effects on domain controller performance.
The Recommendation
If increasing the number of concurrent secure channel calls is not the answer, then what is? Prior to Exchange 2010 SP1, unfortunately there was no answer. Thankfully in Exchange 2010 SP1, we introduced a solution. The solution involves leveraging an alternate service account (ASA) credential mechanism to enable Kerberos authentication for MAPI clients.
The Microsoft Exchange Service Host service that runs on CAS has been extended to use a shared credential for Kerberos authentication. This service host extension monitors the local machine. When credentials are added or removed, the Kerberos authentication package on the local system and the network service context is updated. As soon as a credential is added to the authentication package, all client access services can use it for Kerberos authentication. The Client Access server will also be able to authenticate service requests addressed directly in addition to being able to use the ASA credential. This extension, known as a servicelet, runs by default and requires no configuration or action to run.
The steps to deploy the ASA credential are as follows:
- Create an account to be used as the ASA credential.
- Deploy the ASA credential to the CAS members.
- Convert the OAB virtual directory to an application.
- Assign the SPNs to the ASA credential computer account.
Create an account to be used as the ASA credential
All computers within the Client Access server array must share the same service account. In addition, any Client Access servers that may be activated as part of a datacenter switchover must also share the same service account. In general, it's sufficient to have a single service account per forest.
You can create a computer account or a user account for the alternate service account. Because a computer account doesn’t allow interactive logon, it may have simpler security policies than a user account and is therefore the preferred solution for the ASA credential. If you create a computer account, the password doesn't actually expire, but we still recommend that you update the password periodically. The local group policy can specify a maximum account age for computer accounts and scripts may be scheduled to periodically delete computer accounts that don’t meet current policies. To ensure that your computer accounts aren't deleted for not meeting local policy, update the password for computer accounts periodically. Your local security policy will determine when the password must be changed.
Deploy the ASA credential to the CAS members
To enable deployment of the ASA credential, a script has been created and is shipped with Exchange 2010 SP1. The script is named RollAlternateServiceAccountPassword.ps1 and is located in the Scripts directory. For more information on how to use the script, please see Using the RollAlternateserviceAccountPassword.ps1 Script in the Shell.
Conve rt the OAB virtual directory to an application
Out of the box, the OAB virtual directory is not a web application and, therefore, is not under the control of the Microsoft Exchange Service Host service. As a result, Kerberos authentication requests to the OAB virtual directory cannot be decrypted by the ASA credential.
To convert the OAB virtual directory to a web application, execute the ConvertOABVDir.ps1 script on each CAS member. The script will also create a new application pool for the OAB virtual directory. You can download the script here.
Assign the SPNs to the ASA credential computer account
After you create the alternate service account, you must determine the Exchange service principal names (SPNs) that will be associated with the ASA credentials. The list of Exchange SPNs may vary with your configuration, but should include at least the following.
- http Use this SPN for Exchange Web Services, Offline Address Book downloads, and the Autodiscover service.
- exchangeMDB Use this SPN for RPC Client Access service.
- exchangeRFR Use this SPN for the Address Book service.
- exchangeAB Use this SPN for the Address Book service.
The SPN values must be configured to match the service name being used on the load balancer, rather than on individual servers.
For example, if you have a single Active Directory site, your environment may resemble the one in the following illustration.
Based on the fully qualified domain names that are used by internal Outlook clients in the previous illustration, the following SPNs would need to be deployed on the ASA credential:
- http/mail.corp.contoso.com
- http/autod.corp.contoso.com
- exchangeMDB/outlook.corp.contoso.com
- exchangeRFR/outlook.corp.contoso.com
- exchangeAB/outlook.corp.contoso.com
External or Internet-based clients that use Outlook Anywhere won’t use Kerberos authentication as they cannot directly contact a KDC. Therefore, the fully qualified domain names that are used by these clients don’t have to be added as SPNs to the ASA credential.
After you have identified the required SPNs, you can assign them using the SETSPN command line tool.
For more information, including information on how to plan what SPNs you should deploy with your ASA credential, see Configuring Kerberos Authentication for Load-Balanced Client Access Servers.
Conclusion
Due to scalability limits when using NTLM authentication, Microsoft recommends deploying the ASA credential solution so that domain-joined and domain-connected Outlook clients, as well as other MAPI clients, can leverage Kerberos authentication.