Guidance on Active Directory design for Exchange Server 2007

[The_Exchange_Team]

This post focuses on two aspects of Active Directory (AD) design for Exchange 2007: 

• The recommended ratio of Exchange 2007 servers to Active Directory global catalog (GC) servers
• When to use dedicated Active Directory sites for your Exchange 2007 servers

While most of the existing guidance about AD design for Exchange 2003 applies to Exchange 2007, there are several new factors that must be considered:

• Exchange 2007's new 64-bit architecture
• Availability and mix of both 32-bit and 64-bit AD servers
• New (or upgraded) services and clients (Unified Messaging, anti-virus/anti-spam, Outlook 2007, etc.)
• New role separation and architecture for Exchange 2007 servers (Mailbox, Hub, Client Access Server, Edge, UM.)
• More use of advanced features by typical users (mobile devices, calendaring, etc.)
• Increasing average mail traffic and storage allowances
• Additional message handling (anti-spam/antivirus, compliance features) requires more GC network bandwidth and processor utilization per user/message

How Many Global Catalog Servers Do I Need for My Exchange 2007 Servers? The previous recommendations for Exchange 2003 are available at: Global catalog server placement and ratios in an Exchange 2000 Server organization or in an Exchange... The primary recommendation in the above article is that you should have 1 Global Catalog processor core per 4 Exchange server processor cores. It is important to note that this is not a 1:4 ratio of servers or even processors, but of processor cores: a dual core processor counts as 2 when doing the ratio calculations. With Exchange Server 2007, Microsoft recommends that you deploy one 32-bit GC CPU core for each four Exchange 2007 mailbox server CPU cores.  While the other server roles will influence the number of GC cores required, the Mailbox servers that are deployed influences the deployment of each of the other roles, so basing the number of GC cores on mailbox server cores will suffice. For example, suppose you have an 8 processor (single core) Exchange 2003 mailbox server. This Exchange server's Active Directory needs can be satisfied by two single-processor/single core GC machines, by one dual-processor/single core machine, or by a dual-core single-processor GC. The Exchange 2003 recommendations have been based on domain controllers running 32-bit Windows 2003. Something wonderful happens when you upgrade your domain controllers to 64-bit Windows 2003: efficiency can double! Instead of a 1:4 ratio, you can have a 1:8 ratio between Global Catalog processor cores and Exchange server cores. This assumes that you fulfill one critical requirement: You must have enough RAM installed on the 64-bit server to cache the entire Active Directory database in memory. To check the size of your Active Directory database, look on a global catalog server for the NTDS.DIT file. This file is installed by default in %WINDIR%\NTDS. On a 32-bit machine, no matter how much RAM you install, you're unlikely to be able to cache the whole database unless it is less than 1.5-1.7 GB in size. This is because of inherent memory addressing limitations in the 32-bit platform. For more details about why this is so, see the article links below. On 64-bit Windows, with its massively larger address space compared to 32-bit, you can cache even very large Active Directory databases entirely in RAM. This gets you a significant speed boost in responding to queries along with a dramatic reduction in expensive disk I/O. (If you're thinking, well, my AD database is a couple of gigabytes in size, but what if I were to use the /3GB switch to get some extra 32-bit address space for cache? Theoretically, yes, you can increase the amount of cache this way. As a practical matter, you would likely only get about 2.5-2.7 GB consistently in cache, and that would come at the price of system stability. Use of the /3GB switch on domain controllers is discouraged because it is likely to starve the machine of critical kernel resources. When this happens, the typical result is that after several days or weeks of operation, or after a peak in client demand, the server becomes unstable or unresponsive.) When you upgrade to 64-bit domain controllers, be sure you install enough RAM to cache your entire Active Directory database with some room to spare. Remember: if you don't install enough RAM, your performance will be more like a 32-bit domain controller. Keep in mind that these ratio recommendations are rules- of-thumb, not guaranteed numbers. Your environment may have unusual loads or configurations. Also, if you keep using 32-bit domain controllers with 64-bit Exchange Mailbox servers, you may skew the standard 1:4 recommendation. The ratio recommendation assumes that the DCs and the Exchange Mailbox servers are both on roughly equivalent hardware. But a 64-bit machine is likely to have more advanced processors and other components (busses, memory, etc.). If using older domain controller machines, you may need to lower the ratio. Standard benchmarks, such as the SPEC benchmarks (www.spec.org) , can give you a more realistic basis for comparing older to newer machines. The Performance Tools included in the Toolbox section of Exchange 2007's Exchange Management Console application can be very useful in reporting actual performance and finding bottlenecks. For more detailed recommendations, including lists of performance counters to monitor, take a look at these blog posts and white papers:

Measuring AD Performance (Exchange Team Blog post) Active Directory Performance for 64-bit Versions of Windows Server 2003 (Windows white paper) Exchange 2007 Processor and Memory Recommendations (Exchange Team Blog post) Microsoft Windows Kernel Memory Management and Microsoft Exchange Server (Exchange Team Blog post) Best Practice Active Directory Design for Exchange 2000 (Exchange white paperâ??still highly applicable to Exchange 2007) Planning a Microsoft Exchange Server 2003 Messaging System (Exchange white paperâ??also a good resource that mostly applies to Exchange 2007)

Regardless of your GC processor platform, you should expect some increase in network traffic between GCs and Exchange 2007 servers. This increase may be sizeable if you are taking full advantage of all the new features and roles in Exchange 2007 (antivirus, anti-spam, legal compliance features, unified messaging and so on). At Microsoft, when all of these features are heavily used, the network traffic between AD and Exchange nearly doubles. While this is a substantial increase, it has not had a large practical effect. Historically, network bandwidth has not been a common bottleneck between Exchange and Active Directory. Nonetheless, you should check your current utilization and be sure you have enough network bandwidth between Exchange servers and domain controllers as you deploy additional Exchange 2007 features. Should I Use Dedicated Active Directory Sites for My Exchange 2007 Servers? The short answer to that question is, No, not if you're creating a dedicated site only for the purpose of domain controller load balancing. This is a different answer than for Exchange 2003, so it needs some explaining. In Exchange 2003, message routing and Active Directory site topology were independent of each other. The primary reason to dedicate an Active Directory site to Exchange was to prevent other services and applications from hogging domain controllers needed by Exchange. (Or, to take a less Exchange-centric view of the universe, to prevent Exchange from hogging the DCs needed by other services.) In Exchange 2007, message routing maps directly to Active Directory site topology. Your AD site topology should therefore make sense for message routing, not just for domain controller load balancing. You should avoid creating an Exchange-dedicated Active Directory site unless doing so makes routing better too. Fortunately, the extra headroom available with 64-bit GC servers makes it easier for demanding applications to coexist in the same site, further reducing the motivation to use dedicated sites for Exchange 2007. Let's define a "demanding application" as one that:

• Continuously generates a large number of LDAP queries
• Requires ANR (Ambiguous Name Resolution) to resolve a good percentage of those queries, and
• Makes frequent authentication requests against domain controllers

If you have multiple demanding applications running in a site, you must think not only about their average loads, but also about their peak loads, and the possibility that all of those applications may all at once decide to pick on a single domain controller. As a rule-of-thumb, if you are using 32-bit domain controllers, then Exchange is likely to coexist happily with other demanding applications in a site with up to 10,000 Exchange users. If you are using 64-bit directory servers, you can support up to 20,000 Exchange users in the same site with other demanding applications. Exchange does a pretty good job of load balancing its requests across the domain controllers available to it, including taking into account how busy the DCs are with other work. So Exchange is not likely to be the site bully. But it is a demanding application and will present steady, relatively high load across all the GCs available in the site. If you are above the peak limits described above, you should look at ways to dedicate directory resources to Exchange. That does not necessarily mean isolating Exchange to another site:

• You can try to reserve domain controllers for Exchange use by being clever with DNS. By appropriately configuring SRV record priorities and weights, you can often get nearly the same effect as a dedicated site.

More about how to do this can be found here: Creating an Active Directory Site for Exchange Server (Ignore the first part of this paper and scroll down to the part about how to "Isolate Client Authentication Traffic from Exchange Facing Domain Controllers.")

• You can define a static set of domain controllers for use by a particular Exchange server (using the set-ExchangeServer cmdlet). This lets you keep Exchange from using domain controllers dedicated to other applications. While this is preferable to creating a dedicated site, it is not optimal from the perspective of ongoing management. You have to remember which GCs belong to whom and manually tune and maintain lists instead of letting Exchange automatically optimize the load balancing. 
• If another application cannot be prevented from overwhelming the domain controllers Exchange needs, then investigate your options for restricting or isolating that application before deciding to protect Exchange in a dedicated site.

If you have more than five Active Directory sites in your environment try very hard not to put Exchange 2007 in a dedicated site. In an environment more complex than five sites, it's unlikely that you will be able to design a dedicated site topology that works well for routing. Yes, you can then wrestle the Exchange 2007 routing topology into the shape you want without regard to the AD site topology. But you don't want the ongoing headache of managing and maintaining a message routing topology and an AD site topology that don't mesh together. Exchange and Active Directory are deeply entwined. Exchange 2007 takes advantage of this synergy to simplify message routing and make it more deterministic (and easier to troubleshoot). Let that work in your favor. If your Exchange routing needs are completely at odds with your AD site topology, it may be time to rethink both together. To summarize all this:

• 64-bit Active Directory servers are great news for Exchange and can double the performance of 32-bit servers.
• Take advantage of the doubling in performance you can get with 64-bit Active Directory servers. You can reduce the number of AD servers required to support Exchange and simplify both your Active Directory site and Exchange message routing topologies.

- Mike Lee

[#2]

Can you guys consider doing an article on removing the first exchange 2007 server from the site (similar to the existing KB article regarding removing the first exchange 2003 server from the site) ?

thanks!
Wes

[#3]

pesos,

This is coming, yes, it will be published as part of the Exchange 2007 documentation:

http://technet.microsoft.com/en-us/library/cb24ddb7-0659-4d9d-9057-52843f861ba8.aspx

[#4]

Does Microsoft have a calculator that would allow you to figure out how many GCS you need for Exchange, and also takes into consideration the cores?

[#5]

So CPU and AD processing throughput are increased when you use 64Bit GCs, but what about network requirements?  They're still the same, right?  Doesn't that become the new bottleneck and at what point?

[#6]

Mike, your 500MB/1GB numbers for the size of the DB cache in AD are incorrect. These numbers are *Windows 2000* numbers. Please see the updated numbers for Windows 2003. They apply to all flavors of 2k3, from RTM through SP2. Without 3GB, we tend to observe a db cache (at the ESE layer) of about 1.5-1.7GB, with 3GB just add a gig to that.

For more data, see the AD x64 white paper. We have two sample data sets...one about 2GB in size and the other about 30GB in size (roughly, I'd have to look back at the paper to refresh my memory for specifics)

[#7]

This post focuses on two aspects of Active Directory (AD) design for Exchange 2007: The recommended ratio

[#8]

Mike, you also over simplify things with the assumption that the entire database needs to be cached in memory to gain performance improvements.  This is untrue, only the data that is actually "interesting" needs to be cached.  While the size of the database is a good rule of thumb for guessing how much information there is to cache, white space and information that is "uninteresting" often compose a rather significant portion of the data.  Thus, the end goal is really to avoid going to disk as much as possible for the "heavy lifting" needs of the consumer of AD.

For reference, "interesting" and "uninteresting" are highly subjective terms directly related to the client usage profile.  For example, an AD domain controller servicing only Exchange servers is probably not going to gain much performance benefit from being able to cache all the computers objects in the forest in memory.

Also, you talk about "demanding" applications as having large volumes of LDAP requests, ANR searches, etc.  What constitutes large versus small?  What about a multitude of non-demanding applications?  Does the load in aggregate eventually become essentially equivalent to one demanding application?  This also needs to be taken into consideration if you are no longer isolating Exchange from the rest of the infrastructure.  Eric Fleischman's recommendation of reviewing the AD x64 white paper, will provide the numbers that represent what constitutes "large" relative to load on a specific DC.

[#9]

We created a dedicated site in AD for our Exchange 2003 based on 12k users to isolate traffic.  Durning my upgrade, should I abandon this site, or maintain it for 2007?

[#10]

Does Microsoft, upon booting your 64-bit AD boxes, process a large query to essentially pre-load the cache? If so, can you post the type of query you are doing and can you share how long it takes to complete the full loading of the cache?

[#11]

I have a question regarding Set-ExchangeServer.  If we wanted to use StaticDomainControllers and StaticGlobalCatalogs, how can we definine multiple Domain Controllers and Global Catalogs?  Would it be something like, -StaticDomainControllers server1.domain.com,server2.domain.com????

Also, if we decide that we want to reverse it so it can start using any server available, what would be the command?

Thanks

[#12]

Hi Elan,
The command would be:
set-exchangeserver -id <E2K7 server> -staticDomainControllers 'dc.domain.com', 'dc2.domain.com'

if you want to reverse it:
set-exchangeserver -id <E2K7 server> -staticDomainControllers $NULL

[#13]

PingBack from http://ukstokes.com/blog/?p=61

[#14]

Anyone else have any problems after Roll Up 2 for Exchange 2007?  The same day I updated my clustered servers and my hub transport server to the newest update the active cluster server says it cannot contact the hub transport server so I haven't received or can't send any email since that day.  I also can't get an updated version of the Global Address List.  Luckily this is a test server but I haven't found a solution since the hub server sees the network, the AD and everything else.  I've reinstalled the client and hub transport roles and even readded the server to the domain with no luck.  

[#15]

Why do the staticDomainControllers + staticGlobalCatalogs not appear in the list when I do a get-exchangeserver -id server |fl ?

I only see empty {}

Thanks
Max

[#16]

Hello from Greece too.
This is a complex question i think but also likely to happen.

Lets say that you have 10 sites all with exchange 2000 , when you try to upgrade it stops complaining that all sites must have at least one GC.
Ok lets say that we have upgraded on every site one DC to 2003 SP1 and we have another one running 2000. Both are GC's . So continuing the thoughts lets say that we upgraded the central site with exchange 2007 and all the other sites have exchange 2000 for some time. While in the process of upgrading all sites one DC from a site running 2003 fails and only the 2000 DC is working... What will be the problems...?
Will it effect message routing? I believe strange things will happen :) do you happen to have any ideas what... ?

Thanks for any answers

[#17]

Mio piacere,
voi fatto  speciale atmosfera :D
A ragazza   come a generi:
Se dovete prendere un Biglietti Natale, scattisi qui! Ti amo, chiave al mio cuore! Potete trasmettere i ecards animati, ecards statici, ecards di scherzo e pagine di divertimento: cartoline musicali .. Animali online frasi io via ancora un lei solo biglietto san valentino gratis ricordo.. Ma Noi gradirebbe studio fresco risorse dove presenti, From che del here http://cartoline.biglietti-italia.com/ - cartoline . Provante smeralda usare te riuscita nella biglietto di natale biglietto di natale mio della ritengono quando biglietto di natale d’auguri musica online c
remosi riuniscendo prima quando tutti dato pelle.
Puo essere ricerca poco :\

[#18]

Just carrying on from my previous blog .... Exchange 2007 has been designed to integrate even more closely

[#19]

I am using
set-exchangeserver -id <E2K7 server> -staticDomainControllers 'dc.domain.com', 'dc2.domain.com'

Powershell coems back in a few secodns, no errors then,
when I run the get after I see the {}.

Is something else required after trying to use  
staticDomainControllers  in order for it to take effect?

[Lion_Gates]

Good source :  Globallyinfo