MAPI Messaging Benchmark Being Retired

MAPI Messaging Benchmark (MMB3/MMB4) has been the benchmark for measuring the performance and scalability of computers running Microsoft Exchange Server.  While MMB has served its purpose, it also has limitations in helping our customers compare solutions.  Among the most significant limitations, MMB does not take into account variation in end-user usage profiles, modification and updates to storage solutions, differences in customer topologies, and impacts of other features or applications on performance.  MMB compares single servers, but does not compare solutions. As the era of benchmarking changes to adapt to different types of clients, user profiles and other variables, the current methods are becoming impractical.  With that said, Microsoft will be closing down the MMB3 site as well as the Performance Benchmark result site, in the first quarter of 2008.  Although the MAPI Messaging Benchmark results will not be posted by Microsoft, we will continue providing the tools necessary to allow Microsoft partners to perform their own performance and scalability benchmark analysis for Exchange Server.  These tools include: Exchange Server JetStress, Exchange Load Generator, and Exchange Server Stress and Performance (this tool is in the process of being merged into LoadGen). In place of the current MMB3 site, we will work with Partners to provide links to their respective sites where customers can find real world performance and scalability studies which assess actual customer scenarios and solutions. Q&A Q: What is MAPI Messaging Benchmarks? A: MAPI Messaging Benchmarks (MMB) has been the benchmarking standard for measuring the performance and scalability of computers running Microsoft Exchange Server.  MMB evaluates the messaging performance of a server by measuring how it scales up when adding concurrent load that is modeled after typical corporate e-mail use. An operating environment with higher MMB results than another is able to hold more users per server, although the exact number is not the MMB score.  The MMB3 site gives a good overview of what MMB is designed to do, while the performance benchmarks for computers running Exchange 2003, gives results. Q: Why is Microsoft Exchange no longer investing in MAPI Messaging Benchmarks? A: MMB helps compare single servers, but does not do a good job comparing solutions.  MMB has a number of limitations in helping customers compare solutions:

• MMB does not take into account variation in user profiles > There are many differences in average messages sent/received per day, mailbox size, e-mail client used, etc... which are not factored in with MMB.
• MMB does not take into account variation in storage solutions > MMB factors in the hardware, but not the storage.  Adding a huge impractical storage system will give totally different results and therefore comparison is no longer with just the hardware, but the storage solution as well (Note: regarding the importance of storage to Exchange Server solutions, see the Exchange Solution Reviewed Program website)
• MMB does not take into account variations in customer topologies > Differences exist in the number and size of hub and branch office sites, network bandwidth between sites, and other factors not measured with MMB.
• MMB does not take into account other features or applications that impact performance > Including Microsoft Forefront, other Internet protocols, Outlook cached mode, or desktop search products.