Large Message Processing in Exchange, Part 1: Prevention and Planning

The_Exchange_Team · ‎Jul 07 2009

EDIT 10/14/2009: We updated the section that recommends settings for environments that need to support messages larger than default 10MB message size.

Already an expert on large messages? You'll still want to check out the updated "Best Practices" section below to harden your server against possible outages.

Overview

Although email is not always the best way to share files, the method is frequently used. As an administrator, you probably have to allow messages to be sent with attachments. Sometimes these attachments are relatively large. But you also have to balance this business requirement with making sure that your server hardware does not become overly utilized or that some users are denied service while others are processing super large messages.

In Customer Support Services we see a lot of critical server unresponsive type issues caused by someone trying to attach a really large file, say perhaps someone trying to share a DVD home video .ISO with their friends and coworkers.

Although we've attempted to harden Exchange out of the box, there are still a few things that you should consider doing to further limit the possibility of something like this happening.

Reasonable limits

It all begins with "reasonable" size limits. So the question everyone asks is "what is reasonable?" The answer is that your mileage will certainly vary with your hardware, drive space, number of users, availability requirements, etc. For Exchange 2007, the typical size limits we see are in the range of 10-30 MB. There are business cases where larger messages are required. However, not all hardware is equally equipped to handle it. Think about the following cases:

Maximum allowed msg size	Maximum allowed number of recipients per message	Typical messages per second	Possible worst case peak scenario
10MB	5000	10 msg/sec	500GB/sec
30MB	5000	10 msg/sec	1.5TB/sec
100MB	5000	10 msg/sec	5TB/sec

Do you have that kind of space available? Do you want to pay for it? Can your disks keep up with that amount of I/O? Hub transports? Mailbox databases? Transaction log files? Will your archiving and retention systems be able to keep up?

Ok, so obviously this is an extreme case - every message sent is not going to be as large as the maximum limit, and not every message is going to be sent to as many recipients as possible. But you can easily see that message size is an important piece of planning for server sizing and availability. You must plan for both the average case, but also consider the peak/worst case. Even if the same chart were done with averages instead of worst cases, you can imagine that even if just a few large messages make it out per day, the average size goes up considerably. Also, your server has to be able to handle those few per day in a reasonable amount of time. Remember that it is possible to provide limits for one set of users and still allow the occasional large message to be sent by more important folks, say company executives.

Furthermore, if the possibility exists that someone can send a DVD .ISO to everyone in the company, you can bet that it will happen at some point. Will your server(s) handle it gracefully? Or will business stop because of this incident?

Our capacity and planning tools will help you make the decision, although they are designed around the average case - don't forget to at least consider that raising message sizes will not only affect the average case, it will also raise the peak load case. For more discussion on this, see this post.

Back Pressure

Exchange 2007 introduces a concept within Transport servers called Back Pressure. You can read all about it here. Suffice it to say, if your server becomes too busy, it will stop accepting new messages, and allow itself time to gracefully recover. It does this to protect itself from the extreme cases.

In short, Back Pressure is Exchange 2007's way of monitoring available disk space, memory and uncommitted messages. When any of those resources exceed their corresponding thresholds for a sustained period the HUB server stops accepting anonymous submissions (medium threshold) or all submissions (high threshold). For example:

Event Type: Warning
Event Source: MSExchangeTransport
Event Category: ResourceManager
Event ID: 15004
Description:
Resource pressure increased from Medium to High.

Resource utilization of the following resources exceed the normal level:

Version buckets = 213 [High] [Normal=80 Medium=120 High=200]

Back pressure caused the following components to be disabled:
Inbound mail submission from Hub Transport servers
Inbound mail submission from the Internet
Mail submission from the Pickup directory
Mail submission from the Replay directory
Mail submission from Mailbox servers
Mail delivery to remote domains

With large messages, you have the possibility that a database transaction to commit the message into the database will take some time to complete. During that time, the database is tracking the commit with what is called version buckets or version store. So with large messages, you can guess that version buckets will often be the measure of how the mail queue database is keeping up. A few seconds of back pressure a few times per day is fine, but if your server(s) spend a lot of time in back pressure, then there's the possibility that other messages aren't being processed in a timely fashion.

Let's also take a look at the graph below (Message Size (MB) versus Version buckets). This graph charts the size of the message sent along with the corresponding Version Buckets Allocated while the message was transmitted. The yellow line shows the default medium threshold for Back pressure resource monitoring of 120, and the red line is the default High Threshold of 200. As the message increases, the version buckets allocated continue to increase as well. When the message size reaches approximately 100MB the version buckets allocated have done beyond the medium threshold value of 120. This will cause inbound anonymous SMTP traffic to temporarily be suspended (Internal Hub to Hub and Mailbox submissions will not be affected. This chart shows that Exchange 2007 has the ability for transport to support a single message as large as 150MB without causing internal mail flow disruption - but again, remember, this is an isolated single message going to a single recipient.

Please note that these measurements were taken in the absence of processing a high rate of other smaller messages and it was performed on a server configuration that was I/O constrained (single disk).

For more information about the impact of specific messages on hardware, check out this study.

And for information about the impact of large messages on your mailbox stores, see this post.

Size Limits

Although, you've probably seen this, let's quickly review the common places that size limits are set and discuss possible ways to harden your server(s):

Global/organizational size limits - these settings control the absolute maximum size message that can be submitted to store with Outlook, and accepted by the categorizer throughout the organization. Although it is "global" in scope, there can be exceptions to this limit.
Global/organizational MaxRecipientEnvelopeLimit - this controls how many recipients can be on a single message. It is probably a good idea to consider lowering this from the default value of 5000 once Exchange 2003 is no longer in your organization as Exchange 2003 calculates this after distribution groups are expanded. But Exchange 2007 and later will consider a DG as a single recipient. The nice thing here is that you can setup the larger DGs for broadcast emailing and restrict who is allowed to send to them. You could also lower the value if say you only had 2000 mailboxes in your organization - no need to keep the default value.
Connector limits (send, receive, foreign, routing group, AD site link) - these are useful for what can pass through a particular connector. For example, you may want to allow user SMTP submissions to be 15 MB, but only 10 MB for application servers, and only 5 MB for anonymous "Internet" email. You should always have a reasonable size limit on your Internet facing receive connector(s).

Yawn. You've seen this before, right? Well what about these limits you may not remember?

OWA - By default, OWA users can upload attachments of up to 30MB for submission. This value can be modified.
DAV clients like older versions of Entourage - KB 935848 talks about the issue. You can set the MaxRequestEntityAllowed value on the w3svc/1/ROOT/Exchange key in the metabase to a value similar to the maximum size posting you want to allow from DAV clients.
Public folders - Although public folder posts do not go through transport, they can be equally dangerous to the server. If someone posts a gigantic message, it could cause the public folder store to consume system resources and run out of space. In addition, if the folder is replicated, the problem can spread throughout the organization (replication messages are system messages and bypass most other checks). You can set limits for replication messages, but large single posts will bypass this limit. Contrary to popular belief, this setting is not an upper limit on replication message size - it's more like a lower limit. When we have multiple changes to pack up, we will keep packing each change into the same message until we reach at least the replication message size limit. After that, we send the message, and start packing additional changes into the next replication message. Let's say your replication message size limit is 300k. If you post 500 1k items in a folder, 300 of them will be packed into the first replication message, and then 200 will be packed into the next one and sent. If you post 500 1MB items, each individual item exceeds the message size limit, so that just means that we won't pack any additional changes into each replication message - each item will be sent in its own 1 MB replication message. If you post a 1 GB item, the same applies - you get a 1 GB replication message. We don't split up a large item between replication messages. The only way to control this is to set reasonable item size limits, particularly on replicated folders. You can set these at the database and folder level.

Best Practices

An ounce of prevention is worth a pound of cure. So here are the best practices we recommend to protect your server(s) from large messages that might cause outages.

Install SP1 RU8. This rollup update contains an extremely important fix that should not be missed. KB 960775 is the fix that you need, particularly if you allow Outlook 2003 clients prior to SP3 to connect to your server. These clients will not ask for the maximum limits before synching and submitting a large message to the server. This can easily cause transaction log file growth and performance problems on the Mailbox server. But, worse, the store-generated DSN messages are then submitted to Transport and the problem can spread. This fix eliminates the possibility of Hub servers being affected. Regardless of this fix, it may still be a best practice to update your clients to SP3 and block legacy (unsupported) clients to limit the damage that can happen on the Mailbox server.
Run ExBPA. Although BPA does not know what's reasonable for your organization, it can make sure that at least size limits are in place. ExBPA can check all of your servers quickly.
Set reasonable size limits for your organization based on planning. See above section for commonly missed size limits. You can use the detail output from the BPA to make certain the limits are where you think that they are.
Particularly if you're supporting anything larger than the default 10MB message size, make sure that you've updated your edgetransport.exe.config file to the latest guidance for your version of Exchange. At the time of this publishing, the Exchange 2007 guidance when running the latest service pack is as follows:
- The ESE cache size should be 512MB on any server with more than 4GB of RAM -

For servers with 8GB of RAM or more, particularly if they are dedicated Hub role with transport dumpster enabled, you can set the value as high as 1GB:

The version bucket thresholds should be as follows -

The checkpoint depth should be approximately half of the DatabaseMaxCacheSize -

Or, with DatabaseMaxCacheSize set to 1GB -

The Queue Database Logging Buffer Size should be as follows -

Consider hardening and isolating Internet-facing receive connectors such that spam processing and virus scanning processes for inbound "unclean" message streams are not impacting the rest of mail flow. Set reasonable receive connector limits. This obviously transcends the large message discussion, but this is especially true if you allow larger messages.
Make sure that proper exclusions are set for file-based Antivirus software and that temporary locations are also located on drives with adequate space and speed. Temporary files can be created while converting large messages. Scanning these temporary files can cause problems - use proper Exchange Antivirus for protecting the messages and file-level scanning to protect the server(s).

Conclusion

Message size limits will protect your servers and make sure they stay happily running, but there is not any "one-size-fits-all" guidance. Nevertheless, setting reasonable message limits and following the best practices can save you a great deal of trouble.

- Scott Landry, Mohammad Nadeem and Bill Long

Products (50)

Special Topics (27)

Video Hub (462)

Most Active Hubs