Transports use of Temp Tables during message processing and bifurcation in Exchange Server 2000 and 2003

[The_Exchange_Team]

This blog post discusses Transport's use of the Temp Table folders during message processing (specifically during bifurcation) and mentions some common examples and scenarios in which Temp Tables gets used by Transport during bifurcation. Bifurcation is a process where Transport splits a message into multiple copies when required, for example during Distribution Group Expansion. There are few Support scenarios where deeper level knowledge about Transports' use of the Temp Table folders, physical persistence of messages within Exchange Server system, and which mail flow scenarios utilizes Temp Tables, can be very useful. For example, few of the more common of these scenarios are where duplicate/multiple messages may be send by the Exchange Server due to messages being stuck/orphaned inside Temp Table folders, conversion problems, and in rarest of circumstances where Admins believe that Messages have disappeared on the Exchange Server. But before we want to talk about Transport's use of the Temp Table folders, to put things in proper perspective let's discuss some of the related topics briefly, for example, Store Driver, SMTP System Mailbox, and Temp table itself.

When we look at messages in Exchange System Manager (ESM) Queues, we are merely looking at an in memory object representing the message. This object or memory structure is called IMsg (or IMailMsg). IMsg can be best described as a header + Pointer. Header is the Message header and pointer points back to the file system (NTFS or ExIFS) where the message physically persists. The messages themselves (along with many message properties like the 2821 properties, categorizer status, retry status, submission time, etc) are stored either on the disk (NTFS - Queue folder) or in the Information Store's SMTP mailbox / Temp tables (ExIFS). Where the message physically persists depends on where the message came from:

• Message physically resides in NTFS file system if it was submitted to the local SMTP service via the SMTP port, which is typically port 25, (i.e. off the wire, including SMTP clients) or via the pickup directory.
• Message persists in the store if it was submitted to Transport via the store (MAPI client, OWA client, MTA & Exchange Foreign connectors). In this case, the message will not be stored directly on the NTFS file system at any point during message processing, and it will only physically leave the store when it has been successfully delivered.

STORE DRIVER (SD) is made up of two dlls: ExSMTP.dll (Store Driver on the Store side) and Drviis.dll (Store Driver on the Inetinfo side). They communicate with each other through Epoxy.dll, which is an efficient Inter Process Communication (IPC) facility, and a shared memory queue. Store driver delivers inbound messages from SMTP Advanced Queuing (AQ) to Exchange Store for local delivery. It submits messages from Store to AQ for categorization (and possibly but not necessarily, outbound delivery). It delivers messages from AQ to non-SMTP gateways. It persists queued IMsg to the store if needed (e.g. for content conversion), and deletes messages when transport is done.

The following are the main Message delivery paths:

• Local Delivery: SMTP VS > SD > Store Mailbox
• Gateway Delivery: SMTP VS > SD > Store's MTS-IN
• Persisting and deleting message for bifurcation: SMTP VS (via IMsg) > SD > Store (including. Imail) > SD > SMTP VS
• "Outbound" Message Submission: Store SendQ, TempTable, or MTS-OUT > SD > SMTP VS

The SMTP SYSTEM MAILBOX is a mailbox that Transport SMTP Virtual Servers (utilizing the Transport side store driver (Drviis.dll)) uses to get to messages inside the Store Temp Tables folders during course of message processing. For each mailbox store and public folder store, there is a corresponding SMTP System Mailbox object under the container. The distinguished name of the object is displayed in the property homeMDBBL of the store:

CN=SMTP (RECMAIL5-{A635FADE-55AC-4F29-956C-BC32CD27337D}),CN=Connections,CN=org name,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=companyA,DC=com

If this object is missing, when you send messages to the corresponding mailbox store or public folder store, you receive the event id 326 and / or 327, which is covered in the following article:

Q280489 XADM: Event ID 326 and 327 Messages Are Logged
http://support.microsoft.com/default.aspx?scid=kb;EN-US;Q280489

For more data about SMTP mailbox, see this.

TEMP TABLES are basically subfolders of the SMTP logon mailbox root in a given MDB. They perform the following roles and responsibilities:

• Act as a temporary Storage place for messages submitted via local Store Mailboxes while they are processed by Transport
• Persisting messages for bifurcation (for content conversion, DL expansion).
• Periodically removing messages from Send Queue (preventing "mail stuck in the Outbox" complaints).
• Enumerated via asynchronous op on VS logon to MDB.
• Can exist in every MDB for each virtual server to logon to it. Called TempTable#1, TempTable#2 etc... By default have 20 subfolders each for storing actual messages: configurable via
  "HKLM\Software\Microsoft\Exchange\StoreDriver\Exchange\instance#\TempTable" Registry value.

To gain access to Temp Tables, view messages in them and view mapi properties on the messages MFC Mapi tool can be used. MFCMapi (aka Mapi Editor) can be downloaded from:

http://www.microsoft.com/downloads/details.aspx?FamilyID=55fdffd7-1878-4637-9808-1e21abb3ae37&Displa...

Having discussed above concepts, let's further discuss more details, specific scenarios and examples, which sheds light on specific use of Temp Table folders by Transport during message processing and bifurcation.

There's of course, a bunch of code involved to do this. In Categorizer (Cat), we see the decision to force bifurcation into the store. We call a function to use Store Driver. At this point in the code, we have an in-memory copy of a message that we are ready to associate with a message store. One of these function calls can pass in NULL value for a parameter, which means "use the same storage for the copy as the original". i.e. NTFS > NTFS or EXIFS > EXIFS. Otherwise we pass in a pointer to the Exchange store driver as an override to force * > EXIFS. This triggers a call to reallocate messages in the store driver code.

Let's take an example of inbound SMTP message processing. Inbound SMTP messages aren't copied to temp table folders in a basic MIME to MAPI conversion by the IMail.  This does NOT copy the message in the temp table folder in the SMTP system mailbox of the store.  IMAIL doesn't copy messages to/from folders either. It always operates on messages that are already in a folder. The inbound SMTP scenario involves transport delivering Internet mail to the store (a copy of the message is put into someone's inbox folder) and then IMAIL operates on that message in various ways depending on their client access pattern i.e. no temp table involvement here.  BUT if the same inbound SMTP message requires bifurcation (see bifurcation scenarios below) e.g. message journaling, Transport WILL absolutely create message copies in the temp table folder.  So the answer to the question "Do inbound SMTP messages get persisted in Temp Tables or not" basically is "It really depends on the scenario".  Not all bifurcation scenarios copy messages into the temp table either.  Whether the message gets moved into temp table folder (NTFS > EXIFS, EXIFS > EXIFS) depends on  what kind of bifurcation is required.  Below is a list of possible bifurcation scenarios organized by mailm sg backing storage.

Any mail that arrives through pickup, SMTP or that is generated by our DSN code starts on NTFS. Mail submitted through the store (DAV, X.400, any EDK gateway, MAPI) starts in EXIFS. "EXIFS" means the message is in the store – the original may be in the temp table, some user's outbox, or MTS-Out, and any copies made are always made into the temp table.

1. Preserve store driver: NTFS > NTFS and EXIFS > EXIFS

Messages bifurcated to NTFS are obviously not in the temp table – they're stored as an EML file in the VSI's queue directory. But the EXIFS > EXIFS copies are made into a temp folder, since every copy into EXIFS needs to be put into some folder in the store (that's how the store organizes mail).

EXIFS > EXIFS

• The original message must be in a folder somewhere in the store. It might be in a temp folder, or it might be in a user's Outbox.
• The copy must be in a folder somewhere in the store. Since transport is creating the copy, it always picks its temp table.

A typical content conversion case involving IMAIL is the outbound Internet Message Formats MAPI > MIME conversion. In that case, a message that's in the store needs to be converted into some MIME format because the receiving end may not be MAPI-capable. In those cases, transport applies that policy. Transport makes a separate copy of the message for each different target format that needs to be produced. Those copies need to be made in the store since IMAIL will be rendering the content, and the chosen location for the copies is the temp table. (Transport doesn't want someone to be confused by seeing their outbox fill up with multiple copies of the message they just sent!)  The content conversion is, of course, done by IMail but Transport use temp tables as temporary storage location for persisting messages while they go thru content conversion.

NTFS -> EXIFS

• The original message is an EML file in the queue directory.
• The copy is created by transport, so we put it into our temp table.

There are a number of bifurcation scenarios where we're not trying to leverage special features of a specific mailmsg store driver; we just need another message in these cases. In these scenarios we use the same backing store as whatever the original is already stored on, for example:

• DL report settings
• DL OOF and auto-response suppression settings
• Max one recipient per DSN

2. Force to EXIFS: NTFS > EXIFS and EXIFS > EXIFS

There are cases where we explicitly want to utilize some store behavior so we copy to EXIFS no matter what the source is.  Everything under this heading mentioned below will copy message into temp folder, regardless of the source.  Any bifurcation to "EXIFS" means the copy is in the store. The target location is always one of transport's temp folders in that case.

• In Exchange 2000, all DSNs generated by transport.
• In Exchange 2003, DSNs generated by transport only in cases where the message being DSN-ed was originally submitted through the store driver.
• Journal messages.
• Store-submitted messages that need to be content converted.
• Messages that have been cleaned by transport AV scanning and need to be converted to a MIME format.
• Messages that need to be AV scanned in transport.
• Messages that need transport P2 address rewrite.

3. EXIFS > NTFS is unsupported and never occurs (by design)

For IMail conversion scenarios please see: XFOR: IMAIL Conversion Scenarios in Exchange 2000 (259890)
http://support.microsoft.com/kb/259890

- Mohammad Nadeem