Named Properties, Round 2: What lies Ahead

[The_Exchange_Team]

In my last blog post on named properties, I detailed the history of named properties, how they are arranged, mapped, and how Exchange (since Exchange 2000) has created named property mappings for x-headers on inbound messages. Rollup 8 for Service Pack 1 contains the changes we made to restrict named property mapping further, and with Service Pack 2 on the horizon there are more changes in the waters. This post details what is changed in Service Pack 2, how that affects (or does not) you, and includes a request for feedback.

As mentioned before, Exchange 2007 creates named properties for x-headers during delivery. The RTM version of Exchange 2007 would NDR messages if the named property quota was reached, even if the property were only for an x-header. Service Pack 1 changed this behavior to only NDR the message if a non X-header named property could not be mapped (in other words, a property likely to contain critical data). Rollup 8 for Service pack 1 removes the ability of non-authenticated messages to consume named property IDs. Service Pack 2 goes even further.

Service Pack 2 is in line with Exchange2010 behavior - that is, no x-headers are ever promoted to individual properties if a client has not already requested (and mapped) them. Not even authenticated submissions can create new named property mappings. The x-headers are still stored in PR_TRANSPORT_HEADERS and still accessible to MAPI clients but they are not individual properties. This creates a problem for IMAP clients in particular:

Exchange2007 does not contain an STM file - it doesn't save the original mime content. Messages are converted to MAPI. IMAP has the ability to search based on header values, but if these are not promoted, the result is that the message is not contained in the results. If a client has previously mapped t he header in question (a MAPI client) then messages delivered after this will have the x-header promoted as an individual property.

So, the questions I ask are:

1. Do you own or run an application which requires a custom x-header to be promoted to a property but does NOT create the mapping?
2. Do you own or run an application which uses an IMAP header search for a custom x-header.

Things that aren't on the table:

• Ability to remove named properties.
• Ability to pre-register named properties.

We're aware that both of these would prove useful. Also, this is not a voting contest - we've seen how a vocal minority can skew results before. Just a heads up and if you have real world reasons for arguing for a particular request, let us know.

- Jason Nelson

[#2]

We own a Barracuda Spam Firewall which adds x-header information to all mail from the internet.  Some examples are:

X-Barracuda-Spam-Score
X-Barracuda-Start-Time
X-Barracuda-Connect

These are the only x-header information that is useful to us.  We would like to be able to remove named properties that got added from internet mail before we started stripping them out with the HeaderFliter agent by Codeplex so we don't have to increase the quota on our databases for named properties.

Thanks for the update on this post, but it still doesn't give us the answer that if we are at the quota limit right now before applying Rollup 8 for SP1, after we install the update, what do we do with the quota limit?  Do we still need to increase it?  

In our case, we dont' have any applications that need to promote custom named properties with the exception of people installing stupid mailplugins every now and then.  

[#3]

Do you count IMAP MUAs as an "application"?

I recieve between several hundred and several thousand emails a day from automated process using SMTP. My IMAP MUA (alpine) filters them into folders based on X-headers. Later, I view the contents of those folders and run additional searches based on those same x-headers to further trim what I want to deal with. I know that others at my organization have similar workflows, and it would be extraordinarily inconvenient (at least) should that suddenly stop working.

[#4]

JSC - Yes, if you are at the limit you should increase the quota, I would recommend increasing it by 200 and then monitoring growth closely.  Nothing recovers named properties.  Nothing.  So if you are at the quota, you can either stay there (and NDR messages when they contain NON internet properties) or increase the quota.  Since anonymous submissions are now barred from consuming x-headers the growth rate will be slowed.

Jason Madden -
Yes - I consider an IMAP MUA an application.  Whether or not your MUA will continue to work depends entirely on whether a MAPI client has requested those X-headers (which will cause them to continue to be promoted.  Since this is database wide in Exchange 2007 that means that if any other user on the database has asked via a MAPI client for an x-header or if an authenticated submission contains an x-header it will be available.  

[#5]

We also have an anti-spam device, Ironport, that stamps the header with with a spam score. However we don't search for it later. Any issue with a particular email and we open the header to check the score.

I assume the header will still contain the info just not be be searchable?

[#6]

Havent seen any app. that rely on the propagation of x-headers but why dont make an on/off configuration switch for the propagation

[#7]

Hello,
I work for an antivirus company that relies on these x-headers to pass certain information to the user (like the spam score of an email, what rule was matched, what AV engine version was used to scan the mail and so on). Preliminary tests with update rollout 8 showed that our headers are no longer delivered in the user email. How can we get around this?
Thank you.

[#8]

How does this work for tnef encoded messages?  Consider the scenario:

User A has mapped a xheader ("x-test") via a mapi client.

User B (on a differenent store db) has not mapped the "x-test" header.

User A sends a tnef encoded message with the header encoded as a named prop in the tnef body of the message to user B :
 Tag: 0x808E001E
 Type: PT_STRING8
 Named Prop Name: sz: "x-test"
 Named Prop Guid: {00020386-0000-0000-C000-000000000046} = PS_INTERNET_HEADERS

Will the prop get mapped to user B's database and be available via MAPI?  Would sending the property using  PS_PUBLIC_STRINGS as the Named Prop Guid give different results?

[#9]

Sorry for the typo, my first question at the bottom should be:

Will the "x-test" prop be available to an IMAP client as an x-header?

Also:

Will the x-test prop be available as a named prop to MAPI clients?  Or in the PR_TRANSPORT_HEADERS prop?

[#10]

Radu - If (as you indicate) the client is actually asking for those x-headers by name then in fact they will be promoted to individual named properties.  If the submission is authenticated they will also be promoted to individual x-headers.  On E2k7 if one user on an MDB has mapped an x-header to a named property all other users on that mdb will see it mapped to an individual property.

If no users have mapped an x-header to a named property id and the submission is not authenticated and the application has not requested this id, it will NOT be promoted to an individual property.  It is available in PR_TRANSPORT_HEADERS.  You can get around this by having your application send one user on the MDB an authenticated message with the x-header.  OR you could have one user on the mdb actually retrieve the property.

In Exchange2010 that will not work - mappings are per user and each user would need the property mapped to see it promoted as an individual property.  

James - TNEF should be properly restricted as well.  I believe RU8 does not apply the constriction to TNEF in all cases, something we've fixed.  PS_PUBLIC_STRINGS is not limited...yet.  It's on our radar (and has been for a few months) as something to keep an eye on and consider a similar constraint.

[#11]

Thanks for the response.  

This is probably worse than you've considered for any app that is implemented according to the Exchange 2007 Foreign Connector guidelines and is meant to allow for IMAP clients.  The important thing to keep in mind is that this will break in ways that are very hard to trace and explain to customers, and generate lots of support calls.

Is there any way you would consider leaving this the way it is in RU8, where a FC that creates a tnef encoded message is able to squeeze in new x-headers?  Or possibly Exchange could map x-headers to ids for messages that are sent through the Replay directory only?

Since there is no longer a MAPI gateway architecture, it makes sense to allow a little more flexibility here for those of us porting older gateways to the new 2007 FC architecture.

You mentioned in one post that x-headers/id mappings are changing from per mailbox store to per mailbox.  Does this mean that in Ex 2010 we'll need to find a way to log onto each mailbox that may receive a message from a foreign connector and create the mapping wheras logging into one mailbox per store is good enough for 2007?

[#12]

Another thing to consider is that even for existing well written MAPI clients, it seems like it is generally the sender who will be calling GetIdsFromNames() when the message is generated.  The sender may then be sending to a recipeint on a different store, and the property will be dropped.  For Exchange 2010, if I understand correctly, the map is moving to a per mailbox model, and the every user will have to essentially send one message that may or may not properly display x-headers on IMAP clients of the recipient, so that when they themselves eventually receive a message with x-headers it will display.

This is going to look like very flaky and erratic behavior to any end user.

I think all of this argues for my scenario in my first post above to continue to work as it does in RU8, or at least allow admins the option of allowing it to continue to work this way.  It achieves the goal of culling from the named property mapping table most of the x-headers that just sort of float into the org from the internet, while allowing MAPI clients and foreign gateways to still function.

I'll stop spamming now :)

[#13]

We use Spamassassin which adds x-headers to the message which is used by email clients to sort messages both via filters on the Exchange servers but also from IMAP-clients (10.000+ IMAP users). Because of this we would like to be able to get all headers via IMAP. The IMAP users typically don't have access to a
MAPI client and can't promote an x-header.

We also add a custom x-header for some types of logins and might add an
x-header with information about delays resulting from greylistning of
the message. This information is useful for debuging delays etc.

It's also good to have the full header including all x-headers when
analyzing spam.

[#14]

If you want an x-header promoted the easiest way to get it promoted is to send an authenticated mail to a single user on the MDB.  For E2010 that will not work, but E2010 has different MIME handling that should make IMAP interop better (even for flowed bodies & such).

[#15]

Just had my first problem when debugging an external email. Reading the mail via IMAP I couldn't see all headers and since I didn't know which x-headers would be interesting I couldn't send a mail to promote it. I just needed to see the original unmodified message as it was when it passed through our anti-spam filter. This will be a big problem for me in the future when debugging emails.

Another question I'm waiting to get on my table regarding this change is how we can let users moving to other providers get a full copy of their messages with all header? Before we could use different "imapcopy" tools but that will not be possible in the future. What would you recommend in that situation? pst-files is not an option.

[#16]

I would suggest MAPI or EWS, both of which have access to the transport headers.

[#17]

I have clients who have antivirus and email encryption appliances that apply custom x-headers for incoming/outgoing emails.  They use E2K7 hub transport rules that run against these x-headers (i.e., journal all inbound emails with specific x-headers applied by Ironport).

How will these organizations be affected by these changes?

[#18]

Hub transport rules run before an item is converted to MAPI, and thus are not affected.

[#19]

Then might a solution be to use a hub transport rule to promote the x-header to a property?

[#20]

NICE

[#21]

We are using an IMAP connection that relies on the existence of certain x-headers.  Using IMAP only, is there a way to promote x-headers to individual named properties?