Exchange 2003 Server Service Pack 2 (SP2) Anti-Spam Framework
Published Jul 18 2005 02:18 PM 12.2K Views

"Spam is our e-mail customers' No. 1 complaint today, and Microsoft is innovating on many different fronts to eradicate it" - Bill Gates.

With the release of Exchange 2003 Server SP2 Microsoft and the Exchange Server Product Unit are taking another big step in helping alleviate the Unsolicited Commercial Email (spam) problem.  Announced a year ago, the Coordinated Spam Reduction Initiative clearly outlined the plan for drastically reducing the amount of spam customers may experience through the introduction of new technology solutions.  A year later, Exchange 2003 SP2 delivers exactly that - new technology that significantly helps combat spam.

With the introduction of Exchange 2003 Server, Microsoft built a framework that included a number of features intended to reduce the volume of spam.  The framework called for a solution based on a multi-layered approach around the idea that most if not all spam should be stopped at the gateway, and before reaching the final recipient's mailbox.  The adoption of this approach has proven to be very successful and highly effective, and with the release of SP2 the framework will be refined even further to include the following steps of which I'll summarize below:

1. Connection Filtering
2. SMTP Filtering
3. Content Filtering
4. Inbound mail processing rules (this will be covered separately)

This is an example of typical mailflow:

At a point during mail transmission from the origination point to the final destination (recipient), a mail item enters the Exchange organization.  There it goes through the extensive anti-spam processing framework and the following anti-spam layers:

Layer 1 - Connection Filtering

The first step in spam verification process is to establish an understanding where the mail is coming from.  Highly effective, this method accounts for roughly 25% of all blocked mail inside of Microsoft Corporation.  Exchange 2003 Server SP2 provides the following framework to achieve this functionality:

1. Support for multiple Real Time Block List (also known as DNS Block List) providers (including paid subscriptions)
2. Customized rule-based Block List service configuration
3. Custom-tailored server response (DSN) based on provider and connection initiation source (i.e. open relay, known source of spam)
4. Global Accept and Deny Lists
5. Configurable exception list that override the block list

If enabled, Connection Filtering will snap the IP address on the incoming connection from the winsock and send a DNS query to verify whether the connecting identity has been listed as an open relay or if it is a known source of spam.  If the returned DNS query contains the connecting IP address on the DNS block list, Connection Filtering will close the connection down and trigger a customized server response back to the sender.  If the sender was legitimate and got onto the block list by mistake, s/he can implement corrective actions based on the custom NDR received.  Connection Filtering also allows the remote user from the blocked IP to submit mail to the postmaster or administrator of the Exchange organization for implementing corrective actions if the sending identity was put on the block list in error.  Connection Filtering is very useful as it is capable of defending Exchange deployments from spam without generating additional resource consumption (CPU cycles, further load on the downstream servers, extra mail processing and NDR handling, etc.).  Exchange 2003 Server can also perform reverse DNS lookup on incoming messages to resolve originating IP to a host name through the DNS.

The majority of Exchange 2003 Servers have been deployed behind the perimeter and do not face the Internet directly.  This renders Connection Filtering less useful as the feature relies on getting the original sender's IP to run the DNS query on.

With the release of SP2 this deficiency has been addressed by introducing a new headers parsing algorithm for originating IP retrieval.  Now, Exchange 2003 Server SP2 with Connection Filter deployed can be positioned anywhere in the organization and perform filtering as it would on the perimeter. 

Layer 2 - SMTP Filtering

If the incoming connection passed through the Connection Filtering layer, the next in line is SMTP Filtering.  Exchange 2003 Server SP2 makes extensive use of SMTP protocol filtering and this feature contributes to 35% of all blocked messages in Microsoft Corporation (MSIT department).  SMTP Filtering has been implemented to monitor live SMTP sessions and functions as a real-time filter.

SMTP Filtering has been complemented by comprehensive SMTP session RFC compliance enforcement and starts with the first RFC2821 HELO/EHLO command.  The SMTP implementation in Exchange 2003 Server SP2 will monitor and examine the session for potential violations and will not accept mail when the sending identity does not conform to or severely violates SMTP governing RFCs.  I.e. no mail transaction will be allowed if the remote party does not begin the SMTP session with an appropriately constructed greeting that contains an RFC2821 compliant domain part.  Similarly, RFC compliance will be enforced on MAIL FROM: and RCPT TO: commands to ensure that malicious users can not get around SMTP Filtering by providing, for example, 8-bit characters in the RFC2821 stream.  Another common tactic deployed by spam senders is to confuse anti-spam solutions by changing the order of commands in an SMTP session.  Exchange 2003 SP2 enforces RFC compliance in this area, preventing spam senders from executing an attack this way.

To prevent dictionary attacks and valid e-mail address harvesting, Exchange 2003 Server SP2 will respond to the VRFY command, but will not release the directory information to the remote party.  Exchange 2003 Server SP2 by default does not support the EXPN command.

SMTP Filtering is based on rules configured by the administrator and consists of Sender Filtering and Recipient Filtering.  Sender Filtering starts with the first RFC2821 MAIL FROM: command.

Sender Filtering:

1. Sender Filtering allows a list of senders to be specified that are prohibited from sending messages to a particular Exchange organization. 
2. Sender Filtering can be easily configured to reject messages originating from certain domains or email addresses. 
3. Sender Filtering filters messages with blank sender information and provides a mechanism for spoof detection (e.g. if the message coming from outside the organization claims to be sent from the CEO of organization where Exchange is deployed). 
4. Based on the administrator-configurable actions the filter can drop the incoming connection if the Sender's address matches the filter.
5. To minimize information disclosure to malicious users, Sender Filtering can silently accept mail and delete it without notifying the sender.
6. Sender Filtering provides an option for archiving filtered messages for a forensic analysis as needed. 

Once a mail item gets through Sender Filtering it faces  Recipient Filtering.  Recipient Filtering starts with the first RFC2821 RCPT TO: command.

Recipient Filtering:

1. Recipient Filtering enables inbound mail filtering for a particular recipient in the Exchange organization. 
2. Recipient Filtering supports blocking mail based on wildcards.  This enables administrators to use patterns to block entire ranges of recipients.
3. Recipient Filtering filters messages sent to non-existent recipients, rejecting them at the protocol level.  By rejecting non- existent recipients at the protocol level (on RFC2821 RCPT TO: command), the Exchange server is protected from doing expensive NDR generation work and clogging the Badmail directory.

4. Enabling filtering of the Recipients who are not in the Directory potentially allows spam senders to discover internal directory information (valid e-mail addresses in the Exchange organization).  A malicious user can execute address book mining by monitoring/parsing the server responses to RCPT TO: commands.  To mitigate this threat Exchange 2003 Server SP2 supports SMTP command tarpitting.  An administrator can configure Exchange to implement an n-seconds delay of the server response to the RCPT TO: command if a DHA attack is encountered or if the remote party violates SMTP RFC conformance.  When a malicious user tries to harvest responses the Exchange server significantly slows down its responses (to an admin-defined delay interval) and the attack becomes infeasible. 
5. Ability to restrict Distribution List mail submissions to authenticated users only contributes to the Recipient Filtering Framework.
6. Recipient Filtering applies only to anonymous connections so all authenticated identities bypass Recipient Filtering rules. 

Layer 3 - Content Filtering

If a mail item gets through Recipient Filtering it faces Content Filtering.  Content Filtering in Exchange relies on Microsoft Research SmartScreen machine learning technology incorporated into the Intelligent Message Filter (IMF).

Intelligent Message Filter:

Messages from the Internet arrive at the Exchange SMTP gateway and enter the Exchange 2003 Server anti-spam framework. Previous layers of the Exchange 2003 SP2 anti-spam solution (connection, sender and recipient filtering) block message submission before message data is sent.  If a message passes all of these then the message body is received. A custom event sink (msgfilter.dll) is invoked when the SMTP End of Data event occurs. The sink passes the message to the Intelligent Message Filter SmartScreen DLL. The SmartScreen technology determines the Spam Confidence Level (SCL) rating of the message which is returned to the sink for comparison against the gateway SCL threshold. The Administrator defined action is applied if the message SCL is greater than the gateway threshold.  Otherwise, the SCL rating is added to messages for transmission to the recipient's inbox.

As of today, the SmartScreen technology deployed by the Microsoft IT department blocks 40% of remaining messages that passed through the previous anti-spam layers (Connection and SMTP filtering).

Anti-Phishing:

With the introduction of Exchange 2003 Server SP2 Intelligent Message Filter extends the anti-spam functionality to support anti-phishing.  The new SmartScreen anti-phishing technology will impact the SCL ratings and also expose an independent Phishing Confidence Level (PCL) value as an output of the filter.  The new anti-phishing functionality is totally transparent to administors and includes the following features:

1. Anti-phishing heuristics
2. Anti-phishing consistency list
3. Anti-phishing block list
4. Anti-phishing allow list
5. PCL values map to weights within the DAT file that allow for non-deterministic SCL adjustment  

Anti-phishing technology relies on heuristics, the consistency list, and the allow list to detect phishing scams.  These lists are not exposed for administration and are updated during regular IMF updates (frequency of updates will be determined later).  The PCL score is one of the factors that trigger final SCL assignments. 

Custom Message Weight:

Custom Weighting (also known as Bad Words List) is a file-based implementation with no supporting UI.  Within the file specific words/phrases can be added along with their relative text part location (Subject or Body) and their associated modifier value.  The supported modifier values can include positive and negative increments, as well as MAX and MIN values.  During anti-spam mail processing the IMF will look into the file and search inside the mail item for a string/word/phrase match.  If a match is found the weight of the SCL will be adjusted according to the modifier value.  If MAX or MIN values are specified for particular word or phrase, they will move the SCL towards MAX or MIN.  The Custom Message Weight feature allows administrators to custom tailor the filter to account for specific words or phrases and adjust it on the fly to act on particular message content as needed.

Custom Server Response when IMF is configured in 'Reject' mode:

To mitigate False Positives, a problem common to all types of automated anti-spam technologies, IMF can be configured to work in "Reject" mode.  This allows for False Positives investigations if the mail was submitted by a legitimate sender and rejected by the filter.  SP2 allows customization of the server response string that is generated and appended to the NDR sent back to the sender.  The default response "550 5.7.1 Requested action not taken: message refused" can be altered to provide blocked legitimate users with meaningful explanation as to why their message was blocked.  This will alleviate investigation of False Positives and decrease support costs associated with the anti-spam processing.

Sender ID Filtering:

Sender ID is an industry standard framework created to counter e-mail domain spoofing.  Sender ID is aimed at removing the ambiguity associated with the sender identity by verifying that each e-mail message originates from the Internet domain from which it claims to come based on the sending server's IP address. Eliminating domain spoofing will help legitimate senders protect their domain names and reputations, and help recipients more effectively identify and filter junk e-mail and phishing scams.

The steps in the process are:

1. The Sender sends an e-mail message to the Receiver.
2. The Receiver's inbound mail server receives the mail.
3. The Receiver's server checks for the SPF record of the sending domain published in the Domain Name System (DNS) record.
4. The Receiver's e-mail server determines if the sending e-mail server's IP address matches the IP address that is published in the DNS record.

Sender ID defines an algorithm for detecting the email address of the entity that is most recently responsible for injecting a message into the email system by extracting the Purported Responsible Address (PRA). The extraction of the PRA ensures Sender ID verifies the appropriate sender against the correct IP addresses as email systems can legitimately forward mail on behalf of other mail servers.

The Sender ID feature has 3 modes:

1. Delete (silent delete - no NDR generated)
2. Reject (the mail will be rejected at the protocol level)
3. Accept (the mail item will be stamped with the Sender ID result for IMF consumption). 

The first and second mode will delete or reject mail that failed the Sender ID verification (i.e. a clear case of spoofing), the rest of the mail items will be stamped with the Sender ID status and passed along.  The last option will just stamp the Sender ID status onto the mail item (even in the case of spoofing).  This status will be passed to the new Intelligent Message Filter and trigger appropriate Spam Confidence Level (SCL) score modification.

- Alexander Nikolayev

26 Comments
Not applicable
What is the "out of the box" mode setting for SenderID filtering?
Not applicable
The default setting is "Accept".
Not applicable
Will messages that IMF deem as spam be rejected at the SMTP level; or will it accept the mail and then send a new bounce message to the "sender"? (Or is it an admin configurable option? - I hope that reject is used over accept-than-bounce.)
Not applicable
Shannon, there are no changes in the way IMF functions and the mail needs to be accepted first. Yes, this is "accept-then-bounce", however, with IMF running in REJECT mode you can implement CustomRejectResponse functionality that will help with False Positives investigations.
Not applicable
A detailed post on the new Anti-SPAM framwork being shipped with Exchange 2003 SP2 from the Exchange...
Not applicable
Will IMF be support in a cluster enviroment?
Not applicable
Phishing Confidence Level - Clearly we will need an update of the Exchange SDK with more details on this. Any news on when it will be available?

Also any hints on when SP2 will be available as beta?
Not applicable
Fabio, IMF will not be supported on clusters in E2K3 SP2.
Alexander, SP2 should be available as beta in the nearest future. We are starting the pilot program for SP2 IMF updates with TAP participants in August.
Not applicable
What is the reason that IMF is not available for clusters? There are a lot of single cluster installations that would benefit from IMF.
Not applicable
<p>
Get the details on how Sender ID/SPF filtering work in Exchange 2003 Service Pack 2.
</p>
Not applicable
Can you all please add the action of "re-direct" as an IMF action? We have a Brightmail quarantine database that we want to leverage, and it would be awesome to re-direct IMF positives to the same location.
The reason I ask is because our help desk has direct access to the quarantine, and they don't bug the engineers. With the IMF "Archive" function, we either have to give them direct action to all of our SMTP connector servers to search for "missing email", or take on that responsibility ourselves.
The technology is great, we just need it to be a little more process friendly before we can implement it. If you all do add this feature, we will use IMF in conjunction to Brightmail to provide layers in depth - which is always a security best practice.

Thanks!!!!
Not applicable
In response to Holger's question:
The best practice is not to run the Intelligent Message Filter in Exchange server cluster environment. However, it can run on front-end servers that are in a NLB cluster. The reason we do not support IMF in Exchange cluster is that we recommend IMF to be deployed as close to the perimeter as possible, and clusters usually get positioned far behind the edge and generally do not serve for anti spam processing.
Not applicable
Great article! At which point in the process is Sender ID filtering done?
Not applicable
In response to Kevin Laahs:
Kevin, Sender ID filtering happens right after EndOfData (EOD) and before the content filtering (IMF).
In response to Dan Sheehan:
Dan, I need a little more time to answer your request.
Not applicable
Good stuff, thanks!
How to customize "server response string" for IMF?
Not applicable
"Server Response String":
You need to set CustomRejectResponse string regkey under HKLMSoftwareMicrosoftExchangeContentFilter. If the value is present it will be used in 550 5.7.1 instead of default.

Not applicable
I was wondering what the testing on effects of installing SP2 with a server that already has IMF installed.

Any tricks or gotchas? Do you have to remove the standalone IMF install before installing SP2?
Not applicable
I was wondering what the testing on effects of installing SP2 with a server that already has IMF installed.

Any tricks or gotchas? Do you have to remove the standalone IMF install before installing SP2?
Not applicable
Nexus,

SP2 installation will prompt an error if IMF v1 is installed on the server and IMF v1 has to be removed before SP2 is installed.
Not applicable
Thanks for the answer, Ill spread the news. looking forward for SP2's release.

Sorry about the double post, the blog didnt seem to refresh.
Not applicable
By this time, you all must be already aware of new anti-spam features provided in Exchange Server 2003...
Not applicable
I found this very interesting. I was wondering about what your team thinks about two possible features that I did not see covered, which IMHO would help to further minimize false negatives, false positives, and administration load:

- Custom Message Weight: will it also support a simple list of "Bad Attachment Types"? E.g. any message with .exe, .scr, etc. could be deleted, archived or otherwise processed within the Exchange Server environment.

- Turing test mailback and response processing: I think this is vital, I mean, to give real and honest users a chance, rather than making them collateral victims of the sum of all the technology we increasingly use against spam. And for doing all of this without requiring the admin to take action.

Also, what about integration with client-side whitelists? E.g., can the end user force-enable some senders in Outlook 2003 (Safe Senders, Safe Recipients), and make sure these get through even if Exchange Server might otherwise mark them as UCE? Again, the idea is to empower the user and make the system more responsive to urgent needs, and at the same time reduce the load on the admin.

Thanks for listening.
Not applicable
So when SP2 applied in a cluster IMF wont be installed by default, right ?
Not applicable
Great article - very good stuff!
I did set the CustomRejectResponse key on the SP2 Frontend Server as described, restartet machine - but still see the default response? Anything else to be enabled here?
Not applicable
Harald, in order to take an advantage of this feature, IMF should be configured in 'Reject' mode.
Not applicable
To answer a question from Nawar - IMF will be installed but not enabled by default. IMF is not supported on Exchange clusters, however, you can run IMF on NLB clusters.
Version history
Last update:
‎Jul 01 2019 03:07 PM
Updated by: