What's the difference between Archive Sink and Journaling?

[The_Exchange_Team]

I have been asked this question several times.
 
The short answer is that Archive Sink allows capturing emails that go through a specific SMTP virtual server.  Journaling will capture every email that is sent and/or received by users that are hosted in a specific MDB.

Let me clarify with a few scenarios:

A - I want to capture every email that is sent to the internet, but I am not interested in internal emails
B - I want to capture every email sent and received by the Executives in my company
C - I want to capture the emails between two of the companies my company owns

In scenario A we would use Archive Sink.  In fact, we are really interested in the traffic that goes to the Internet, which means I can simply install the Archive Sink on the Exchange bridgehead server(s) that are relaying email to the Internet (or to Internet Gateways). 

In B I will use Journaling, as I am really interested on the emails that some users are sending or receiving.  In this case Journaling is the easiest feature to use.  There's a catch.  If the executives are in different MDBs then you will need to turn on journaling for every MDB that is hosting an executive.

In the last scenario - C - we can go back to using Archive Sink.  In fact, in this scenario I will likely have an internal SMTP Relay server between the two companies and I can simply install the Archive Sink on this server.

Aside from the usage pattern, I want also make sure that we understand another big difference between Archive Sink and Journaling: message format.

Journaling will require a mailbox (or PF) as "archive repository". Archive Sink will drop a copy of the message (and an extra xml file) to a local NTFS folder on the box.  This is a pretty big difference and again it highlights the fundamental difference in usage pattern between these two features.

Just for fun...an historical perspective: Archive Sink was not originally "designed" as an archiving tool, but rather as a troubleshooting tool.  In fact, the problem we were trying to solve was the "I received corrupted messages" case.  Let me elaborate just a bit more on this.  There are cases in which users receive an email with strange characters (=garbage), if the email was received from the internet the first thing you want to know is whether the email was corrupted before it got to your Internet Gateway or if it was corrupted by Exchange during the propagation from the Internet Gateways to the mailbox server.  Well...Archive Sink was designed to help answer that specific question.  The idea was to capture the message as soon as it is received by the gateway so that the administrator could check whether it was corrupted before we received it.  As customers started using the tool...they realized that they wanted to use it for more than just troubleshooting.

Below I included some good documentation for your reference.  Now, before I get questions about "are you guys improving journaling?"...the short answer is "Yes, we are committed to make the archiving experience much better as we move forward".  I can also say that we received very good feedback on what we did with our Envelope Journaling in Exchange Server 2003 SP1.  The build in feature set augmented with our archiving partners' solutions should help our customers with Compliance requirements.

References:
Journaling in Exchange Server 2003 (SP1):
http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/journaling.mspx

Archive Sink link:
http://www.microsoft.com/downloads/details.aspx?FamilyId=5D3475BD-6915-4110-959D-6E4CB233D79D&displa...

Tool to enable Envelope Journaling:
http://www.microsoft.com/downloads/details.aspx?FamilyId=E7F73F10-7933-40F3-B07E-EBF38DF3400D&displa...

Exchange archiving partners:
http://www.microsoft.com/exchange/partners/archivingandcompliance.asp

- Max Ciccotosto

[#2]

Compliance requirements - The issue is much larger then just Sarbanes-Oxley and HIPPA.

The need of companies to be able to produce copies of e-mail when discovery happens in civil ligitgation is going to be more common place faster then most people think.

Several changes in the procedures followed in various jurisdictions have all ready happened and this trend will acclerate.

It is there responsiblity of the producing party (the one served with the discovery
motion) to supply the e-mails. This effects ANY business that has an e-mail server.

There is a need for the ablity to be able to index, search and produce all the e-mails sent with in an organization based on the terms of the discovery order issued by the judge.

This is NOT just a big company issue, it is all company with e-mail.

Hopefully the Exchange Team is having some discussion with the computer forensic's / electronic discovery community about these issues.

drh at hibbeln dot net

[#3]

Great article Max, thank you. As David points out there are larger compliance considerations and we thank you for also pointing out your archiving partners who can assist in this space. Sidbar; there is an email forensics team internal to Microsoft somewhere and this group's opinion would be interesting.

Onto a question I have - ArchiveSink is particular to Exchange Servers in that it relies on Event Sink stuff from Exchange. Is there a way (and I haven't done any R&D, this just popped up at the right time) where an IIS SMTP Server can be made to drop a copy of all messages passing through to a disk queue for "archiving"? EML format would be suitable. Or perhaps another way of doing essentially what ArchiveSink does, but without Exchange involved? I'm thinking of SMTP gateway boxes where IIS alone is being used.

Cheers

[#4]

Errr - okay, now I feel like a fool for posting that question, the first bit of google-fishing (bring on the new MSN Search!!) I did brought up an older blog (http://blogs.msdn.com/exchange/archive/2004/02/19/76432.aspx) from the Exchange team - which answers my question in a basic fashion at this point - but it does rely on the SMTP box being the endpoint for that email traffic. I'm also interested in mail passing through, rather than being at the end of it's travel... hence ArchiveSink-like functionality would be cool...I'm thinking the sink sample provided in the blog post I just referenced could be modded to do similar things?

Cheers

[#5]

This is good and very close, but I find outbound messages are not logged? have I missed something..
I find the messages 'pre SMTP format' in Mapi-Gateway Messages, but this does not show what the server actually sent to the destination.

Personal view, but I find Exchange 5.5's protocol logging and archival better and far easier to trace and diagnose what came in/went out. Yes I know there could be inprovements, like logging the IP address and noting the message archive file in the protocol log and perhaps even having a copy of the protocol session data at the top and bottom of each message archive file.

The IIS protocol log is a mess (again personal view).. can we go back to one logfile per session. and can we have the ability to FULLY log everything INCLUDING the message in the protocol log, so things are consistent and very easy to diagnose.
Oh and while I'm asking can we see exposure of the DNS MX lookup, that was performed to find the destination server, in the protocol log.

The sinks are a very good idea and provide the ability to do lots of good stuff (like global footers etc) but for diagnostics and LEGAL purposes we do need to be able to see an exact log of what the server actually sent and recieved. with sinks you can't be certain that another sink existed after/before the archive one that modified the message..
My thoughts as a techie is that an archive journal would not stand up in court as I would not be able to prove/disprove the existance of any other sinks.
In the UK we don't worry so much about Sarbanes-Oxley and HIPPA. What we do wory about is e-mail is a legal comunication method and a copy of the message send must be just that a copy of what was sent/recieved over the wire. I've put exchange 2003 in for some firms of solicitors here and we still have to feed everything through an Exchange 5.5 server as the logging and journal is accepted PROOF, but I'm told that everyones still arguing about 2000+ [This could be totally wrong by now though].