How the M: Drive came about
In Exchange 2000, we introduced a new feature called IFS. IFS stands for “Installable File System”. This uses a little known and even less used feature of NT that allows the OS’s file system (like NTFS or FAT) to be replaced. The initial reason for doing that was as an optimization: it would allow protocols, such as NNTP and SMTP, to transfer the MIME messages directly as files. In Exchange 5.5, MIME messages are broken down into MAPI properties and stored in database tables. When they need to be accessed as MIME, they are put back together. In E2K, MIME messages are stored as MIME files in IFS and only converted into MAPI if a MAPI client (such as Outlook) accesses them. The other perceived benefit of IFS was that the Exchange storage objects could then be made visible through the file system. So you could go to a drive letter (M: was chosen for two reasons: first, “M” for “Mail”, and second, because it was in the middle of the alphabet and least likely to collide either with actual storage drives -which start at A and move up - or mapped network drives - which start at Z and move down), and get a list of mailboxes, navigate to mail folders via cmd or windows explorer and look at actual messages. This was considered pretty neat at the time and since it didn’t seem to be much more work to allow that access, it was thrown in (there may have been other, better reasons but I’m not aware of them). This ended up causing some challenges down the line related to the intricacies in how email objects need to be handled and mapping the file access behavior to them One of the biggest problems encountered was around security descriptors. This is difficult to explain without a detailed understanding of NT security descriptors, so I will simplify the explanation for the purpose of this discussion. The main part of an NTSD is called a DACL (discretionary access control list). It contains a list of users and groups and what they can do to that object. There are two main types of entries: allows, which say what an entity can do; and denies, which say what they can’t. The order of this list is very important. A standard sequence of entry types is called “canonical”. NT canonical form calls for a particular sequence. Because of legacy issues, MAPI canonical form requires a difference sequence of entry types. Applications that modify security expect a particular sequence and will behave erratically if the sequence is wrong. By creating or modifying objects through the M: drive, it will change the canonical format of the DACL’s and result in unexpected security behavior. This is bad. A related issue here has to do with item level security. E2K also introduced this feature, which is that items in a folder can be secured independently of each other and the folder. While this has some great uses, for many email systems this level of security is not needed. When a message has the folder default security, it simply references that property in the folder. When a message has its own security, there is an additional property that needs to be stored (this also has an affect on how folder aggregate properties, such as unread count, are computed). Having lots of individual security descriptors can result in both increased storage size and poor performance. When a message is created or modified through the M: drive, it will always get an individual security descriptor stamped on it, even if it is exactly the same as the folder default. This can also lead to unexpected behavior. For instance, if you change the default security on the folder, it will not change the security on any messages in it that have their own security descriptors. They have to be resecured individually. Another challenge is in relation to virus scanners. Virus scanners typically look for valid storage drives and spin through all the files on those drives and check them against virus signatures. The M: drive appears as a normal drive, so virus scanners were picking this up and processing it. This can have very detrimental affects on system performance and may also result in message corruption in some cases. Finally, IFS runs in kernel mode. This is a privileged execution path and it means that problems in this area can have much more severe affects (and be harder to track down) than in other areas of Exchange, which all run in user mode. Blue screens are one possibility if something goes wrong. IFS has given Exchange 2000 and Exchange 2003 a lot of advantages: we maintain content parity for MIME and make MIME message handling faster and more efficient as well as increasing the performance of such messages retrieved via internet protocols. But as I described above, there can be problems if IFS is misused via the M: drive. In Exchange 2003 we have disabled the M: drive by default to hopefully help reduce the likelihood that customers will encounter any of the issues described above. I encourage every system administrator to keep this disabled on E2K3 and disable it on all E2K servers as well. Jon AvnerWhy is OOF an OOF and not an OOO?
Here's an interesting historical question - when we say Out of Office, why does it sometimes get shortened to ‘OOF’? Shouldn’t it be ‘OOO’? Inside Microsoft, ‘OOF’ means not just the message which says you’re Out of Office, but it has grown to mean the act of being Out of the Office too - so you’ll get people putting sticky notes on their door saying ‘OOF Thurs & Fri’ or even people verbally saying things like, "Oh, Kevin’s OOF on vacation for the rest of the week’. I suppose that sounds better than "Oh, Kevin’s OOO on vacation ..." OOF was a command used in the days of Microsoft’s Xenix mail system, which set a user as ‘Out of Facility’ - ie Out of the Office. The usage of the term ‘OOF’ just stuck, as did the term ‘Little r’ (e.g. on an email sent to a distribution list, "Who wants to go to the cinema tonight? Little ‘r’ if you’re interested", meaning reply just to me) - as preserved in Outlook with CTRL+R for Reply, and CTRL+SHIFT+R (aka Big R) for Reply All. Ewan DaltonA brief history of Exchange Team gifts over the years...
Note: There is absolutely no useful content in this posting whatsoever. If you are looking for some little nuggets to help you administrator Exchange, you aren't going to find it here. If, however, you're a bit bored because Exchange is working so well and there's nothing for you to do, read on. Whenever we release a new product or a major new version of a product at Microsoft (or even when we hit a milestone during a release), we celebrate the accomplishment. We have ship parties, and we get ship gifts. We also have a party and get a gift around the holidays. The parties have mellowed a bit over the years but something about the gifts remains timeless. Rather than being like gifts that keep on giving, however, they are more typically gifts that never really gave in the first place. Getting gifts is a lot of fun, and I love getting them, but somehow the search for a single item that is fun or useful for several hundred people tends to yield some highly odd results. In the early years, we tended to get clothing: hats, shirts, even jackets. These are typically the most popular and successful gifts (which is why we've stuck with T-shirts for Exchange 2007), but like all gifts of clothing they're pretty boring, so let's take a look at a few more creative examples. I have tacked up to my cork board a small gift that I think came for one of our beta releases for Exchange 2000 (that was a while ago, so I don't remember exactly when we got it). It's a VR Puppy. Remember these? It's got a small LCD display and buttons on it that you can use to pretend to feed it, walk it, etc. Mine's still in the original packaging, which either means it's worth some money to a collector or it died of starvation several years ago. Next, let's turn to my memorabilia shelf and consider one of the more notorious ship gifts we've: a ceramic popcorn bucket. It came with a package of microwave popcorn. We got this as a holiday gift a few years ago when our morale budget was a bit tight due to other expenditures that year. The really impressive thing about the popcorn bucket is its weight. Perhaps it's lined with lead, possibly to prevent any leftover microwaves from leaking out of the popcorn and into your lap while you're eating from it, although I highly doubt anyone has been so bold (or strong of thigh) as to actually use it for that purpose. Below that we have a very nice, elegant, Exchange 2000 pocket watch. It comes in a very nice, elegant wood case. It's still in its nice, elegant wood case. And last on our tour we'll take a look at the pocket combination tool. This is one of those combo tools that can fold out into a pliers, or individual blades can be extracted like a Swiss Army Knife. This was actually a pretty good idea, and I've even tried to use it once or twice. Mine could be the last one in existence, though, because most of them broke almost immediately upon opening them. Let's just say they weren't particularly well made. And then there's the micro-camera, which we got for Exchange 2003. Another quite good idea that didn't quite pan out due to quality problems. Like my VR puppy, it's also still in its original packaging. Anyway, I hope this posting isn't read by any of the people responsible for purchasing these gifts, because I'd rather get another popcorn bucket than a lump of coal in my next stocking. I hope your Exchange server continues to provide you the free time necessary to read more useless posts like this. - Jon AvnerMe Too!
One way of telling how long a Microsoft employee has been working here is their reaction to the phrase “Bedlam DL3”. Just for grins, I was at lunch in the cafeteria with a bunch of co-workers and I blurted out, totally out of context: “Bedlam DL3”. About 3 of the old-timers in the group responded, in chorus “Me Too!” So why does everyone know about this rather mysterious phrase? Well, Microsoft’s a pretty big organization. We’ve got well over 100,000 mailboxes in our email infrastructure, and at times it can become rather cumbersome to manage all these. One of the developers in our Internal Technologies Group (also known as ITG, basically the MIS department at Microsoft) was working on a new tool to manage communications with the various employees at Microsoft, and as a part of this tool, he created several distribution lists. Each distribution list had about a quarter of the mailboxes in the company on it (so there were about 13,000 mailboxes on each list). For whatever reason, the distribution lists were named “Bedlam DL<n>” (maybe the tool was named Bedlam? I’m not totally sure). Well the name of the lists certainly proved prophetic. It all started one morning when someone looked at the list of DL’s they were on, and discovered that they were on this mysterious distribution list called “Bedlam DL3”. So they did what every person should do in that circumstance (not!). They sent the following email: To: Bedlam DL3 From: <User> Subject: Why am I on this mailing list? Please remove me from it. Remember, there are 25,000 people on this mailing list. So all of a sudden, all 25,000 people received the message. And almost to a person, they used the “reply-all” command and sent: To: Bedlam DL3 From: <User> Subject: RE: Why am I on this mailing list? Please remove me from it. Me too! In addition, there were some really helpful people on the mailing list too: They didn’t respond with just “Me Too!” They responded with: To: Bedlam DL3 From: <User> Subject: RE: Why am I on this mailing list? Please remove me from it. Stop using reply-all – it bogs down the email system. You know what? They were right - the company’s email system did NOT deal with this gracefully. Why? Well, you’ve got to know a bit more about how Exchange works internally. First off, the original mail went to 13,000 users. Assuming that 1,000 of those 13,000 users replied, that means that there are 1,000 replies being sent to those 13,000 users. And it turns out that a number of these people had their email client set to request read receipts and delivery receipts. Each read and delivery receipt causes ANOTHER email to be sent from the recipient back to the sender (all 13,000 recipients). Assuming that 20% of the 1,000 users replying had read receipts or delivery receipts set, that meant that every one of the message that they sent caused another message to be sent for every one of the 13,000 recipients. So how many messages were sent? First there were the basic messages – that’s 13,000,000 messages. Next there were the receipts – 200 users, 13,000 receipts – that’s and additional 2,600,000 messages. So about 15.5 MILLION messages were sent through the system. In about an hour. So at a minimum, 15,600,000 email messages will be delivered into peoples mailboxes. But Exchange can handle 15,600,000 email messages EASILY. There’s another problem that’s somewhat deeper. An Exchange email message actually has TWO recipient lists – there’s the recipient list that the user sees in the To: line on their email message. This is called the P2 recipient list. This is the recipient list that the user typed in. There’s also a SECOND recipient list, called the P1 recipient list that contains the list of ACTUAL recipients of the message. The P1 recipient list is totally hidden from the user, it's used by the MTA to route email messages to the correct destination server. Internally, the P1 list is kept as the original recipient list, plus all of the users on the destination servers. As a result, the P1 list is significantly larger than the P2 list. For the sake of argument, let’s assume that 10% of the recipients on each message (130) are on each server. So each message had 100 recipients in the P1 header, plus the original DL. Assuming 100 bytes per recipient email address, this bloats each email message by 13K. And this assumes that there are 0 bytes in the message – just the headers involve 13K. So those 15,000,000 email messages collectively consumed 195,000,000,000 bytes of bandwidth. Yes, 195 gigabytes of bandwidth bouncing around between the email servers. Compounding this problem was a bug in the MTA that caused the MTA to crash that occurred only when it received a message with more than 8,000 recipients. But it crashed only AFTER processing up to 8,000 recipients. So 8,000 of the 13,000 recipients of the message would get it and 5,000 wouldn’t. When the MTA was restarted, it would immediately start processing the messages in its queue – and since the messages hadn’t been delivered yet, it would retry to deliver the message, sending to the SAME 8,000 recipients and crashing. And because of the way the Exchange store interacts with the MTA, even if we shut down the MTA, the messages would still queue up waiting on delivery to the MTA –shutting down the MTA wouldn’t fix the problem, it would only defer the problem (since the message store would immediately start delivering the queued messages into the MTA the second the MTA came back up). So what did we do to fix it? Well, the first thing that we did was to fix the MTA. And we tried to scrub the MTA’s message queues. This helped a lot, but there were still millions of copies of this message floating around the system. It took about 2 days of constant work before the email system recovered from this one. When it was over, the team firefighting the crisis had t-shirts made with “I survived Bedlam DL3” on the front and “Me Too! (followed by the email addresses of everyone who had replied)” on the back. To prevent anything like this happening in the future, we added a message recipient limit to Exchange – the server now has the ability to enforce a site-wide limit on the number of recipients in a single email message, which neatly prevents this from being a problem in the future. Larry OstermanFrom crush to product documentation: The story of Squeaky Lobster
When customers first hear about being able to enable extra JET Blue or ESE Database performance counters via adding a "Squeaky Lobster" registry value, they often think it must be a joke or ask you to repeat it. And invariable the question comes up ... _why_ "Squeaky Lobster"? Various lore or conjecture has surfaced around it ... that it was a developer's child's toy or that is just thought up after lunch one day. Not exactly ... and retrieving the information was a little tricky as the initial checkin was 2 years before I started ... The story starts with a certain developer, let's call him, "Andrew Goodsell", for the time being, but I call (or called) him, "boss", and I can assure you those explanations are not quite right. If you were to get incriminating information about your boss, you might wonder what is the proper time to blog about it? Right after you stop working for him I think is the right answer to that ... and as of today I have transferred to the Exchange organization, while "Andrew" has not. I am as they say, unsupervised. ;-) Andrew, is a very sharp man who prides himself on his professional decorum, and a bit of a perfectionist to boot. In fact his professionalism is what makes this singular lapse in judgment so amusing and fairly uncharacteristic of any decision he would make (or have let me make ;-) today. Andrew has been involved in a fair share of the database performance work in ESE and as anyone decent at perf knows, one of most critical steps is having a way of accurately measuring performance, base-lining performance, at several levels in a software stack. Exchange 5.5 added ESE Database performance counters to facilitate this work. Most of these perf counters were intended for only internal ESE and Exchange developer usage (as you can probably deduce from one of the code snipits below, and looking at what these counters measure... things like B+ Tree Inserts/sec for example). Enter cute girl ... So that autumn while Andrew was working on adding perf counters and working on Exchange 5.5's performance, a cute girl was managing computers in the Exchange Performance lab. Andrew was working on performance and adding performance counters while this girl worked with the powerful computers in the performance lab. It isn't a wonder how they ran into each other. She was cute, it is no wonder Andrew got a crush. Some of the simplest ESE performance counters would also be helpful for advanced administrators in debugging Exchange server issues (Cache Size, File Operations/sec, etc). Andrew mentioned he thought if competing companies had access to some of the more detailed perf counters, they might be able to reverse engineer our implementation details and steal intellectual capital (now that's the professional Andrew I know, trying to protect our intellectual capital). After he says this, he rolls his eyes and says something to the effect of "I don't know what I was thinking!" (this is because the implementation details are more complex than what the counters show, but at the time this made sense). So there was a need to split the ESE Database perf counters into two kinds. ... internal (dev only) and external (admins). The internal ones would have to be hidden. Now, fairly smitten with this girl, ANYTHING she did was fascinating (to Andrew). Oh come on, we've all been there. And one such thing she did was participate in a Giving Campaign, basically an event where we raise money for charities and non-profits. In order to encourage people to give money, various prizes would be donated and a raffle sort of mechanism to win the prizes would be done. The girl of interest donated a toy Squeaky Lobster to the charity raffle. Andrew thought that was the funniest and most random thing to contribute as a prize for a giving campaign ever. Who would want a Squeaky Lobster? Most random prize in giving campaign I guess equals most uneasy to guess registry value ever. Anyway, a name was chosen that was about as random as you could get for 1997 (as there weren't very many words available in 1997, I think the Seattle was recovering from that awful grunge thing). Knowing the crowd Andrew hung out with back then, it would not surprise me if they decided it at or on the way back from lunch as they used to work 12 hour days and take extra long lunches. An excerpt of Andrew's fateful check in from SLM logs (note, I've deleted the developer actual email aliases), relevant information highlighted: #F eseperf.cxx v1 #K text #O in #P 1.00 #T Fri Sep 26 11:15:00 1997 #A <dev_alias_1_deleted>3 #C 47424, 41913 finalize perf ctrs #I 2 #D 1113 27a28,32 > #pragma const_seg( ".text" ) > const char szDisplayDevOnly[] = "Squeaky Lobster"; #pragma const_seg() As an aside, note the ".text" segment notation, I had forgotten about that, now that's old school. Man, 8 years is a LONG time in the computer industry. And so a little over 4 months later, the first Squeaky Lobster Enabled product shipped on Feb 3, 1998, with ESE97 in Exchange 5.5. In fact this is when the database engine under Exchange was renamed from JET Blue to ESE (to avoid confusion with JET Red, which has only vestigial relations to JET Blue / ESE). ESE97 shipped in 1998, just to confuse everyone. The above picture is of the original Exchange 5.5 packaging with the original Squeaky Lobster. But at this point in our story, the counters are secret so it's not a big deal. But there is no way to keep any sort of more extensive analysis mechanism a secret for long ... eventually someone will need the information. And eventually an Exchange performance case came along sometime in 1998 (according to the earliest record I can find in the PSS DB) that required the extra analysis these performance counters offered. The PSS engineer told a customer how to enable the counters so they could analyze the customer's issue. I mean, it comes down to the customer, you do what is necessary and possible to support them. Then another case came in, and another, and eventually someone from PSS thought to publish a KB on 4/17/2000, KB 259895, what counters are enabled by squeaky lobster, _official_ Microsoft documentation admitting the existence of Squeaky Lobster!!! Well, that didn't last long (about 6 months). In about 2000 someone public (Brian Sheaffer / Paul Thurrott) noticed and of course such a thing was far too silly for the serious professionals that control the web site of most big corporations. ;-) About as silly as Squeaky Lobster for a prize in a charity benefit! I mean we're a professional organization. Now there are about 300 PSS related investigations, service tickets and KBs referencing this phrase (including the above one). There is a Squeaky Lobster in Andrew's office as well, but do not be fooled, it is not the original. That Squeaky Lobster was given to him by a PSS engineer who thought it was funny, or maybe to thank him for being repeatedly asked by customers, "What was that? Can you spell it?". A mere 4 and a half years after the checkin, someone wised up ... and for Windows 2003 RTM, and Exchange 2003 SP1, code was checked in to try "Show Advanced Counters" first, and if that fails try "Squeaky Lobster". The comment in the code is: > // deprecated name (yes, we are ending the insanity) > err = DwPerfUtilRegQueryValueEx(hkeyPerf,(char*)"Squeaky Lobster",&Type,&lpbData); Note: we moved away from that archaic .text seg stuff, the compilers are now smart enough. And finally to come full circle, once this happened the name of the registry value appropriate for product documentation on our web site. Note that page specifies Exchange 2003 SP1. If you replace "Show Advanced Counters" with "Squeaky Lobster", the instructions should work for Exchange 5.5 through Exchange 2003 RTM as well as current products, though the registry key you use for step 2, varies. For Ex 5.5 it is "ESE97" instead of "ESE". In Ex2k you have to use "ESE98" before SP2, and finally for SP2 and later, just "ESE". The same process works for Windows, but use "ESENT" instead of "ESE", oh and you have to enable the performance counters for Windows first. I promised an Exchange MVP, Michael B. Smith, about a year ago and I apologize for the delay, some would even mock me for taking so long with this single post, but at that time, the fact that there was a Ms. Squeaky Lobster unknowingly influencing Microsoft product development was not known. You got to wait until you get all the details, Eric. It wasn't till Andrew, accidentally spilled the information at a dinner. After that I blurted out, "I'm SOOO BLOGGING THAT!", Andrew had an immediately look of regret on his face, and now he won't really talk about it anymore. ;-) So that is about all the Squeaky Lobster trivia I could collect. The next time you have to really dig into Exchange (or Active Directory) database performance, just remember how responsible professional ESE software engineers are adding easily discoverable (based upon the activities of their current crush at the time) mechanisms for diagnosing your top issues. So with that in mind, the next time you go to buy a software product, remember to check the box to see if this is one of the many Squeaky Lobster Enabled products. Oh just checked an Exchange 2003 box, not actually listed on the box, hmmm, I guess serious professionals are in control of the packaging 😉. too. So Sept 26th of this year is Squeaky Lobster Day, the 9 year anniversary of the checkin of the Squeaky Lobster registry value, be sure to Squeaky Lobster a server at 11:15, and then go to an extra long lunch, and make a silly decision you'll regret and feel embarrassed about 9 years down the road ... sorry, Andrew, you can't stop the insanity ... Cheers, Brett Shirley ESE (aka JET Blue) DeveloperSpeaking of the Melissa virus
As an Exchange Support Engineer, I dread the onset of any virus that will affect our customers. However, given that my name is Melissa, this virus was particularly painful for me! Imagine answering a support call during this outbreak “Thank you for calling Microsoft Exchange Support, this is Melissa” only to hear a deafening ‘click’ on the other end. A more typical response to my greeting was an awkward silence followed by “You’re kidding, right?” For a time, I seriously considered using Missy or even Mel when answering calls at work. A lot of customers would tease me that I was the Melissa that spurned the guy that started all of this. But I was ok with it, hey if people needed someone to blame, I’m a big girl, I can take it. Of course my co-workers couldn’t resist taking advantage of a ripe situation. They creatively decorated my cube with yellow police tape and assorted anti-virus CDs. But it was all in fun and I kept my sense of humor about it. It started to get a little extreme though when some of my emails would bounce back because of my name on the From line. Alas the Melissa virus is now a distant memory. However to this day, I have a former colleague who still refers to me only as “Virus”. Since he left Dove dark chocolate nuggets on my keyboard every morning, I am willing to overlook it. He actually left chocolates for all the women on our team. Truly this is a man who knows what women really want! So I think it’s only fair to dedicate this blog entry to my “Dove fairy” Edwah, from “Virus”. I miss you buddy! Melissa TraversExchange is 10 years old!
Seems like yesterday that Exchange 4.0 hit the market. In March of 1996 Exchange 4.0 was released. I thought I would give you a link for the geeks among you that have been along for the ride and for those that go even further back to ALL-IN-ONE, MailWorks, cc:Mail, IBM PROFS etc. ITPRO has a great look back if you're interested in taking a look back at the evolution of hardware, Exchange Clients, Mobile devices, OWA, etc. http://www.windowsitpro.com/Common/adforceimages/Decade_of_exchange.pdf It's been a whirl wind ride and now we are nearing Beta 2 of Exchange "12"! What will Exchange look like 10 years from now? What features would you want to see (hey why wait - ask for them now!). If Exchange 2016 was ready for release, what features would you be most excited by? So, take a look back, reflect on where we have been, then put on your thinking caps and tell us what you want to see in the next release and the one after that! - David EspinozaFirst post!!! of the Microsoft Exchange team blog
This is a group blog written by folks from the Microsoft Exchange server team. We are developers, testers, program managers, technical writers and product support engineers who work with Exchange. At the end of each blog entry, you'll see the author's name and a link to their bio. The bios of all participants can be seen in the "Team Bios" link on the left as well. We're planning on posting to this blog at least 3 times a week, with tips and tricks about Exchange, the history behind certain features or decisions, and other information about Exchange, Outlook, and email in general. And without further ado, the M drive...
