From crush to product documentation: The story of Squeaky Lobster
When customers first hear about being able to enable extra JET Blue or ESE Database performance counters via adding a "Squeaky Lobster" registry value, they often think it must be a joke or ask you to repeat it. And invariable the question comes up ... _why_ "Squeaky Lobster"? Various lore or conjecture has surfaced around it ... that it was a developer's child's toy or that is just thought up after lunch one day. Not exactly ... and retrieving the information was a little tricky as the initial checkin was 2 years before I started ... The story starts with a certain developer, let's call him, "Andrew Goodsell", for the time being, but I call (or called) him, "boss", and I can assure you those explanations are not quite right. If you were to get incriminating information about your boss, you might wonder what is the proper time to blog about it? Right after you stop working for him I think is the right answer to that ... and as of today I have transferred to the Exchange organization, while "Andrew" has not. I am as they say, unsupervised. ;-) Andrew, is a very sharp man who prides himself on his professional decorum, and a bit of a perfectionist to boot. In fact his professionalism is what makes this singular lapse in judgment so amusing and fairly uncharacteristic of any decision he would make (or have let me make ;-) today. Andrew has been involved in a fair share of the database performance work in ESE and as anyone decent at perf knows, one of most critical steps is having a way of accurately measuring performance, base-lining performance, at several levels in a software stack. Exchange 5.5 added ESE Database performance counters to facilitate this work. Most of these perf counters were intended for only internal ESE and Exchange developer usage (as you can probably deduce from one of the code snipits below, and looking at what these counters measure... things like B+ Tree Inserts/sec for example). Enter cute girl ... So that autumn while Andrew was working on adding perf counters and working on Exchange 5.5's performance, a cute girl was managing computers in the Exchange Performance lab. Andrew was working on performance and adding performance counters while this girl worked with the powerful computers in the performance lab. It isn't a wonder how they ran into each other. She was cute, it is no wonder Andrew got a crush. Some of the simplest ESE performance counters would also be helpful for advanced administrators in debugging Exchange server issues (Cache Size, File Operations/sec, etc). Andrew mentioned he thought if competing companies had access to some of the more detailed perf counters, they might be able to reverse engineer our implementation details and steal intellectual capital (now that's the professional Andrew I know, trying to protect our intellectual capital). After he says this, he rolls his eyes and says something to the effect of "I don't know what I was thinking!" (this is because the implementation details are more complex than what the counters show, but at the time this made sense). So there was a need to split the ESE Database perf counters into two kinds. ... internal (dev only) and external (admins). The internal ones would have to be hidden. Now, fairly smitten with this girl, ANYTHING she did was fascinating (to Andrew). Oh come on, we've all been there. And one such thing she did was participate in a Giving Campaign, basically an event where we raise money for charities and non-profits. In order to encourage people to give money, various prizes would be donated and a raffle sort of mechanism to win the prizes would be done. The girl of interest donated a toy Squeaky Lobster to the charity raffle. Andrew thought that was the funniest and most random thing to contribute as a prize for a giving campaign ever. Who would want a Squeaky Lobster? Most random prize in giving campaign I guess equals most uneasy to guess registry value ever. Anyway, a name was chosen that was about as random as you could get for 1997 (as there weren't very many words available in 1997, I think the Seattle was recovering from that awful grunge thing). Knowing the crowd Andrew hung out with back then, it would not surprise me if they decided it at or on the way back from lunch as they used to work 12 hour days and take extra long lunches. An excerpt of Andrew's fateful check in from SLM logs (note, I've deleted the developer actual email aliases), relevant information highlighted: #F eseperf.cxx v1 #K text #O in #P 1.00 #T Fri Sep 26 11:15:00 1997 #A <dev_alias_1_deleted>3 #C 47424, 41913 finalize perf ctrs #I 2 #D 1113 27a28,32 > #pragma const_seg( ".text" ) > const char szDisplayDevOnly[] = "Squeaky Lobster"; #pragma const_seg() As an aside, note the ".text" segment notation, I had forgotten about that, now that's old school. Man, 8 years is a LONG time in the computer industry. And so a little over 4 months later, the first Squeaky Lobster Enabled product shipped on Feb 3, 1998, with ESE97 in Exchange 5.5. In fact this is when the database engine under Exchange was renamed from JET Blue to ESE (to avoid confusion with JET Red, which has only vestigial relations to JET Blue / ESE). ESE97 shipped in 1998, just to confuse everyone. The above picture is of the original Exchange 5.5 packaging with the original Squeaky Lobster. But at this point in our story, the counters are secret so it's not a big deal. But there is no way to keep any sort of more extensive analysis mechanism a secret for long ... eventually someone will need the information. And eventually an Exchange performance case came along sometime in 1998 (according to the earliest record I can find in the PSS DB) that required the extra analysis these performance counters offered. The PSS engineer told a customer how to enable the counters so they could analyze the customer's issue. I mean, it comes down to the customer, you do what is necessary and possible to support them. Then another case came in, and another, and eventually someone from PSS thought to publish a KB on 4/17/2000, KB 259895, what counters are enabled by squeaky lobster, _official_ Microsoft documentation admitting the existence of Squeaky Lobster!!! Well, that didn't last long (about 6 months). In about 2000 someone public (Brian Sheaffer / Paul Thurrott) noticed and of course such a thing was far too silly for the serious professionals that control the web site of most big corporations. ;-) About as silly as Squeaky Lobster for a prize in a charity benefit! I mean we're a professional organization. Now there are about 300 PSS related investigations, service tickets and KBs referencing this phrase (including the above one). There is a Squeaky Lobster in Andrew's office as well, but do not be fooled, it is not the original. That Squeaky Lobster was given to him by a PSS engineer who thought it was funny, or maybe to thank him for being repeatedly asked by customers, "What was that? Can you spell it?". A mere 4 and a half years after the checkin, someone wised up ... and for Windows 2003 RTM, and Exchange 2003 SP1, code was checked in to try "Show Advanced Counters" first, and if that fails try "Squeaky Lobster". The comment in the code is: > // deprecated name (yes, we are ending the insanity) > err = DwPerfUtilRegQueryValueEx(hkeyPerf,(char*)"Squeaky Lobster",&Type,&lpbData); Note: we moved away from that archaic .text seg stuff, the compilers are now smart enough. And finally to come full circle, once this happened the name of the registry value appropriate for product documentation on our web site. Note that page specifies Exchange 2003 SP1. If you replace "Show Advanced Counters" with "Squeaky Lobster", the instructions should work for Exchange 5.5 through Exchange 2003 RTM as well as current products, though the registry key you use for step 2, varies. For Ex 5.5 it is "ESE97" instead of "ESE". In Ex2k you have to use "ESE98" before SP2, and finally for SP2 and later, just "ESE". The same process works for Windows, but use "ESENT" instead of "ESE", oh and you have to enable the performance counters for Windows first. I promised an Exchange MVP, Michael B. Smith, about a year ago and I apologize for the delay, some would even mock me for taking so long with this single post, but at that time, the fact that there was a Ms. Squeaky Lobster unknowingly influencing Microsoft product development was not known. You got to wait until you get all the details, Eric. It wasn't till Andrew, accidentally spilled the information at a dinner. After that I blurted out, "I'm SOOO BLOGGING THAT!", Andrew had an immediately look of regret on his face, and now he won't really talk about it anymore. ;-) So that is about all the Squeaky Lobster trivia I could collect. The next time you have to really dig into Exchange (or Active Directory) database performance, just remember how responsible professional ESE software engineers are adding easily discoverable (based upon the activities of their current crush at the time) mechanisms for diagnosing your top issues. So with that in mind, the next time you go to buy a software product, remember to check the box to see if this is one of the many Squeaky Lobster Enabled products. Oh just checked an Exchange 2003 box, not actually listed on the box, hmmm, I guess serious professionals are in control of the packaging too. ;-) So Sept 26th of this year is Squeaky Lobster Day, the 9 year anniversary of the checkin of the Squeaky Lobster registry value, be sure to Squeaky Lobster a server at 11:15, and then go to an extra long lunch, and make a silly decision you'll regret and feel embarrassed about 9 years down the road ... sorry, Andrew, you can't stop the insanity ... Cheers, Brett Shirley ESE (aka JET Blue) DeveloperLet's talk about Server Extension Objects (SEO)
I’ve worked for Microsoft for about 8 years, all in the Exchange group, and I really enjoy it. The people who work here are bright, self-motivated, and we have the resources we need to get our jobs done. It feels great to ship world-class products that our customers love. However, not everything is perfect, it never is. As my father-in-law once said, “It’s not all grapes and gravy.” For example, the engineering group must work well together with the user education group to produce documentation that lets customers easily get the most out of the software we’ve produced. The single most frustrating episode I’ve endured has to do with the documentation for SEO, or the relative lack thereof. It’s not like I’m free of blame here, I worked on SMTP when these Server Extension Objects (SEO) were first implemented many years ago. So without further adieu, let’s talk about what I think should have been documented about SEO. Please read on, if you’d like to learn a little bit more about a soon-to-be obsolete technology (it will remain in some SKUs of Windows SMTP, but will be replaced in E12), and help me get something off my mind at the same time. Let’s first describe SEO a bit. SEO is a COM-based technology that allows customization of the Windows SMTP service. In the SEO model, information flows from the source (the SMTP server) to the sink (the SEO COM object), whenever an interesting event in processing an SMTP connection is reached. After the SMTP server is finished calling all the sinks, it then proceeds with handling the SMTP session, guided by the actions of the sinks. The last piece of the puzzle is the event bindings database, where the ordered collection of sinks to call for each processing event is persisted. One piece of code that customizes the Windows SMTP service is Exchange 2000/2003. Without the customization and extension made possible by SEO, Exchange would not be the compelling mail server that it is. So let’s pick apart the parts of the IIS metabase that contains the SEO information, and we’ll discover an object model along the way. To get us started, let’s use the adsutil.vbs script that comes with an IIS install, to inspect the “eventmanager” part of the metabase. If you run “cscript.exe %systemdrive%\inetpub\adminscripts\adsutil.vbs /enum_all eventmanager” you’ll see a big bunch of unintelligible output. Let’s dissect just a fraction of it: [/eventmanager/SourceTypes/{FB65C4DC-E468-11D1-AA67-00C04FA345F6}/EventTypes/{F6628C8D-0D5E-11d2-AA68-00C04FA35B82}] What does this mean? The first GUID can be seen in the public SDK file SmtpGuid.h, where it is given the name g_szGuidSmtpSourceType. The second GUID, located in the same file, corresponds with g_szcatidSmtpOnInboundCommand. To understand the rest of the string, we need to take a quick peek at the SEO object model. The root object here is IEventManager, which has a programmatic ID of “Event.Manager”, and is reflected in the “eventmanager” string. This object is the one you create and use to get at all the rest of the SEO entities. The IEventManager object contains an IEventSourceTypes object (please notice the “s” at the end here, we’ll see other similarly-named collection classes later), and the IEventSourceTypes object just contains a collection of IEventSourceType objects. Each IEventSourceType object contains an IEventTypes object, which, of course, just contains a set of IEventType objects. So, if we re-parse the string above, we can read it as telling us that the IEventManager contains an IEventSourceType with the ID {FB65C4DC-E468-11D1-AA67-00C04FA345F6}, and that this source knows how to generate {F6628C8D-0D5E-11d2-AA68-00C04FA35B82}, or g_szcatidSmtpOnInboundCommand, events. The rest of the info in the eventmanager part of the IIS metabase serves the same purpose, to define the two sources of SEO events, the SMTP and NNTP server, and to list the types of events that each source may generate. The other SEO information, which describes which consumers are to be called for which SEO event types, and in what order, is located in other parts of the metabase. For example, the first SMTP virtual server has its SEO information listed under smtpsvc/1/eventmanager. So if you type “cscript.exe %systemdrive%\inetpub\adminscripts\adsutil.vbs enum_all smtpsvc/1/eventmanager”, you’ll see info like [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/DisplayName] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/SinkClass] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/SourceProperties] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/SourceProperties/priority] [/smtpsvc/1/eventmanager/EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82}/Bindings/{2EFF593B-5451-4D20-891C-1D32BFE95A1C}/SourceProperties/rule] What could this mean? The first portion of the string -- /smtpsvc/1/eventmanager/ -- just lets us know we are looking at the IEventSource for the first SMTP virtual server instance. Note that there is a difference between an IEventSourceType and an IEventSource. The IEventSourceType is an abstract, like “SMTP server”, while the IEventSource is a literal instance, like “the first SMTP virtual server instance on this machine”. The next part of the string -- EventTypes/{F6628C8F-0D5E-11d2-AA68-00C04FA35B82} -- tells us we are looking at the g_szcatidSmtpOnInboundCommand event type. To understand the rest of the string, we need to hearken back to the SEO object model, which defines the IEventSource as containing an IEventBindingManager object. The IEventBindingManager object is used to read, write, create, and destroy IEventBinding objects. There is one IEventBinding object contained in the metabase for each point of customization between the SMTP server and external code. The event binding is how this customization is persisted when IIS is recycled or the system restarted. Finally, the IEventBinding object contains an IEventPropertyBag object, which allows the event binding to contain additional information about itself. Now that we know a little more, we can look at the second fragment of output and see that it’s referring to information regarding an SEO binding. The binding is attached to the first virtual SMTP server instance, and it’s for the SmtpOnInboundCommand event type. The binding has a few properties, too. But why can’t we see more? It turns out that the adsutil.vbs script makes use of the ADSI schema for the metabase to understand and display the data contained in the metabase. The SEO information is not contained in the ADSI schema, and so doesn’t display much interesting information at all. If we run the “displaybindings” script (see near the end of this posting for the full vbs script), we’ll see some more descriptive detail displayed: \Source Types\Sources\Bindings\{F6628C8D-0D5E-11d2-AA68-00C04FA35B82} Display Name = SMTP Protocol OnInboundCommand ID = {2EFF593B-5451-4D20-891C-1D32BFE95A1C} priority = 9000 rule = X-EXPS SinkClass = Exchange.PS-EXPS DisplayName = Exchange SMTP Protocol Security EXPS Sink The SEO information looks a little nicer. We have a bit more context given to the GUIDs, and we can see all of the properties and their values. All of the other information under /smtpsvc/1/eventmanager/ is there for the same purpose, to describe event bindings. Now we’ve looked at the SEO object model a bit, as well as pulled apart the SEO information in the metabase. And we’ve mentioned that SEO is not part of our future product shipments. I would describe ways you can use SEO to improve your system, but there really aren’t any. SEO information should just be left alone, there is nothing to be gained from tweaking it. So just as we’ve started, we’re already done. If you have more questions about SEO, or the customization that can be gained from working with it, take a look at MSDN, and let us know if you have any more questions. It was interesting, from my side, to get this technology out into the world, and it would be interesting to hear from you how you’re using it. Here is the content of the displaybindings vbs script: Option Explicit ' ' \nt\private\inet\iis\staxinc\export\smtpdisp.idl ' Const catidPtSourceType = "{FB65C4DC-E468-11D1-AA67-00C04FA345F6}" Const catidPtSmtpSvc = "{1B3C0666-E470-11D1-AA67-80C04FA345F6}" REM ======================================================================================================== REM Function: REM e -- keystroke saver for WScript.Echo REM REM Arguments: REM strMessage -- message to echo to screen REM REM ======================================================================================================== Sub e (strMessage) WScript.Echo strMessage End Sub REM ======================================================================================================== REM Function: REM WriteCollection -- echoes a property bag REM REM Arguments: REM objCollection -- object to write (must implement _NewEnum) REM REM ======================================================================================================== Sub WriteCollection(objCollection) Dim strProperty For Each strProperty In objCollection If IsObject(objCollection(strProperty)) Then WriteCollection objCollection(strProperty) Else e strProperty & " = " & objCollection(strProperty) End If Next End Sub Function GetEventTypeDesc(strEventType) On Error Resume Next Dim strRetVal Dim objComCat Set objComCat = CreateObject("Event.ComCat") strRetVal = objComCat.GetCategoryDescription(strEventType,0) If(0 = Len(strRetVal)) Then strRetVal = "Unknown (" & strEventType & ")" End If Set objComCat = Nothing GetEventTypeDesc = strRetVal End Function Dim objEventManager Dim objEventUtil Dim GUID_SourceGuid Dim strEventType Dim objBindingManager Dim objBindings Dim objBinding Dim strEventTypeDesc ' ' create server event objects ' Set objEventManager = CreateObject("Event.Manager") Set objEventUtil = CreateObject("Event.Util") ' ' make source guid for instance 1 ' GUID_SourceGuid = objEventUtil.GetIndexedGUID(catidPtSmtpSvc,CLng(1)) ' ' get binding manager ' Set objBindingManager = objEventManager.SourceTypes(catidPtSourceType).Sources(GUID_SourceGuid).GetBindingManager ' ' iterate through the event types ' For Each strEventType In objBindingManager strEventTypeDesc = GetEventTypeDesc(strEventType) e "Event Type : " & strEventTypeDesc 'strEventType e "====================================================" Set objBindings = objBindingManager.Bindings(strEventType) For Each objBinding In objBindings WriteCollection objBinding.EventBindingProperties e "" Next e "" Next ' ' cleanup ' Set objEventManager = Nothing Set objEventUtil = Nothing - Ed PrescottWhy is OOF an OOF and not an OOO?
Here's an interesting historical question - when we say Out of Office, why does it sometimes get shortened to ‘OOF’? Shouldn’t it be ‘OOO’? Inside Microsoft, ‘OOF’ means not just the message which says you’re Out of Office, but it has grown to mean the act of being Out of the Office too - so you’ll get people putting sticky notes on their door saying ‘OOF Thurs & Fri’ or even people verbally saying things like, "Oh, Kevin’s OOF on vacation for the rest of the week’. I suppose that sounds better than "Oh, Kevin’s OOO on vacation ..." OOF was a command used in the days of Microsoft’s Xenix mail system, which set a user as ‘Out of Facility’ - ie Out of the Office. The usage of the term ‘OOF’ just stuck, as did the term ‘Little r’ (e.g. on an email sent to a distribution list, "Who wants to go to the cinema tonight? Little ‘r’ if you’re interested", meaning reply just to me) - as preserved in Outlook with CTRL+R for Reply, and CTRL+SHIFT+R (aka Big R) for Reply All. Ewan Dalton/Mixed-ing it up: Multipart/Mixed Messages and You
Greetings, Exchange Administrators! In today’s edition of “and You”, we’ll be covering Exchange’s handing of messages generated by iPhones, iPads, and Macintosh Mail clients. Specifically, we’re going to cover what /mixed content body messages are, how Exchange handles them now, and how we handle them down the line. Before we can dive into the content, you first have to understand how internet messages are structured, and that means learning a little bit about how Exchange stores messages, and quite a bit about MIME . Not the mute freaks – Multipurpose Internet Mail Extensions, also known as “The Mime not everyone hates.” Exchange, Messages, and LOL Cats Exchange stores messages as a series of properties, where each property has a name and a value. For instance, PR_SUBJECT is the subject property, and “Test Message” is the value. Messages in Exchange have one body with multiple forms of representing it (HTML, Rich Text and Plain text). The HTML view of the body looks like so: This is a cat. Do Not Want The RTF (rich text format) view of the same body is also capable of containing both formatting and images, and so would look like: This is a cat. <Mentally insert picture of cat here – it’s only eight lines up, so honestly, you can do it. > Do Not Want The plain text version of the body is composed of plain text, a fact that should be obvious based on its name. A good way of simulating plain text body generation is to paste your content into notepad. If it survives the paste into notepad, it will be part of the plain text. Sadly, the cat picture does not: This is a cat. Do Not Want Messages in Exchange can have multiple attachments. This is good, because even if Exchange is forced to generate a plain text version of the body, the cat picture can come along so that should you decide to click it, you can view a picture of a cat which does not want something. Key points for non technical and allergic to cats people: For Exchange, messages can have one body and can have many attachments. And in this corner, MIME MIME is a plain text format for email messages. MIME messages are divided into “parts”, each of which might have content or even child parts, like a series of Russian dolls. Each MIME part (even the root part, or the message itself) has a header called Content-Type, which describes the type of content in the part. Content type is divided into major parts and minor parts, separated by a slash. For example, consider the content-types multipart/alternative or multipart/mixed. Every part has a type, even going so far as to define a Miranda type for parts which can't afford to assign one (text/plain). Sometimes the types are quite helpful at understanding the meaning of the content, sometimes not so much. For instance, Content-Type: Application/PDF – that one means that it is Adobe’s Portable Document Format. On the other hand, Content-Type: application/octet means “I can’t tell what this is. Here’s a binary blob for your troubles. Hopefully you can figure out what to do with it” Multipart/ is a general type, meaning that this MIME part may contain many child MIME parts. The sub type of the part (the part after the slash) tells us more about the child parts, and in this case, how they are related to each other. Now we will take a closer look at some of the multipart sub types to see where things can go wrong. Relativity First off we’re going to look at Multipart/related, (also called a “related” body part). Related, in this case, means that the sub MIME parts are actually related to each other – in other words, that give the following MIME structure: 1. Multipart/related 1.1. Text/HTML 1.2. Image/Gif That 1.1 and 1.2 are not meant to be interpreted as “separate” parts – they have meaning as one. In this case the html contains image links to the 1.2 image (our friend the cat). Key point for non technical and allergic to cats people: Multipart/Related means “We belong together.” Alternatives Multipart/alternative means that each child of this part is a different representation of the same data. They are “Alternative” versions of each other. The intention is that a client picks the type that it can best display and displays that one. So given this mime structure: 1. Multipart/alternative 1.1.1. Text/Plain 1.1.2. Text/Html 1.1.3. Application/Pdf The client doesn’t have to show the text/plain part as the body. No, if the client knows how to display a text/html body, it is free to do that. So multipart/alternative is a way of grouping a number of different formats of the same data together and letting the client decide which one it shows best – it’s like kid’s beauty pageant, except that instead of the ladies from the rotary club, you have the email client as the judge. Key point for non technical and allergic to cats people: Multipart/Alternative means “Pick the one you like best.” Mixed Up Multipart/Mixed, according to RFC 1521, means that the parts are completely independent of each other (not related to each other) but that their order matters. What is the expected behavior? “Clients usually display the parts one after the other.” This, however, brings into play another parameter on the MIME part – Content-Disposition. This parameter has a couple of normal values – Inline and Attachment. Attachment is easy to understand – in the context of Exchange, it means “Show me in the well, that I may be blocked by Outlook from being saved or opened.” Inline, on the other hand, we handle differently. Remember that whole “messages have one body, and maybe many attachments” thing? Keep that in mind while we look at how our Cat message looks like with a /mixed body: 1. Multipart/Mixed 1.1. Text/Html - Inline 1.2. Application/Gif - Inline 1.3. Text/Html - Inline And the intention among clients that generate this is that the receiving client should display the text/html part first and then glob on the image to the end of it, and then the rest of the body. There’s no limit to multipart/madness, you can combine them (and dispositions) into nigh endless combinations. For example: 2. Multipart/Mixed 2.1. Text/Html -Inline 2.2. Image/Gif -Inline 2.3. Text/Html -Inline 2.4. Text/Plain -Inline Means “Show 2.1, followed by the image from 2.1 then the html from 2.3 and then the text from 2.4. Do it NOW.” 3. Multipart/Mixed 3.1. Text/HTML -Inline 3.2. Image/Gif –Attachment 3.3. Text/Html –Inline 3.4. Text/Plain –Attachment Means “Show the text from 2.1, NOT the attachment from 3.2 unless someone does something, the text from 3.3, but NOT the text in 3.4 (unless they do something like click an attachment in the well). The problem, of course, comes from Exchange’s original definition of a message – one body (with multiple representations), maybe many attachments. Key point for non technical and allergic to cats people: Multipart/Mixed can mean “Maybe show all of these, in the order listed.” Combo #5 MIME Types are not exclusive. I can combine Multipart/Mixed, Multipart/Alternative and Multipart/Related into a single message, and actually have a meaning 1. Multipart/Mixed 1.1. Multipart/Alternative 1.1.1.1. Text/Plain 1.1.1.2. Multipart/Related 1.1.1.2.1.1. Text/HTML 1.1.1.2.1.2. Image/Gif - inline 1.2. Image/Jpeg - attachment Yes, this structure is legal. And it is meaningful. To understand this you unwrap in order, one level at a time. Multi Mixed – this message is different parts, put together, and the order matters. Multipart/Alternative- I have two children, pick the prettier one and show it off. Text/Plain – I am a blob of plain text Multipart/Related – My children are bits and pieces of each other Text/HTML – Pretty, Pretty Text. Image/Gif – I am a picture of a cat, referenced by pretty, pretty text, I hope. Image/Jpeg – I’m an attachment (in case you couldn’t see the picture of the cat above). Exchange has always dealt well with multipart/alternative bodies, picking the one which we can best support and promoting it. We deal well with multipart/related as well – not every attachment on a message is visible – attachments have a disposition, which is either not set, inline or attachment. Setting neither indeterminate – the client does what the client does (and good luck establishing an algorithm that works for everyone). On the other hand, messages with /mixed content bodies where multiple parts are inline, those do not work so well. Blender’d Messages In the case of /mixed bodies, there are multiple MIME parts which are meant to combine together like Japanese robots to form Voltron, or an image of a cat and some text. Today, if you receive such a message, we do the best we can with it (which is pretty dis-satisfying): We pick the first “body type part” – aka, a text/<something> part, and that one becomes the body as seen by Outlook. All of the rest of them, those are attachments and we shove them into the attachment well. Oh, sure they might have a disposition of inline, but because the most common usage of inline is in HTML, we actually check, and anything that isn’t referenced by a link from the body, we won’t be fooled by. Into the well it goes. From an Outlook perspective you get the first part of the message, then two attachments, one of which is the picture and the other is the trailing text. You are welcome to open the attachments in order, and combine them in your head to form a message, but once you get more than a couple parts it isn’t reasonable. Exchange has never supported “proper” display of /mixed body messages in OWA or Outlook, until now. Blended Messages Starting with Exchange 2007 service pack 3 roll up 3 (E12SP3RU3) and Exchange 2010 service pack 1 rollup 4 (E14SP1RU4) are a set of changes to how Exchange handles /mixed body messages. We think that in general your users will enjoy the less mangled nature of these messages, but if I’ve learned anything from fourteen years of working on Exchange, it’s that “different == angry”. People get used to our behavior, so even when it’s wrong (or incomplete) they expect the same behavior, and come to rely on it. Consider this is your fair warning that this change is coming, some details on how it works, where it works, and where it doesn’t. We're adding support for Exchange to combine multiple body parts into a single, aggregate body. The short of this is that broken up messages should show combined together, and readable in OWA and Outlook. There are, however, limitations to this. First off, right now this will only work for message generated by Apple iPods, iPads, or Apple Mail clients. This isn’t an accident – we developed the rules used to combine these bodies using test data from our counterparts at Apple, and while we handle messages by them well, the internet is wide and wondrous, and anyone can write messages with multipart/mixed bodies. For now this is restricted to clients we have good test data on, good rules for and a good way to identify. To create the aggregate body, we check each MIME part in the /mixed body. If a MIME part has a disposition of Attachment, it goes to the attachment well. I am not going to argue with a client that specifies that it is an attachment. If a body part has a disposition of inline or not set, if it is a plain text or html body part, we add it to the aggregate body. If the body part is an image which can be displayed inline, we add a link to it in the aggregate body. If the body part is not text, or is an image we can't display inline, it goes to the well. How do I know if this breaks me? If you're the owner of an application which is used to sending in /mixed content messages with MIME parts that you rely on Exchange treating as well attachments, and you send them from an Apple platform, and you haven’t been setting a content disposition, and the parts are text or image types (and you are setting content type), then you need to add a Content-Disposition to the MIME parts you want to be attachments, and set them to Attachment. If you're a normal consumer wondering why messages with images or signatures got split into pieces, you don’t need to do anything. Conclusions Today we’ve covered different MIME structures, different representations of bodies, dispositions, types, and a host of other things, but the hard boiled summary is that mail between Apple clients and Exchange clients will be handled better. The best case scenario isn’t that your users call up and say “Wow, I’m really impressed that the messages I got from John on his Mac aren’t chopped up into a dozen pieces anymore.” The best case scenario is that things just work. No one has to call anyone, because you neither notice nor care what platform the message came from, or what format it was sent in. You can see your email, your signatures, and yes, your LOL cats. Enjoy! Epilogue: How things went wrong Already I’ve read (and responded to) reports on the Exchange Update Forums that this new code introduced a new problem. PDF attachments from Mac clients which declare their disposition to be Inline aren't visible in the attachment well. Visible in Outlook Web Access, yes, but MAPI, no. So how on earth did this happen? And how did we miss this bug? (“Do you do any quality control?” as one person asked.) The answer to this is yes, we do. But rather than just say that, let’s look at the journey we took to get this fix included in a rollup, what the problem is, how we missed it, and what we're doing about it. This fix begins with a conference call between me, a few team mates, and our counterparts at Apple, in which someone asked “Well, can you guys do anything to actually support this style of message body?” I spent the next few days researching what it would take to build a synthetic body out of the parts and eventually concluded that it would be possible. With that we began the discussions of “Why now?” and “What is it going to take?” That phase took quite a while. The core problem was that we were well aware that producing a new type of body was going to require a large investment in testing. Eventually we concluded that we could and would do it. And we did. We wrote the code that supports this type of body, we wrote the code that tests it. We have a huge number of tests that test various types and formats of bodies. We had only the new ones to test the new body. Line for line there's more code to test this than there's to support it. So the fix was done, the code was tested, but we still weren’t ready to check in the code. Instead, we took a much more cautious approach. The fix was available as an interim update and that interim update was tightly controlled since we wanted it to go to customers who would be putting it into daily use. After a month or so we loosened it up and the fix went out to four more customers, eventually being in play at around 12 customers for about three months. After three months of field deployment with positive results, we decided to schedule it for a rollup (which should've been RU2, but wound up being RU3). But there’s a problem. You see, Exchange 2007 and 2010 don’t pay much attention to Content-Disposition, because for years inline has been a great way to make sure the attachment is essentially invisible (an inline attachment has its hidden property set to true, since it is displayed inline with body content). And the test code missed this case – it’s an inline attachment which can't be displayed inline by Exchange, but in a message which contains multiple body parts which can be merged to build the synthetic body. Unfortunately neither our in-house nor field testing encountered this. The bug is that in processing these messages, we honor attachment disposition in a case where we should not. Any attachment which can't be displayed inline should never be allowed to become part of the synthetic body. We know of two types of attachments – TIFF and PDF, which can be displayed inline on Mac clients but not on Windows clients. The fix for these two also fixes any other types where the client might render inline but we can't. How did we miss this? We went back over our test code and test data and said “It contains PDFs, and validates the attachment well status.” It does indeed (and TIFFs as well). It also doesn't ever attempt to create a PDF attachment and render it inline in the body. That’s the missed case which I certainly wish we had found before it ever hit any customers. So to resolve this we're creating an interim update, and we’ll be rolling it into the main branch to prevent further incidents of this. Over time I expect that we'll expand the number of mailers we produce synthetic bodies for. For now, I think this experience has validated that we should keep it limited and expand support where we can closely test. I remain convinced that improving interop between Mac clients and Exchange is a good idea. So there we have it: The problem, the fix, the problem with the fix, and the fix for the problem with the fix. We now return to your regularly scheduled blog. Jason NelsonHow the M: Drive came about
In Exchange 2000, we introduced a new feature called IFS. IFS stands for “Installable File System”. This uses a little known and even less used feature of NT that allows the OS’s file system (like NTFS or FAT) to be replaced. The initial reason for doing that was as an optimization: it would allow protocols, such as NNTP and SMTP, to transfer the MIME messages directly as files. In Exchange 5.5, MIME messages are broken down into MAPI properties and stored in database tables. When they need to be accessed as MIME, they are put back together. In E2K, MIME messages are stored as MIME files in IFS and only converted into MAPI if a MAPI client (such as Outlook) accesses them. The other perceived benefit of IFS was that the Exchange storage objects could then be made visible through the file system. So you could go to a drive letter (M: was chosen for two reasons: first, “M” for “Mail”, and second, because it was in the middle of the alphabet and least likely to collide either with actual storage drives -which start at A and move up - or mapped network drives - which start at Z and move down), and get a list of mailboxes, navigate to mail folders via cmd or windows explorer and look at actual messages. This was considered pretty neat at the time and since it didn’t seem to be much more work to allow that access, it was thrown in (there may have been other, better reasons but I’m not aware of them). This ended up causing some challenges down the line related to the intricacies in how email objects need to be handled and mapping the file access behavior to them One of the biggest problems encountered was around security descriptors. This is difficult to explain without a detailed understanding of NT security descriptors, so I will simplify the explanation for the purpose of this discussion. The main part of an NTSD is called a DACL (discretionary access control list). It contains a list of users and groups and what they can do to that object. There are two main types of entries: allows, which say what an entity can do; and denies, which say what they can’t. The order of this list is very important. A standard sequence of entry types is called “canonical”. NT canonical form calls for a particular sequence. Because of legacy issues, MAPI canonical form requires a difference sequence of entry types. Applications that modify security expect a particular sequence and will behave erratically if the sequence is wrong. By creating or modifying objects through the M: drive, it will change the canonical format of the DACL’s and result in unexpected security behavior. This is bad. A related issue here has to do with item level security. E2K also introduced this feature, which is that items in a folder can be secured independently of each other and the folder. While this has some great uses, for many email systems this level of security is not needed. When a message has the folder default security, it simply references that property in the folder. When a message has its own security, there is an additional property that needs to be stored (this also has an affect on how folder aggregate properties, such as unread count, are computed). Having lots of individual security descriptors can result in both increased storage size and poor performance. When a message is created or modified through the M: drive, it will always get an individual security descriptor stamped on it, even if it is exactly the same as the folder default. This can also lead to unexpected behavior. For instance, if you change the default security on the folder, it will not change the security on any messages in it that have their own security descriptors. They have to be resecured individually. Another challenge is in relation to virus scanners. Virus scanners typically look for valid storage drives and spin through all the files on those drives and check them against virus signatures. The M: drive appears as a normal drive, so virus scanners were picking this up and processing it. This can have very detrimental affects on system performance and may also result in message corruption in some cases. Finally, IFS runs in kernel mode. This is a privileged execution path and it means that problems in this area can have much more severe affects (and be harder to track down) than in other areas of Exchange, which all run in user mode. Blue screens are one possibility if something goes wrong. IFS has given Exchange 2000 and Exchange 2003 a lot of advantages: we maintain content parity for MIME and make MIME message handling faster and more efficient as well as increasing the performance of such messages retrieved via internet protocols. But as I described above, there can be problems if IFS is misused via the M: drive. In Exchange 2003 we have disabled the M: drive by default to hopefully help reduce the likelihood that customers will encounter any of the issues described above. I encourage every system administrator to keep this disabled on E2K3 and disable it on all E2K servers as well. Jon AvnerThe history of content conversion in Exchange
The Exchange server and the client (that evolved into Outlook) were originally (circa 1992) based on the MAPI standard (stands for Messaging Application Programming Interface). Broadly this can be divided into the MAPI data model, and the MAPI object schema. Very simplistically, the MAPI data model can be summed as: message stores (think mailboxes) that contain folders that contain items that contain attachments. And all of the above entities are simply a collection of scalar properties (name, value pairs). The original MAPI schema laid out the common list of attributes applicable to email messages, for eg: Subject, Received Time etc. This has since been extensively extended by Outlook and Exchange for various advanced functionality. The Exchange server store was implemented to store the above data model in a very efficient manner, using a database technology called internally as Jet Blue. This data model, although simplistic in this day and age, has proved extremely flexible and surprisingly powerful to model many things that the original inventors could never have even imagined. And the Exchange server, due to its excellent implementation of this data model, has reaped great rewards. The primary mail transport protocol used for business e-mail back then was X.400. At about 1995 or so, the internet wave came along. The Web, the Browser, and HTML changed the world. Along with them the popularity of other standards based protocols for email increased, such as SMTP for mail exchange and MIME as the serialization format for email messages. Both of these rocked the ship of Exchange back then. The Exchange product had already been in development since '92 and more delays to complete it and ship it would have pretty much meant disbanding it altogether. The pressure from the executive level to ship it was intense. But folks in Exchange back then were wise enough to see that this internet wave was too big to just be a passing fancy and that these new standards must be embraced. As a compromise, support for SMTP as a mail exchange protocol (instead of just X.400) and MIME as a data interchange format (instead of just MAPI) were added to the product, but on the periphery so that the core product won't be affected much. Fortunately, the existing transport architecture had supported the notion of Gateways to connect to disparate other mail systems that existed then (again circa '92) who were foolish enough to not support X.400 natively! For example to connect with ccMail, GroupWise and the likes. We decided to lump this new kid on the block (SMTP) in the same category and modelled it simply as a gateway add-on to connect to this thing called the "internet"! This was called the Internet Mail Connector or simply the IMC. This IMC also then got the fun task of back and forth conversion between the data format stored native in the Exchange store (MAPI) and the data format that was the new internet standard (MIME). A conversion library was created for this purpose and was named, somewhat oddly now, as IMAIL (Internet Mail). This was the how we shipped Exchange 4.0 (March 13 1996). The next release, 5.0, was a very focused one in which we made very minor changes, although we did also add POP support. Both the IMC and IMAIL stayed pretty much the same as they were in 4.0. In the next release, 5.5, we decided to add support for IMAP4 as well. Now, POP and IMAP were similar to SMTP in insisting on MIME as the data interchange format. So it was decided to move this IMAIL conversion library closer to the actual data store (that was still a MAPI store of course) to improve performance. In addition, we invested significantly in this library to do a top notch job of translating to and from these 2 formats, and this has been quite a success overall. Exchange 5.5 was a very successful release thanks to the right balance of functionality and simplicity. The next release was Exchange 2000. By this time the domination of the internet protocols was so complete that we was decided to swap the roles of SMTP and X.400. That is, to make SMTP our primary mail exchange protocol even between Exchange servers, and treat X.400 as a one-off Gateway to reach pockets of the world that were still using it (mostly Europe). Also, we decided to go even further along the road of embracing MIME as the data format. We decided to actually enhance the core Exchange store to make it natively store and "understand" MIME. In this release we also invested in a new file system technology that would let us store these large "streams" of MIME content more efficiently. See Jon Avner's blog on the M: drive for more details on this one. In order to satisfy our richest client (Outlook) that continued to demand a MAPI data model, we had to engineer the Exchange store to do on-demand or deferred conversions back and forth between the 2 formats. As one can imagine, this was a very challenging piece of engineering, fraught with subtle traps. In the Exchange 2003 release we did not make any significant changes in this area, and status quo from the 2000 release remains. Naresh SundararajanHow does your Exchange garden grow?
One question we often get asked when talking to customers contemplating an Exchange upgrade or a switch from a competitor’s mail system is, "how many users per server can Exchange handle?" Nowadays, that’s an open question - it very much depends on what kind of users you have, what kind of storage you’re using and how powerful your servers are. When Exchange 4.0 was released in 1996, a decent server might have had 256Mb of RAM and a 90MHz Pentium processor, with maybe a handful of GBs of SCSI disk in the box and possibly a DAT tape drive. Users’ mailboxes might have been in the 10-20Mb size range, and the average user sent or received only a small amount of email per day. At that time, server sizing was pretty much a function of how much computing horsepower you could afford - the CPU power, disk size & speed and memory capacity available (along with the all-important user profile) would determine the number of users per server, and arbitrary decisions would be made about maximum message sizes, mailbox size etc. Now, it’s possible to buy even mid-range servers that will cope with many thousands of users, and the bottleneck has moved down to the storage level in many instances as user mailboxes have grown in size and we send and receive far more mail, and many larger messages too. We’d be interested in hearing anecdotes about particularly unusual messaging environments you might have - how large do your servers grow? What’s the biggest mailbox you’ve ever seen (assuming you’re not restricting with quotas)? How much data does your total Exchange system manage? What’s the biggest single database - etc, etc... you get the picture. Some factoids to get you started... if you have anything along these lines, please post a comment: - One customer (who is part of the Exchange Technology Adoption Program) sets no limit on message sizes internally - and one time saw a single message with attachments totaling 2.4Gb in size! (which Exchange delivered! - the recipient’s mailbox quota was instantly blown apart though...) - The largest single Exchange 5.5 Public Folder Information Store I’ve come across was 800Gb in size - There’s a limit of 1,000 Exchange 2000/2003 servers in an org - has anyone come close, or hit that limit? - One customer deploys around 250,000 Exchange users per server and has over 9,000,000 users in total (you guessed it, they’re an ISP) - The most heavy user of public folders in the world (that we know about) has about 1,500,000 public folders containing around 8Tb of data So come on: let’s hear your war stories - post a comment with your amazing data :) - Ewan DaltonSqueaky Lobster rears its head again in the Microsoft employee giving campaign
This post is mainly for other softies who read the blog, but we hope our customers and partners enjoy it with us as the squeaky lobster has long been part of Exchange culture and lore and -- if we have anything to do with it -- will continue to embarrass the developer who started it all for years to come. A little over 11 years ago, a developer on the JET team made an infamous checkin, to create a reg key controlling some obscure perf counters, named "Squeaky Lobster". The name was a reference to a girl he had a crush on at the time who donated a squeaky lobster to the annual Microsoft Giving Campaign charity auction. He thought it was a funny thing to donate to a giving campaign and gee whilikers, he had a checkin to make that day that surely nobody else would ever see, so why not? And so, it began. Not with a bang, but a whimper. But it doesn't end there, no no no. Fortunately, said developer has some friends and co-workers who won't let him forget about this incident of his youth, and celebrate Squeaky Lobster Day every September 26 th by attempting to embarrass him. This year, the celebration is a little delayed, because the giving campaign doesn't start until October. and we decided that after 11 years (we turned it up to 11, awww yeah), it was time to donate another Squeaky Lobster to the giving campaign - except this time, it would be signed by Andrew himself! Unfortunately this auction is only available inside Microsoft, but for those of you who do work here we encourage you to please bid on this (priceless, really) piece of Exchange history and continue the noble tradition of never letting Andrew forget: (URL to donation site removed) - KC Lemson
Restore Teams private chat history to new mailbox
Hi, I'm having an problem with restoring Teams chat history to new user accounts and and their mailbox. I disabled Azure AD sync for a few users and deleted their account in the cloud tenant from the recyclebin and enabled the sync again to solve an issue with calendar syncing with on-premise mailboxes. Unfortunately their Teams chat history also vanished from their Teams account and now I'm searching for a way to restore their history. I can see the deleted mailboxes and should be able to restore the whole mailbox to a temporary account but I have no idea if the missing data is in there or how I can move the Teams chat history to the new mailbox. Can anyone give me hint or solution? Thanks and Regards
