/Mixed-ing it up: Multipart/Mixed Messages and You
Greetings, Exchange Administrators! In today’s edition of “and You”, we’ll be covering Exchange’s handing of messages generated by iPhones, iPads, and Macintosh Mail clients. Specifically, we’re going to cover what /mixed content body messages are, how Exchange handles them now, and how we handle them down the line. Before we can dive into the content, you first have to understand how internet messages are structured, and that means learning a little bit about how Exchange stores messages, and quite a bit about MIME . Not the mute freaks – Multipurpose Internet Mail Extensions, also known as “The Mime not everyone hates.” Exchange, Messages, and LOL Cats Exchange stores messages as a series of properties, where each property has a name and a value. For instance, PR_SUBJECT is the subject property, and “Test Message” is the value. Messages in Exchange have one body with multiple forms of representing it (HTML, Rich Text and Plain text). The HTML view of the body looks like so: This is a cat. Do Not Want The RTF (rich text format) view of the same body is also capable of containing both formatting and images, and so would look like: This is a cat. <Mentally insert picture of cat here – it’s only eight lines up, so honestly, you can do it. > Do Not Want The plain text version of the body is composed of plain text, a fact that should be obvious based on its name. A good way of simulating plain text body generation is to paste your content into notepad. If it survives the paste into notepad, it will be part of the plain text. Sadly, the cat picture does not: This is a cat. Do Not Want Messages in Exchange can have multiple attachments. This is good, because even if Exchange is forced to generate a plain text version of the body, the cat picture can come along so that should you decide to click it, you can view a picture of a cat which does not want something. Key points for non technical and allergic to cats people: For Exchange, messages can have one body and can have many attachments. And in this corner, MIME MIME is a plain text format for email messages. MIME messages are divided into “parts”, each of which might have content or even child parts, like a series of Russian dolls. Each MIME part (even the root part, or the message itself) has a header called Content-Type, which describes the type of content in the part. Content type is divided into major parts and minor parts, separated by a slash. For example, consider the content-types multipart/alternative or multipart/mixed. Every part has a type, even going so far as to define a Miranda type for parts which can't afford to assign one (text/plain). Sometimes the types are quite helpful at understanding the meaning of the content, sometimes not so much. For instance, Content-Type: Application/PDF – that one means that it is Adobe’s Portable Document Format. On the other hand, Content-Type: application/octet means “I can’t tell what this is. Here’s a binary blob for your troubles. Hopefully you can figure out what to do with it” Multipart/ is a general type, meaning that this MIME part may contain many child MIME parts. The sub type of the part (the part after the slash) tells us more about the child parts, and in this case, how they are related to each other. Now we will take a closer look at some of the multipart sub types to see where things can go wrong. Relativity First off we’re going to look at Multipart/related, (also called a “related” body part). Related, in this case, means that the sub MIME parts are actually related to each other – in other words, that give the following MIME structure: 1. Multipart/related 1.1. Text/HTML 1.2. Image/Gif That 1.1 and 1.2 are not meant to be interpreted as “separate” parts – they have meaning as one. In this case the html contains image links to the 1.2 image (our friend the cat). Key point for non technical and allergic to cats people: Multipart/Related means “We belong together.” Alternatives Multipart/alternative means that each child of this part is a different representation of the same data. They are “Alternative” versions of each other. The intention is that a client picks the type that it can best display and displays that one. So given this mime structure: 1. Multipart/alternative 1.1.1. Text/Plain 1.1.2. Text/Html 1.1.3. Application/Pdf The client doesn’t have to show the text/plain part as the body. No, if the client knows how to display a text/html body, it is free to do that. So multipart/alternative is a way of grouping a number of different formats of the same data together and letting the client decide which one it shows best – it’s like kid’s beauty pageant, except that instead of the ladies from the rotary club, you have the email client as the judge. Key point for non technical and allergic to cats people: Multipart/Alternative means “Pick the one you like best.” Mixed Up Multipart/Mixed, according to RFC 1521, means that the parts are completely independent of each other (not related to each other) but that their order matters. What is the expected behavior? “Clients usually display the parts one after the other.” This, however, brings into play another parameter on the MIME part – Content-Disposition. This parameter has a couple of normal values – Inline and Attachment. Attachment is easy to understand – in the context of Exchange, it means “Show me in the well, that I may be blocked by Outlook from being saved or opened.” Inline, on the other hand, we handle differently. Remember that whole “messages have one body, and maybe many attachments” thing? Keep that in mind while we look at how our Cat message looks like with a /mixed body: 1. Multipart/Mixed 1.1. Text/Html - Inline 1.2. Application/Gif - Inline 1.3. Text/Html - Inline And the intention among clients that generate this is that the receiving client should display the text/html part first and then glob on the image to the end of it, and then the rest of the body. There’s no limit to multipart/madness, you can combine them (and dispositions) into nigh endless combinations. For example: 2. Multipart/Mixed 2.1. Text/Html -Inline 2.2. Image/Gif -Inline 2.3. Text/Html -Inline 2.4. Text/Plain -Inline Means “Show 2.1, followed by the image from 2.1 then the html from 2.3 and then the text from 2.4. Do it NOW.” 3. Multipart/Mixed 3.1. Text/HTML -Inline 3.2. Image/Gif –Attachment 3.3. Text/Html –Inline 3.4. Text/Plain –Attachment Means “Show the text from 2.1, NOT the attachment from 3.2 unless someone does something, the text from 3.3, but NOT the text in 3.4 (unless they do something like click an attachment in the well). The problem, of course, comes from Exchange’s original definition of a message – one body (with multiple representations), maybe many attachments. Key point for non technical and allergic to cats people: Multipart/Mixed can mean “Maybe show all of these, in the order listed.” Combo #5 MIME Types are not exclusive. I can combine Multipart/Mixed, Multipart/Alternative and Multipart/Related into a single message, and actually have a meaning 1. Multipart/Mixed 1.1. Multipart/Alternative 1.1.1.1. Text/Plain 1.1.1.2. Multipart/Related 1.1.1.2.1.1. Text/HTML 1.1.1.2.1.2. Image/Gif - inline 1.2. Image/Jpeg - attachment Yes, this structure is legal. And it is meaningful. To understand this you unwrap in order, one level at a time. Multi Mixed – this message is different parts, put together, and the order matters. Multipart/Alternative- I have two children, pick the prettier one and show it off. Text/Plain – I am a blob of plain text Multipart/Related – My children are bits and pieces of each other Text/HTML – Pretty, Pretty Text. Image/Gif – I am a picture of a cat, referenced by pretty, pretty text, I hope. Image/Jpeg – I’m an attachment (in case you couldn’t see the picture of the cat above). Exchange has always dealt well with multipart/alternative bodies, picking the one which we can best support and promoting it. We deal well with multipart/related as well – not every attachment on a message is visible – attachments have a disposition, which is either not set, inline or attachment. Setting neither indeterminate – the client does what the client does (and good luck establishing an algorithm that works for everyone). On the other hand, messages with /mixed content bodies where multiple parts are inline, those do not work so well. Blender’d Messages In the case of /mixed bodies, there are multiple MIME parts which are meant to combine together like Japanese robots to form Voltron, or an image of a cat and some text. Today, if you receive such a message, we do the best we can with it (which is pretty dis-satisfying): We pick the first “body type part” – aka, a text/<something> part, and that one becomes the body as seen by Outlook. All of the rest of them, those are attachments and we shove them into the attachment well. Oh, sure they might have a disposition of inline, but because the most common usage of inline is in HTML, we actually check, and anything that isn’t referenced by a link from the body, we won’t be fooled by. Into the well it goes. From an Outlook perspective you get the first part of the message, then two attachments, one of which is the picture and the other is the trailing text. You are welcome to open the attachments in order, and combine them in your head to form a message, but once you get more than a couple parts it isn’t reasonable. Exchange has never supported “proper” display of /mixed body messages in OWA or Outlook, until now. Blended Messages Starting with Exchange 2007 service pack 3 roll up 3 (E12SP3RU3) and Exchange 2010 service pack 1 rollup 4 (E14SP1RU4) are a set of changes to how Exchange handles /mixed body messages. We think that in general your users will enjoy the less mangled nature of these messages, but if I’ve learned anything from fourteen years of working on Exchange, it’s that “different == angry”. People get used to our behavior, so even when it’s wrong (or incomplete) they expect the same behavior, and come to rely on it. Consider this is your fair warning that this change is coming, some details on how it works, where it works, and where it doesn’t. We're adding support for Exchange to combine multiple body parts into a single, aggregate body. The short of this is that broken up messages should show combined together, and readable in OWA and Outlook. There are, however, limitations to this. First off, right now this will only work for message generated by Apple iPods, iPads, or Apple Mail clients. This isn’t an accident – we developed the rules used to combine these bodies using test data from our counterparts at Apple, and while we handle messages by them well, the internet is wide and wondrous, and anyone can write messages with multipart/mixed bodies. For now this is restricted to clients we have good test data on, good rules for and a good way to identify. To create the aggregate body, we check each MIME part in the /mixed body. If a MIME part has a disposition of Attachment, it goes to the attachment well. I am not going to argue with a client that specifies that it is an attachment. If a body part has a disposition of inline or not set, if it is a plain text or html body part, we add it to the aggregate body. If the body part is an image which can be displayed inline, we add a link to it in the aggregate body. If the body part is not text, or is an image we can't display inline, it goes to the well. How do I know if this breaks me? If you're the owner of an application which is used to sending in /mixed content messages with MIME parts that you rely on Exchange treating as well attachments, and you send them from an Apple platform, and you haven’t been setting a content disposition, and the parts are text or image types (and you are setting content type), then you need to add a Content-Disposition to the MIME parts you want to be attachments, and set them to Attachment. If you're a normal consumer wondering why messages with images or signatures got split into pieces, you don’t need to do anything. Conclusions Today we’ve covered different MIME structures, different representations of bodies, dispositions, types, and a host of other things, but the hard boiled summary is that mail between Apple clients and Exchange clients will be handled better. The best case scenario isn’t that your users call up and say “Wow, I’m really impressed that the messages I got from John on his Mac aren’t chopped up into a dozen pieces anymore.” The best case scenario is that things just work. No one has to call anyone, because you neither notice nor care what platform the message came from, or what format it was sent in. You can see your email, your signatures, and yes, your LOL cats. Enjoy! Epilogue: How things went wrong Already I’ve read (and responded to) reports on the Exchange Update Forums that this new code introduced a new problem. PDF attachments from Mac clients which declare their disposition to be Inline aren't visible in the attachment well. Visible in Outlook Web Access, yes, but MAPI, no. So how on earth did this happen? And how did we miss this bug? (“Do you do any quality control?” as one person asked.) The answer to this is yes, we do. But rather than just say that, let’s look at the journey we took to get this fix included in a rollup, what the problem is, how we missed it, and what we're doing about it. This fix begins with a conference call between me, a few team mates, and our counterparts at Apple, in which someone asked “Well, can you guys do anything to actually support this style of message body?” I spent the next few days researching what it would take to build a synthetic body out of the parts and eventually concluded that it would be possible. With that we began the discussions of “Why now?” and “What is it going to take?” That phase took quite a while. The core problem was that we were well aware that producing a new type of body was going to require a large investment in testing. Eventually we concluded that we could and would do it. And we did. We wrote the code that supports this type of body, we wrote the code that tests it. We have a huge number of tests that test various types and formats of bodies. We had only the new ones to test the new body. Line for line there's more code to test this than there's to support it. So the fix was done, the code was tested, but we still weren’t ready to check in the code. Instead, we took a much more cautious approach. The fix was available as an interim update and that interim update was tightly controlled since we wanted it to go to customers who would be putting it into daily use. After a month or so we loosened it up and the fix went out to four more customers, eventually being in play at around 12 customers for about three months. After three months of field deployment with positive results, we decided to schedule it for a rollup (which should've been RU2, but wound up being RU3). But there’s a problem. You see, Exchange 2007 and 2010 don’t pay much attention to Content-Disposition, because for years inline has been a great way to make sure the attachment is essentially invisible (an inline attachment has its hidden property set to true, since it is displayed inline with body content). And the test code missed this case – it’s an inline attachment which can't be displayed inline by Exchange, but in a message which contains multiple body parts which can be merged to build the synthetic body. Unfortunately neither our in-house nor field testing encountered this. The bug is that in processing these messages, we honor attachment disposition in a case where we should not. Any attachment which can't be displayed inline should never be allowed to become part of the synthetic body. We know of two types of attachments – TIFF and PDF, which can be displayed inline on Mac clients but not on Windows clients. The fix for these two also fixes any other types where the client might render inline but we can't. How did we miss this? We went back over our test code and test data and said “It contains PDFs, and validates the attachment well status.” It does indeed (and TIFFs as well). It also doesn't ever attempt to create a PDF attachment and render it inline in the body. That’s the missed case which I certainly wish we had found before it ever hit any customers. So to resolve this we're creating an interim update, and we’ll be rolling it into the main branch to prevent further incidents of this. Over time I expect that we'll expand the number of mailers we produce synthetic bodies for. For now, I think this experience has validated that we should keep it limited and expand support where we can closely test. I remain convinced that improving interop between Mac clients and Exchange is a good idea. So there we have it: The problem, the fix, the problem with the fix, and the fix for the problem with the fix. We now return to your regularly scheduled blog. Jason NelsonLife in a Post TMG World – Is It As Scary As You Think?
Let’s start this post about Exchange with a common question: Now that Microsoft has stopped selling TMG, should I rip it out and find something else to publish Exchange with? I have occasionally tried to answer this question with an analogy. Let’s try it. My car (let’s call it Threat Management Gateway, or TMG for short), isn’t actively developed or sold any more (like TMG). However, it (TMG) works fine right now, it does what I need (publishes Exchange securely) and I can get parts for it and have it serviced as needed (extended support for TMG ends 2020) and so I ‘m keeping it. When it eventually either doesn’t meet my requirements (I want to publish something it can’t do) or runs out of life (2020, but it could be later if I am ok to accept the risk of no support) then I’ll replace it. Now, it might seem odd to offer up a car analogy to explain why Microsoft no longer selling TMG is not a reason for Exchange customers to panic, but I hope you’ll agree, it works, and leads you to conclude that when something stops being sold, like your car, it doesn’t immediately mean you replace it, but instead think about the situation and decide what to do next. You might well decide to go ahead and replace TMG simply based on our decision to stop selling or updating it, that’s fine, but just make sure you are thinking the decision through. Of course, you might also decide not to buy another car. Your needs have changed. Think about that. Here are some interesting Exchange-related facts to help further cement the idea I’m eventually going to get to. We do not require traffic to be authenticated prior to hitting services in front of Exchange Online. We do not do any form of pre-authentication of services in front of our corporate, on-premises messaging deployments either. We have spent an awfully large amount of time as a company working on securing our code, writing secure code, testing our code for security, and understanding the threats that exist to our code. This is why we feel confident enough to do #1 and #2. We have come to learn that adding layers of security often adds little additional security, but certainly lots of complexity. We have invested in getting our policies right and monitoring our systems. This basically says we didn’t buy another car when ours didn’t meet our needs any more. We don’t use TMG to protect ourselves any more. Why did we decide that? To explain that, you have to cast your mind back to the days of Exchange and Windows 2000. The first thing to admit is that our code was less ‘optimal’ (that’s a polite way of putting it), and there were security issues caused by anonymous access. So, how did we (Exchange) tell you to guard against them? By using something called ISA (Internet Security and Acceleration – which is an odd name for what it was, a firewall). ISA, amongst other things, did pre-authentication of connections. It forced users to authenticate to it, so it could then allow only authenticated users access to Exchange. It essentially stopped anonymous users getting to Windows and Exchange. Which was good for Windows and Exchange, because there were all kinds of things that they could do if they got there anonymously. However once authenticated users got access, they too could still do those bad things if they chose to. And so of course could anyone not coming through ISA, such as internal users. So why would you use ISA? It was so that you would know who these external users were wouldn’t you? But do you really think that’s true? Do you think most customers a) noticed something bad was going on and b) trawled logs to find out who it was who did it? No, they didn’t. So it was a bit like an insurance policy. You bought it, you knew you had it, you didn’t really check to see if it covers what you were doing until you needed it, and by then, it was too late, you found out your policy didn’t cover that scenario and you were in the deep doo doo. Insurance alone is not enough. If you put any security device in front of anything, it doesn’t mean you can or should just walk away and call it secure. So at around the same time as we were telling customers to use ISA, back in the 2000 days, the whole millennium bug thing was over, and the proliferation of the PC, and the Internet was continuing to expand. This is a very nice write up on the Microsoft view of the world. Those industry changes ultimately resulted in something we called Trustworthy Computing. Which was all about changing the way we develop software – “The data our software and services store on behalf of our customers should be protected from harm and used or modified only in appropriate ways. Security models should be easy for developers to understand and build into their applications.” There was also the Secure Windows Initiative. And the Security Development Lifecycle. And many other three letter acronyms I’m sure, because whatever it was you did, it needed a good TLA. We made a lot of progress over those ten years since then. We delivered on the goal that the security of the application can be better managed inside the OS and the application rather than at the network layer. But of course most people still seem to think of security as being mainly at the network layer, so think for a moment about what your hardware/software/appliance based firewall does today. It allows connections from a destination, on some configurable protocol/port, to a configured destination protocol/port. If you have a load balancer, and you configure it to allow inbound connections to an IP on its external interface, to TCP 443 specifically, telling it to ignore everything else, and it takes those packets and forward them to your Exchange servers, is that not the same thing as a firewall? Your load balancer is a packet filtering firewall. Don’t tell your load balancing vendor that, they might want to charge you extra for it, but it is. And when you couple that packet level filtering firewall/load balancer with software behind it that has been hardened for 10 years against attacks, you have a pretty darn secure setup. And that is the point. If you hang one leg of your load balancer on the Internet, and one leg on your LAN, and you operate a secure and well managed Windows/Exchange Server – you have a more secure environment than you think. Adding pre-authentication and layers of networking complexity in front of that buys you very little extra, if anything. So let’s apply this directly to Exchange, and try and offer you some advice from all of this. What should YOU do? The first thing to realize is that you now have a CHOICE. And the real goal of this post is to help you make an INFORMED choice. If you understand the risks, and know what you can and cannot do to mitigate them, you can make better decisions. Do I think everyone should throw out that TMG box they have today and go firewall commando? No. not at all. I think they should evaluate what it does for them, and, if they need it going forward. If they do that, and decide they still want pre-auth, then find something that can do it, when the time to replace TMG comes. You could consider it a sliding scale, of choice. Something like this perhaps; So this illustrated that there are some options and choices; Just use a load balancer – as discussed previously, a load balancer only allowing in specified traffic, is a packet filtering firewall. You can’t just put it there and leave it though, you need to make sure you keep it up to date, your servers up to date and possibly employ some form of IDS solution to tell you if there’s a problem. This is what Office 365 does. TMG/UAG – at the other end of the scale are the old school ‘application level’ firewall products. Microsoft has stopped selling TMG, but as I said earlier, that doesn’t mean you can’t use it if you already have it, and it doesn’t stop you using it if you buy an appliance with it embedded. In the middle of these two extremes (though ARR is further to the left of the spectrum as shown in the diagram) are some other options. Some load balancing vendors offer pre-authentication modules, if you absolutely must have pre-auth (but again, really… you should question the reason), some use LDAP, some require domain joining the appliance and using Kerberos Constrained Delegation, and Microsoft has two options here too. The first, (and favored by pirates the world over) is Application Request Routing, or ARR! for short. ARR! (the ! is my own addition, marketing didn’t add that to the acronym but if marketing were run by pirates, they would have) “is a proxy based routing module that forwards HTTP requests to application servers based on HTTP headers and server variables, and load balance algorithms” – read about it here, and in the series of blog posts we’ll be posting here in the not too distant future. It is a reverse proxy. It does not do pre-authentication, but it does let you put a non-domain joined machine in front of Exchange to terminate the SSL, if your 1990’s style security policy absolutely requires it, ARR is an option. The second is WAP. Another TLA. Recently announced at TechEd 2013 in New Orleans is the upcoming Windows Server 2012 R2 feature – Web Application Proxy. A Windows 2012 feature that is focused on browser and device based access and with strong ADFS support and WAP is the direction the Windows team are investing in these days. It can currently offer pre-authentication for OWA access, but not for Outlook Anywhere or ActiveSync. See a video of the TechEd session here (the US session) and here (the Europe session). Of course all this does raise some tough questions. So let’s try and answer a few of those; Q: I hear what you are saying, but Windows is totally insecure, my security guy told me so. A: Yes, he’s right. Well he was right, in the yesteryear world in which he formed that opinion. But times have changed, and when was the last time he verified that belief? Is it still true? Do things change in this industry? Q: My security guy says Microsoft keeps releasing security patches and surely that’s a sign that their software is full of holes? A: Or is the opposite true? All software has the potential for bugs and exploits, and not telling customers about risks, or releasing patches for issues discovered is negligent. Microsoft takes the view that informed customers are safer customers, and making vulnerabilities and mitigations known is the best way of protecting against them. Q: My security guy says he can’t keep up with the patches and so he wants to make the server ‘secure’ and then leave it alone. Is that a good idea? A: No. It’s not (I hope) what he does with his routers and hardware based firewalls is it? Software is a point in time piece of code. Security software guards against exploits and attacks it knows of today. What about tomorrow? None of us are saying Windows, or any other vendor’s solution is secure forever, which is why a well-managed and secure network keeps machines monitored and patched. If he does not patch other devices in the chain, overall security is compromised. Patches are the reality of life today, and they are the way we keep up with the bad guys. Q: My security guy says his hardware based firewall appliance is much more secure than any Windows box. A: Sure. Right up to the point at which that device has a vulnerability exposed. Any security device is only as secure as the code that was written to counter the threats known at that time. After that, then it’s all the same, they can all be exploited. Q: My security guy says I can’t have traffic going all the way through his 2 layers of DMZ and multitude of devices, because it is policy. It is more secure if it gets terminated and inspected at every level. A: Policy. I love it when I hear that. Who made the policy? And when? Was it a few years back? Have the business requirements changed since then? Have the risks they saw back then changed any? Sure, they have, but rarely does the policy get updated. It’s very hard to change the entire architecture for Exchange, but I think it’s fair to question the policy. If they must have multiple layers, for whatever perceived benefit that gives (ask them what it really does, and how they know when a layer has been breached), there are ways to do that, but one could argue that more layers doesn’t necessarily make it better, it just makes it harder. Harder to monitor, and to manage. Q: My security guy says if I don’t allow access from outside except through a VPN, we are more secure. A: But every client who connects via a VPN adds one more gateway/endpoint to the network don’t they? And they have access to everything on the network rather than just to a single port/protocol. How is that necessarily more secure? Plus, how many users like VPN’s? Does making it harder to connect and get email, so people can do their job, make them more productive? No, it usually means they might do less work as they cannot bothered to input a little code, just so they can check email. Q: My security guy says if we allow users to authenticate from the Internet to Exchange then we will be exposed to an account lockout Denial of Service (DoS). A: Yes, he’s right. Well, he’s right only because account lockout policies are being used, something we’ve been advising against for years, as they invite account lockout DoS’s. These days, users typically have their SMTP address set to equal their User Principal Name (UPN) so they can log on with (what they think is) their email address. If you know someone’s email address, you know their account logon name. Is that a problem? Well, only if you use account lockout policies rather than using strong password/phrases and monitoring. That’s what we have been telling people for years. But many security people feel that account lockouts are their first line of defense against dictionary attacks trying to steal passwords. In fact, you could also argue that a bad guy trying out passwords and getting locked out now knows the account he’s trying is valid… Note the common theme in these questions is obviously – “the security g uy said…..”. And it’s not that I have it in for security guys generally speaking, but given they are the people who ask these questions, and in my experience some of them think their job is to secure access by preventing access. If you can’t get to it, it must be safe right? Wrong. Their job is to secure the business requirements. Or put another way, to allow their business to do their work, securely. After all, most businesses are not in the business of security. They make pencils. Or cupcakes. Or do something else. And is the job of the security folks working at those companies to help them make pencils, or cupcakes, securely, and not to stop them from doing those things? So there you go, you have choices. What should you choose? I’m fine with you choosing any of them, but only if you choose the one that meets your needs, based on your comfort with risk, based on your operational level of skill, and based on your budget. Greg Taylor Principal Program Manager Lead Exchange Customer Adoption Team
