All Posts
431 TopicsWhy is OOF an OOF and not an OOO?
Here's an interesting historical question - when we say Out of Office, why does it sometimes get shortened to ‘OOF’? Shouldn’t it be ‘OOO’? Inside Microsoft, ‘OOF’ means not just the message which says you’re Out of Office, but it has grown to mean the act of being Out of the Office too - so you’ll get people putting sticky notes on their door saying ‘OOF Thurs & Fri’ or even people verbally saying things like, "Oh, Kevin’s OOF on vacation for the rest of the week’. I suppose that sounds better than "Oh, Kevin’s OOO on vacation ..." OOF was a command used in the days of Microsoft’s Xenix mail system, which set a user as ‘Out of Facility’ - ie Out of the Office. The usage of the term ‘OOF’ just stuck, as did the term ‘Little r’ (e.g. on an email sent to a distribution list, "Who wants to go to the cinema tonight? Little ‘r’ if you’re interested", meaning reply just to me) - as preserved in Outlook with CTRL+R for Reply, and CTRL+SHIFT+R (aka Big R) for Reply All. Ewan Dalton383KViews42likes8CommentsSupport of DANE and DNSSEC in Office 365 Exchange Online
Microsoft is committed to providing world-class email security solutions and the support for the latest Internet standards in order to provide advanced email protection for our customers. Today we are announcing that Exchange Online will be adding support for two new Internet standards specific to SMTP traffic.84KViews25likes30CommentsOffice 365 Message Attribution
When a message arrives at Office 365, one of the first things we need to do is figure out which organization it belongs to. At first, this sounds simple – just look at the recipient, right? Well, it is more complicated than that, because of Hybrid and complex routing scenarios.105KViews16likes14CommentsBasic Authentication and Exchange Online – February 2021 Update
We previously announced we would begin to disable Basic Auth for five Exchange Online protocols in the second half of 2021. we are announcing some important changes to our plan to disable Basic Auth in Exchange Online.242KViews14likes67CommentsBasic Authentication and Exchange Online – July Update
Today we are pleased to announce some new changes to Modern Authentication controls in the Microsoft 365 Admin Center, exposing simpler options for customers to manage both Modern and Basic Authentication requirements within their organizations.141KViews14likes25CommentsModern Auth and Unattended Scripts in Exchange Online PowerShell V2
Today, we are happy to announce the Public Preview of a Modern Auth unattended scripting option for use with Exchange Online PowerShell V2. This feature provides customers the ability to run non-interactive scripts using Modern Authentication.253KViews14likes148CommentsAnnouncing OAuth 2.0 support for IMAP and SMTP AUTH protocols in Exchange Online
Ever since we announced our intention to disable Basic Authentication in Exchange Online we said that we would add Modern Auth (OAuth 2.0) support for the IMAP, POP and SMTP protocols. Today, we’re excited to announce the availability of OAuth 2.0 authentication for IMAP and SMTP AUTH protocols to Exchange Online mailboxes.256KViews14likes101CommentsMe Too!
One way of telling how long a Microsoft employee has been working here is their reaction to the phrase “Bedlam DL3”. Just for grins, I was at lunch in the cafeteria with a bunch of co-workers and I blurted out, totally out of context: “Bedlam DL3”. About 3 of the old-timers in the group responded, in chorus “Me Too!” So why does everyone know about this rather mysterious phrase? Well, Microsoft’s a pretty big organization. We’ve got well over 100,000 mailboxes in our email infrastructure, and at times it can become rather cumbersome to manage all these. One of the developers in our Internal Technologies Group (also known as ITG, basically the MIS department at Microsoft) was working on a new tool to manage communications with the various employees at Microsoft, and as a part of this tool, he created several distribution lists. Each distribution list had about a quarter of the mailboxes in the company on it (so there were about 13,000 mailboxes on each list). For whatever reason, the distribution lists were named “Bedlam DL<n>” (maybe the tool was named Bedlam? I’m not totally sure). Well the name of the lists certainly proved prophetic. It all started one morning when someone looked at the list of DL’s they were on, and discovered that they were on this mysterious distribution list called “Bedlam DL3”. So they did what every person should do in that circumstance (not!). They sent the following email: To: Bedlam DL3 From: <User> Subject: Why am I on this mailing list? Please remove me from it. Remember, there are 25,000 people on this mailing list. So all of a sudden, all 25,000 people received the message. And almost to a person, they used the “reply-all” command and sent: To: Bedlam DL3 From: <User> Subject: RE: Why am I on this mailing list? Please remove me from it. Me too! In addition, there were some really helpful people on the mailing list too: They didn’t respond with just “Me Too!” They responded with: To: Bedlam DL3 From: <User> Subject: RE: Why am I on this mailing list? Please remove me from it. Stop using reply-all – it bogs down the email system. You know what? They were right - the company’s email system did NOT deal with this gracefully. Why? Well, you’ve got to know a bit more about how Exchange works internally. First off, the original mail went to 13,000 users. Assuming that 1,000 of those 13,000 users replied, that means that there are 1,000 replies being sent to those 13,000 users. And it turns out that a number of these people had their email client set to request read receipts and delivery receipts. Each read and delivery receipt causes ANOTHER email to be sent from the recipient back to the sender (all 13,000 recipients). Assuming that 20% of the 1,000 users replying had read receipts or delivery receipts set, that meant that every one of the message that they sent caused another message to be sent for every one of the 13,000 recipients. So how many messages were sent? First there were the basic messages – that’s 13,000,000 messages. Next there were the receipts – 200 users, 13,000 receipts – that’s and additional 2,600,000 messages. So about 15.5 MILLION messages were sent through the system. In about an hour. So at a minimum, 15,600,000 email messages will be delivered into peoples mailboxes. But Exchange can handle 15,600,000 email messages EASILY. There’s another problem that’s somewhat deeper. An Exchange email message actually has TWO recipient lists – there’s the recipient list that the user sees in the To: line on their email message. This is called the P2 recipient list. This is the recipient list that the user typed in. There’s also a SECOND recipient list, called the P1 recipient list that contains the list of ACTUAL recipients of the message. The P1 recipient list is totally hidden from the user, it's used by the MTA to route email messages to the correct destination server. Internally, the P1 list is kept as the original recipient list, plus all of the users on the destination servers. As a result, the P1 list is significantly larger than the P2 list. For the sake of argument, let’s assume that 10% of the recipients on each message (130) are on each server. So each message had 100 recipients in the P1 header, plus the original DL. Assuming 100 bytes per recipient email address, this bloats each email message by 13K. And this assumes that there are 0 bytes in the message – just the headers involve 13K. So those 15,000,000 email messages collectively consumed 195,000,000,000 bytes of bandwidth. Yes, 195 gigabytes of bandwidth bouncing around between the email servers. Compounding this problem was a bug in the MTA that caused the MTA to crash that occurred only when it received a message with more than 8,000 recipients. But it crashed only AFTER processing up to 8,000 recipients. So 8,000 of the 13,000 recipients of the message would get it and 5,000 wouldn’t. When the MTA was restarted, it would immediately start processing the messages in its queue – and since the messages hadn’t been delivered yet, it would retry to deliver the message, sending to the SAME 8,000 recipients and crashing. And because of the way the Exchange store interacts with the MTA, even if we shut down the MTA, the messages would still queue up waiting on delivery to the MTA –shutting down the MTA wouldn’t fix the problem, it would only defer the problem (since the message store would immediately start delivering the queued messages into the MTA the second the MTA came back up). So what did we do to fix it? Well, the first thing that we did was to fix the MTA. And we tried to scrub the MTA’s message queues. This helped a lot, but there were still millions of copies of this message floating around the system. It took about 2 days of constant work before the email system recovered from this one. When it was over, the team firefighting the crisis had t-shirts made with “I survived Bedlam DL3” on the front and “Me Too! (followed by the email addresses of everyone who had replied)” on the back. To prevent anything like this happening in the future, we added a message recipient limit to Exchange – the server now has the ability to enforce a site-wide limit on the number of recipients in a single email message, which neatly prevents this from being a problem in the future. Larry Osterman320KViews13likes32Comments