Introducing the Microsoft Exchange Server Profile Analyzer
Included in this month’s Exchange Tools Web release (direct link) is a new analysis tool called the Microsoft Exchange Server Profile Analyzer Tool (or EPA). We created this tool to help server administrators understand their user profile to help with the process of server sizing or capacity planning. A user profile describes how many actions an average user performs in an average day. When we refer to actions, we are talking about things that a user would do within an e-mail client application like send mail, delete mail, browse folder, etc. The EPA tool attempts to use whatever information is available in user mailboxes to generate an estimated profile, and depending on the characteristics of your users, the estimated profile may not be very accurate (for example, if messages are frequently deleted from the Sent Items folder, EPA will not generate accurate sent message statistics). For those of you who have worked with older versions of Exchange and Outlook, the EPA tool is building a profile in a similar way to the storstat.exe tool. This package includes three separate tools: EPA (available as EPAWin.exe and EPACmd.exe) is designed to generate a user profile based on the contents of user mailboxes. EPAOWA is designed to generate a user profile by analyzing OWA log files EPASummarizer is an assistant tool for EPA that can combine output files from EPA to summarize statistics across multiple data collections. This article will focus specifically on the EPA tool. We will explain how it gathers data, what data it gathers, and specific requirements for running the tool successfully. The major steps that EPA takes to collect data are: Step 1: Load topology from Active Directory. EPA reads Exchange configuration data such as organization, server, storage group, private mailbox store, and http virtual server from AD. This means that the account that runs EPA is required to have the ability to read Exchange configuration data from AD. In the GUI version of EPA (EPAWin.exe), you will see a tree view of the topology once this step is complete. Step 2: Scan HTTP virtual server and virtual directory configuration for each server. For each http virtual server found on the server, EPA reads configuration information from the IIS metabase for two attributes: running state and SSL accessibility. For each virtual server, EPA maintains a list of mailbox virtual directory access links. The format of the link looks like this: http(s)://servername(:portnumber)/virtualdirectoryname/. For a particular virtual server: If EPA detects its state as not running, EPA will ignore it. If EPA fails to read the running state, EPA assumes the virtual server is running. If EPA reads that SSL is not enabled, for each mailbox virtual directory found on the virtual server, EPA adds a corresponding “http://” link to the list If EPA reads that SSL is enabled, for each mailbox virtual directory found on the virtual server, EPA adds a corresponding “https://” link to the list If EPA fails to read the SSL attribute, EPA adds two entries of this virtual server to the list. One assumes the SSL is enabled (https://), and the other does not (http://). At the end, EPA sorts the list to put “https://” links on the top so that we will always attempt to use encrypted connections if they are available. When EPA moves on to Step 4 to collect data from each user mailbox, it will attempt to use each link in the list until it finds one that works. EPA will attempt to match the configured SMTP domain of a virtual server with each user’s SMTP domains to facilitate hosted Exchange scenarios. You may notice that this step would require the account that runs EPA to have access to IIS metabase. If you don’t have access to the IIS metabase, EPA will make some assumptions about your configuration that should allow it to run under most circumstances, however you may see some error messages reported in the log if metabase access is not available. Step 3: For each server and mailbox store we found in the topology, EPA determines whether or not they will be included in the data collection stage based on configuration input, Step 4: Data collection. Based on user’s configuration of the ServerThread and MailboxThreadPerServer attributes in the configuration file, EPA creates single/multiple working threads for data collection. If ServerThread is configured to 1 (which is the default setting), EPA will collect data sequentially on servers. Otherwise, it will create multiple threads for servers, and starts to collect data from multiple servers at the same time. Similar logic applies to MailboxThreadPerServer. The mailbox thread is the actual thread that is doing the data collection from a single mailbox (see the section below for the details of this function). After each mailbox is processed, the statistics generated from the collected data will be summarized to the parent mailbox store. The collection process for a mailbox store is done when all the mailboxes in the store are processed. The statistics for the server level are calculated when the collections for all the mailbox stores on the server are done. Similarly, the statistics at the organization level will be calculated when the collections on all the servers are finished. How EPA collects data from single mailbox EPA accesses items in user’s mailbox by using HTTP/Web Distributed Authoring and Versioning (WebDAV). This requires that the account that runs EPA has to have Full Mailbox Access. Note that EPA only supports Integrated Windows Authentication (NTLM) authentication at this time. If your server does not have Integrated Windows Authentication enabled for HTTP virtual servers, EPA will fail to collect data. EPA will also fail if HTTP virtual servers on back-end mailbox servers have OWA FBA (Forms-Based Authentication) enabled. The primary WebDAV methods that EPA uses are SEARCH, PROPFIND, BPROPFIND and X-MS-ENUMATTS. The SEARCH method is used to get the hierarchy table and content table from a folder. To get the actual properties of a folder or of a message item, EPA uses the PROPFIND and BPROPFIND methods. X-MS-ENUMATTS is used to get the attachment table from a message item. Table 1 shows the sets of properties that EPA reads for different types of store objects such as folder, message, attachment and appointment using WebDAV. Table 2 shows a list of collectors EPA calculates based on the properties listed in Table 1. Table 1: Properties EPA reads by using WebDAV Property Name Namespace Root Mailbox Folder contacts urn:schemas:httpmail: calendar urn:schemas:httpmail: deleteditems urn:schemas:httpmail: drafts urn:schemas:httpmail: inbox urn:schemas:httpmail: outbox urn:schemas:httpmail: sentitems urn:schemas:httpmail: tasks urn:schemas:httpmail: notes urn:schemas:httpmail: journal urn:schemas:httpmail: MsgFolderRootURI urn:schemas:httpmail: Folder Properties href DAV: displayname DAV: childcount DAV: objectcount DAV: haschildren DAV: hassubs DAV: creationdate DAV: foldersize http://schemas.microsoft.com/exchange/ 0x0FFF0102 (EntryID) http://schemas.microsoft.com/mapi/proptag/ 0x0e080003 (Size) http://schemas.microsoft.com/mapi/proptag/ 0x36010003 (FolderType) http://schemas.microsoft.com/mapi/proptag/ 0x663A000B (HasRules) http://schemas.microsoft.com/mapi/proptag/ Message Properties href DAV: creationdate DAV: contentclass DAV: subject urn:schemas:mailheader: to urn:schemas:mailheader: cc urn:schemas:mailheader: bcc urn:schemas:mailheader: hasattachment urn:schemas:httpmail: read urn:schemas:httpmail: htmldescription urn:schemas:httpmail outlookmessageclass http://schemas.microsoft.com/exchange/ 0x0e060040 (DeliveryTime) http://schemas.microsoft.com/mapi/proptag/ 0x0e080003 (Size) http://schemas.microsoft.com/mapi/proptag/ Attachment Properties 0x3703001f (Extension) http://schemas.microsoft.com/mapi/proptag/ 0x0e200003 (AttachmentSize) http://schemas.microsoft.com/mapi/proptag/ 0x3704001f (AttachmentName) http://schemas.microsoft.com/mapi/proptag/ Appointment Properties dtstart urn:schemas:calendar: Note: Please refer to http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wss/wss/_webdav_x-ms-enumatts.asp for a complete list of supported WebDAV properties and detailed descriptions of the data containined within the properties listed above. Table 2: Exchange Server Profile Analyzer Data Collectors Collector Name Description MailboxTotalSize The total size, in bytes, of the mailbox RulesTotalCount Total number of rules defined in the mailbox FolderTotalCount Total number of visible folders in the mailbox FolderMaxMessageCount Maximum number of messages in any one folder FolderTopLevelCount Number of visible folders in the mailbox FolderUserCreatedTopLevelCount Number of user created folders that are direct children of the root of the mailbox SearchFolderCount Number of search folders in the current mailbox FolderHierarchyHeight The height of the folder tree FolderTopLevelAverageSubfolders Average number of children each child of the folder tree root has. FolderTopLevelAverageHeight Average height of each of the root folder’s children's subtrees. FolderSize Size in bytes of all the messages in a folder (takes list of folders to measure, e.g. "Inbox, Deleted Items, Sent Items") FolderSizeAggregates Folder size statistics (across all folders in mailbox) - provides Avg, Min, Max. MessageMailboxCount Number of messages in the mailbox MessageFolderCount Number of messages in a folder (takes a list of folders to measure, e.g. ("Inbox","Deleted Items","Sent Items") MessageUnreadCount Number of messages that are unread MessageDAMCount Number of Deferred Action Messages in the mailbox MessageReplyCount Number of messages where the subject prefix is "RE:" or equivalent subject prefix for the given culture MessageForwardCount Number of messages where the subject prefix is "FW:" or equivalent subject prefix for the given culture MessageContainsAtLeastOneDLCount Number of messages containing at least 1 distribution list in the recipients table MessageContainsAtLeastOneAttachmentCount Number of messages containing at least 1 attachment MessageSizeDistribution Counts messages in a size range (takes a list of ranges, e.g. "2,10,100,1024" would provide counts of messages from 0-2,2-10,10-100,100-1024,1024-beyond) MessageSizeAggregates Message size statistics across all messages (provides Avg,Min,Max) MessageReceivedPerDayAggregates Average number of messages received per day (provides Avg,Min,Max, and can restrict to the last N days) MessageSentPerDayAggregates Average number of rows in each sub table when the sent items folder is categorized by date (provides Avg,Min,Max and can restrict to last N days) MessageRepliesSentPerDayAggregate Average number of messages prefixed with "RE:" sent per day (provides Avg,Min,Max and can restrict to last N days) MessageForwardsSentPerDayAggregates Average number of messages prefixed with "FW:" sent per day (provides Avg,Min,Max and can restrict to last N days) MessageBodyTypesCount Number of messages in each body type requested (takes list of body types as input, e.g. "RTF,HTML,Other") RecipientsPerMessageSentAggregates Average number of recipients of each message in the Sent Items folder RecipientsDLPerMessageSentAggregates Average number of distribution list recipients of each message in the Sent Items folder AttachmentSizeAggregates Attachment size statistics across all attachments (provides Avg,Min,Max) AttachmentSizeDistribution Counts attachments in specified size ranges (takes a range list of inputs, e.g. "2,10,100,1024" provides counts for ranges 0-2,2-10,10-100,100-1024,1024-up) AttachmentPerMessageAggregates Statistics on number of attachments per message (provides Avg,Min,Max) ContactCount Number of contacts in the mailbox ContactCreatedPerDay Number of contacts created per day (provides Avg,Min,Max, can be restricted to last N days) AppointmentCount Number of appointments in the calendar AppointmentCreatedPerDay Number of appointments created per day (provides Avg,Min,Max, can be restricted to last N days) MeetingRequestCount Number of meeting requests in the calendar. MeetingRequestReceivedPerDayAggregates Statistics on number of meeting requestes received per day (provides Avg,Min,Max, can be restricted to last N days) We want to hear from you! Feel free to send your feedback on this tool directly to epafb AT microsoft DOT com and we will use your input to help make future versions of this tool even better. Frequently Asked Questions Question: EPA reports CompletedWithException. Where can I find what exceptions occurred during the data collection process? Answer: EPA reports exception information exceptions in a log file. The default log file will be located at the user's application data path. For example, C:\Documents and Settings\<username>\Application Data\Microsoft\Epa\epalogYYYYMMDD_HHMMSS.log Question: Error: Unable to connect to Active Directory. Please make sure that the user account has enough permission. Error: Unable to check the state of HTTP virtual server 1 on ServerName. Error: Unknown error (0x80005000) Answer: The errors are reported because the account that runs EPA does not have permission to read IIS metabase. The error code may vary. EPA will continue to attempt to collect data from user mailboxes by making some assumptions about the configuration of your Exchange topology. See "Step 2" described above for details. Question: Error: Unable to find an available URI for user ServerName\MDBName\Mailbox x. Answer: The error is reported for two reasons normally. 1. No HTTP virtual server is running. 2. For each HTTP virtual server on the exchange server, SMTP domain of the virtual server does not match with user's SMTP domains. Question: Error: User ServerName\MDBName\Mailbox x cannot access any of the following links:http://ServerName/Exchange/. Answer: The error will come up in the following five cases. 1. The account that runs EPA does not have full mailbox access on the user's mailbox. This can be verified through Exchange System Manager. Please see the EPA documentation for details on how to configure and verify permissions. If the account does have full mailbox access rights, then try using Internet Explorer to access the user's mailbox with OWA. If you are not able to access the mailbox with OWA, EPA will not be able to access it either. 2. The authentication methods configured on the virtual directory do not include Integrated Windows Authentication which is the only method EPA supports. 3. There are restrictions set up on the Exchange Server that restrict TCP/IP traffic in some way so that HTTP or HTTPS traffic from the machine you are running EPA on is blocked. You can work around this by running EPA on a machine that has the ability to connect with the back-end Exchange Server (such as an Exchange front-end server). 4. When SSL is required on the server and the name on the SSL certificate does not match the name that EPA is using to access the server, EPA will fail due to SSL certificate validation errors. We will have a fix for this in next Web Release. 5. When OWA Forms-Based Authentication is enabled on the HTTP virtual server, EPA will fail since the DAV requests submitted by EPA will get "440 Login Timeout" error. This is similar to http://support.microsoft.com/default.aspx?scid=kb;en-us;817379 . We plan to provide better feedback for this case in the next web release version. If these troubleshooting suggestions don't solve your EPA problems, please contact epafb AT microsoft DOT com for further investigation. - Jessie ZhuWhy is OOF an OOF and not an OOO?
Here's an interesting historical question - when we say Out of Office, why does it sometimes get shortened to ‘OOF’? Shouldn’t it be ‘OOO’? Inside Microsoft, ‘OOF’ means not just the message which says you’re Out of Office, but it has grown to mean the act of being Out of the Office too - so you’ll get people putting sticky notes on their door saying ‘OOF Thurs & Fri’ or even people verbally saying things like, "Oh, Kevin’s OOF on vacation for the rest of the week’. I suppose that sounds better than "Oh, Kevin’s OOO on vacation ..." OOF was a command used in the days of Microsoft’s Xenix mail system, which set a user as ‘Out of Facility’ - ie Out of the Office. The usage of the term ‘OOF’ just stuck, as did the term ‘Little r’ (e.g. on an email sent to a distribution list, "Who wants to go to the cinema tonight? Little ‘r’ if you’re interested", meaning reply just to me) - as preserved in Outlook with CTRL+R for Reply, and CTRL+SHIFT+R (aka Big R) for Reply All. Ewan DaltonExchange is 10 years old!
Seems like yesterday that Exchange 4.0 hit the market. In March of 1996 Exchange 4.0 was released. I thought I would give you a link for the geeks among you that have been along for the ride and for those that go even further back to ALL-IN-ONE, MailWorks, cc:Mail, IBM PROFS etc. ITPRO has a great look back if you're interested in taking a look back at the evolution of hardware, Exchange Clients, Mobile devices, OWA, etc. http://www.windowsitpro.com/Common/adforceimages/Decade_of_exchange.pdf It's been a whirl wind ride and now we are nearing Beta 2 of Exchange "12"! What will Exchange look like 10 years from now? What features would you want to see (hey why wait - ask for them now!). If Exchange 2016 was ready for release, what features would you be most excited by? So, take a look back, reflect on where we have been, then put on your thinking caps and tell us what you want to see in the next release and the one after that! - David EspinozaThe SDE career path at Microsoft
In Microsoft, there are a number of standard job titles. The most important one is SDE – Software Development Engineer. The rest can be considered the supporting cast (one requirement of being an SDE is to be so supremely confident that such a statement as the previous one can be made without actually laughing out loud while doing so). The life of an SDE at Microsoft is as close to heaven as a computer geek can get. Chocolate donuts for breakfast, a quick bite in the cafeteria for lunch, any fast food at all for dinner, and before, in between and after is glorious coding. Of course, other things intervene occasionally – like annoying testers who keep insisting the code actually works, or program managers who insist the code actually be useful to someone (honestly, I don’t know where they find these people), or bosses who insist on “talking”, sometimes even in the form of meetings. These are unfortunately necessary evils and an SDE will tolerate them and, on occasion, even indulge them from time to time. There is a cloud on the horizon in many SDE’s careers however, and it’s called management. At Microsoft, there aren’t a whole lot of real managers, they are mostly technical people who at one time or another did real work and then – probably due to chemical imbalances in the brain brought on by a diet of donuts, cafeteria and fast food – suddenly “stepped up” to the management plate. It’s pretty insidious. One day you’re a happy go lucky developer enjoying work to its legal limit and with masses of extremely marketable skills, and then your boss suggests that maybe you should take on a report or two. Just junior people, you understand, to see how you like it. Zap. Six months later you have a team of 15 and you can’t write a line of code to save your life anymore (but you can create a mean Powerpoint presentation). Microsoft does have an alternative career path for SDE’s (and the other, lesser, disciplines) which enables an IC (individual contributor) to remain an IC but continue to progress in that role. While the ultimate destination of a manager is a VP, the equivalent IC role for an SDE is a DE (Distinguished Engineer). DE is quite a bit harder to attain then VP, actually. An intermediate IC role is usually some kind of architect. I know some people who have followed this path successfully, but it is a bit of a harder road to hoe. In my case, I’ve taken the Bilbo Baggins approach (“There and Back Again”). Prior to 1998, I had spent 13 years as an IC (not all at Microsoft, although I wish I had joined the company in ’85). I then took on two reports. Then six. Then, after Exchange 2000 shipped, I took on a team of fifteen. Then suddenly I had seventy developers working for me. I even attained the exalted level of PUM (Product Unit Manager). Those were the dark times. I forgot how to code. I forgot how to debug. I pretty much forgot how to think. I finally snapped out of it and ever since I’ve been striving once more for IC status. I currently am down to a team of five, but I’m not likely to get much lower than that any time soon. No matter. I’m coding again and once more feel like a contributing member of the team rather than a roadblock on the way to success. And those chocolate donuts are yummy. - Jon AvnerOutlook REST API beta and Outlook REST API v2.0 Deprecation Notice
Today we are announcing the deprecation of the Outlook REST API beta and Outlook REST API v2.0 and that they will be decommissioned on November 30, 2022. Once past this date, the services will be retired, and developers may no longer access them.How does your Exchange garden grow?
One question we often get asked when talking to customers contemplating an Exchange upgrade or a switch from a competitor’s mail system is, "how many users per server can Exchange handle?" Nowadays, that’s an open question - it very much depends on what kind of users you have, what kind of storage you’re using and how powerful your servers are. When Exchange 4.0 was released in 1996, a decent server might have had 256Mb of RAM and a 90MHz Pentium processor, with maybe a handful of GBs of SCSI disk in the box and possibly a DAT tape drive. Users’ mailboxes might have been in the 10-20Mb size range, and the average user sent or received only a small amount of email per day. At that time, server sizing was pretty much a function of how much computing horsepower you could afford - the CPU power, disk size & speed and memory capacity available (along with the all-important user profile) would determine the number of users per server, and arbitrary decisions would be made about maximum message sizes, mailbox size etc. Now, it’s possible to buy even mid-range servers that will cope with many thousands of users, and the bottleneck has moved down to the storage level in many instances as user mailboxes have grown in size and we send and receive far more mail, and many larger messages too. We’d be interested in hearing anecdotes about particularly unusual messaging environments you might have - how large do your servers grow? What’s the biggest mailbox you’ve ever seen (assuming you’re not restricting with quotas)? How much data does your total Exchange system manage? What’s the biggest single database - etc, etc... you get the picture. Some factoids to get you started... if you have anything along these lines, please post a comment: - One customer (who is part of the Exchange Technology Adoption Program) sets no limit on message sizes internally - and one time saw a single message with attachments totaling 2.4Gb in size! (which Exchange delivered! - the recipient’s mailbox quota was instantly blown apart though...) - The largest single Exchange 5.5 Public Folder Information Store I’ve come across was 800Gb in size - There’s a limit of 1,000 Exchange 2000/2003 servers in an org - has anyone come close, or hit that limit? - One customer deploys around 250,000 Exchange users per server and has over 9,000,000 users in total (you guessed it, they’re an ISP) - The most heavy user of public folders in the world (that we know about) has about 1,500,000 public folders containing around 8Tb of data So come on: let’s hear your war stories - post a comment with your amazing data :) - Ewan DaltonMicrosoft Ignite 2020 - One Week Away!
In about a week from today Microsoft Ignite 2020 gets underway. It’s quite a bit different this year but the one thing you can be certain of is we have lots of new and interesting content for you to enjoy. This post is to highlight the Exchange, Outlook and Bookings sessions we have created and curated for your viewing pleasure. The links won’t be live until the event starts, but we wanted to give you a peek into what to expect.Exchange and SameSite Updates
The Stable release of the Google Chrome web browser scheduled for release beginning February 17, 2020 features a change in how cookies are handled. Microsoft is committed to addressing this change in behavior in its products and services before the February 4, 2020 rollout date to ensure our customers are minimally impacted.
