Introducing: Log Parser Studio
To download the Log Parser Studio, please see the attachment on this blog post. Anyone who regularly uses Log Parser 2.2 knows just how useful and powerful it can be for obtaining valuable information from IIS (Internet Information Server) and other logs. In addition, adding the power of SQL allows explicit searching of gigabytes of logs returning only the data that is needed while filtering out the noise. The only thing missing is a great graphical user interface (GUI) to function as a front-end to Log Parser and a ‘Query Library’ in order to manage all those great queries and scripts that one builds up over time. Log Parser Studio was created to fulfill this need; by allowing those who use Log Parser 2.2 (and even those who don’t due to lack of an interface) to work faster and more efficiently to get to the data they need with less “fiddling” with scripts and folders full of queries. With Log Parser Studio (LPS for short) we can house all of our queries in a central location. We can edit and create new queries in the ‘Query Editor’ and save them for later. We can search for queries using free text search as well as export and import both libraries and queries in different formats allowing for easy collaboration as well as storing multiple types of separate libraries for different protocols. Processing Logs for Exchange Protocols We all know this very well: processing logs for different Exchange protocols is a time consuming task. In the absence of special purpose tools, it becomes a tedious task for an Exchange Administrator to sift thru those logs and process them using Log Parser (or some other tool), if output format is important. You also need expertise in writing those SQL queries. You can also use special purpose scripts that one can find on the web and then analyze the output to make some sense of out of those lengthy logs. Log Parser Studio is mainly designed for quick and easy processing of different logs for Exchange protocols. Once you launch it, you’ll notice tabs for different Exchange protocols, i.e. Microsoft Exchange ActiveSync (MAS), Exchange Web Services (EWS), Outlook Web App (OWA/HTTP) and others. Under those tabs there are tens of SQL queries written for specific purposes (description and other particulars of a query are also available in the main UI), which can be run by just one click! Let’s get into the specifics of some of the cool features of Log Parser Studio … Query Library and Management Upon launching LPS, the first thing you will see is the Query Library preloaded with queries. This is where we manage all of our queries. The library is always available by clicking on the Library tab. You can load a query for review or execution using several methods. The easiest method is to simply select the query in the list and double-click it. Upon doing so the query will auto-open in its own Query tab. The Query Library is home base for queries. All queries maintained by LPS are stored in this library. There are easy controls to quickly locate desired queries & mark them as favorites for quick access later. Library Recovery The initial library that ships with LPS is embedded in the application and created upon install. If you ever delete, corrupt or lose the library you can easily reset back to the original by using the recover library feature (Options | Recover Library). When recovering the library all existing queries will be deleted. If you have custom/modified queries that you do not want to lose, you should export those first, then after recovering the default set of queries, you can merge them back into LPS. Import/Export Depending on your need, the entire library or subsets of the library can be imported and exported either as the default LPS XML format or as SQL queries. For example, if you have a folder full of Log Parser SQL queries, you can import some or all of them into LPS’s library. Usually, the only thing you will need to do after the import is make a few adjustments. All LPS needs is the base SQL query and to swap out the filename references with ‘[LOGFILEPATH]’ and/or ‘[OUTFILEPATH]’ as discussed in detail in the PDF manual included with the tool (you can access it via LPS | Help | Documentation). Queries Remember that a well-written structured query makes all the difference between a successful query that returns the concise information you need vs. a subpar query which taxes your system, returns much more information than you actually need and in some cases crashes the application. The art of creating great SQL/Log Parser queries is outside the scope of this post, however all of the queries included with LPS have been written to achieve the most concise results while returning the fewest records. Knowing what you want and how to get it with the least number of rows returned is the key! Batch Jobs and Multithreading You’ll find that LPS in combination with Log Parser 2.2 is a very powerful tool. However, if all you could do was run a single query at a time and wait for the results, you probably wouldn’t be making near as much progress as you could be. In lieu of this LPS contains both batch jobs and multithreaded queries. A batch job is simply a collection of predefined queries that can all be executed with the press of a single button. From within the Batch Manager you can remove any single or all queries as well as execute them. You can also execute them by clicking the Run Multiple Queries button or the Execute button in the Batch Manager. Upon execution, LPS will prepare and execute each query in the batch. By default LPS will send ALL queries to Log Parser 2.2 as soon as each is prepared. This is where multithreading works in our favor. For example, if we have 50 queries setup as a batch job and execute the job, we’ll have 50 threads in the background all working with Log Parser simultaneously leaving the user free to work with other queries. As each job finishes the results are passed back to the grid or the CSV output based on the query type. Even in this scenario you can continue to work with other queries, search, modify and execute. As each query completes its thread is retired and its resources freed. These threads are managed very efficiently in the background so there should be no issue running multiple queries at once. Now what if we did want the queries in the batch to run concurrently for performance or other reasons? This functionality is already built-into LPS’s options. Just make the change in LPS | Options | Preferences by checking the ‘Process Batch Queries in Sequence’ checkbox. When checked, the first query in the batch is executed and the next query will not begin until the first one is complete. This process will continue until the last query in the batch has been executed. Automation In conjunction with batch jobs, automation allows unattended scheduled automation of batch jobs. For example we can create a scheduled task that will automatically run a chosen batch job which also operates on a separate set of custom folders. This process requires two components, a folder list file (.FLD) and a batch list file (.XML). We create these ahead of time from within LPS. For more details on how to do that, please refer to the manual. Charts Many queries that return data to the Result Grid can be charted using the built-in charting feature. The basic requirements for charts are the same as Log Parser 2.2, i.e. The first column in the grid may be any data type (string, number etc.) The second column must be some type of number (Integer, Double, Decimal), Strings are not allowed Keep the above requirements in mind when creating your own queries so that you will consciously write the query to include a number for column two. To generate a chart click the chart button after a query has completed. For #2 above, even if you forgot to do so, you can drag any numbered column and drop it in the second column after the fact. This way if you have multiple numbered columns, you can simply drag the one that you’re interested in, into second column and generate different charts from the same data. Again, for more details on charting feature, please refer to the manual. Keyboard Shortcuts/Commands There are multiple keyboard shortcuts built-in to LPS. You can view the list anytime while using LPS by clicking LPS | Help | Keyboard Shortcuts. The currently included shortcuts are as follows: Shortcut What it does CTRL+N Start a new query. CTRL+S Save active query in library or query tab depending on which has focus. CTRL+Q Open library window. CTRL+B Add selected query in library to batch. ALT+B Open Batch Manager. CTRL+B Add the selected queries to batch. CTRL+D Duplicates the current active query to a new tab. CTRL+ALT+E Open the error log if one exists. CTRL+E Export current selected query results to CSV. ALT+F Add selected query in library to the favorites list. CTRL+ALT+L Open the raw Library in the first available text editor. CTRL+F5 Reload the Library from disk. F5 Execute active query. F2 Edit name/description of currently selected query in the Library. F3 Display the list of IIS fields. Supported Input and Output types Log Parser 2.2 has the ability to query multiple types of logs. Since LPS is a work in progress, only the most used types are currently available. Additional input and output types will be added when possible in upcoming versions or updates. Supported Input Types Full support for W3SVC/IIS, CSV, HTTP Error and basic support for all built-in Log Parser 2.2 input formats. In addition, some custom written LPS formats such as Microsoft Exchange specific formats that are not available with the default Log Parser 2.2 install. Supported Output Types CSV and TXT are the currently supported output file types. Log Parser Studio - Quick Start Guide Want to skip all the details & just run some queries right now? Start here … The very first thing Log Parser Studio needs to know is where the log files are, and the default location that you would like any queries that export their results as CSV files to be saved. 1. Setup your default CSV output path: a. Go to LPS | Options | Preferences | Default Output Path. b. Browse to and select the folder you would like to use for exported results. c. Click Apply. d. Any queries that export CSV files will now be saved in this folder. NOTE: If you forget to set this path before you start the CSV files will be saved in %AppData%\Microsoft\Log Parser Studio by default but it is recommended that y ou move this to another location. 2. Tell LPS where the log files are by opening the Log File Manager. If you try to run a query before completing this step LPS will prompt and ask you to set the log path. Upon clicking OK on that prompt, you are presented with the Log File Manager. Click Add Folder to add a folder or Add File to add a single or multiple files. When adding a folder you still must select at least one file so LPS will know which type of log we are working with. When doing so, LPS will automatically turn this into a wildcard (*.xxx) Indicating that all matching logs in the folder will be searched. You can easily tell which folder or files are currently being searched by examining the status bar at the bottom-right of Log Parser Studio. To see the full path, roll your mouse over the status bar. NOTE: LPS and Log Parser handle multiple types of logs and objects that can be queried. It is important to remember that the type of log you are querying must match the query you are performing. In other words, when running a query that expects IIS logs, only IIS logs should be selected in the File Manager. Failure to do this (it’s easy to forget) will result errors or unexpected behavior will be returned when running the query. 3. Choose a query from the library and run it: a. Click the Library tab if it isn’t already selected. b. Choose a query in the list and double-click it. This will open the query in its own tab. c. Click the Run Single Query button to execute the query The query execution will begin in the background. Once the query has completed there are two possible outputs targets; the result grid in the top half of the query tab or a CSV file. Some queries return to the grid while other more memory intensive queries are saved to CSV. As a general rule queries that may return very large result sets are probably best served going to a CSV file for further processing in Excel. Once you have the results there are many features for working with those results. For more details, please refer to the manual. Have fun with Log Parser Studio! & always remember – There’s a query for that! Kary Wall Escalation Engineer Microsoft Exchange SupportPerformance troubleshooting basics
Since I occasionally troubleshoot performance problems, I thought I’d write up the basics of the process I use. Due to the nature of this topic, this isn’t going to be a comprehensive coverage of all possible performance problems. I just want to focus on the main process, and list a few of the things I look at first to break the problem down. As I see it, there are two main symptoms of a server performance problem: High average rpc latencies or rpc requests Large or rising queues This blog will focus on identifying possible server bottlenecks related to the first symptom, although the steps for rising queues are pretty much the same. In general, the causes of a performance problem fall into two categories: 1. problems due to increased load 2. problems with a resource bottleneck When troubleshooting, I always attempt to differentiate between these two categories of causes first. It is important to note than an increase in load will cause a resource bottleneck, unless the increase is sufficiently small that no symptom of poor performance is observed. Thus, if an increase in load is found, I attempt to identify the cause for the increase in load first, and then identify the resource which is bottlenecked. Resource bottlenecks may occur on any of the following resources: Disk Memory Processor Network (between client and server, or server and any other server) Active directory server Other server resources Increased load can be caused by a change or increase in user activity, or by other applications using server resources. In this blog, I’ll only talk about load caused by MAPI client load, though the principles apply to any type of load. Determining if the server has an RPC performance problem First, find out when and for how long the problem repros. Check to see if you are reproing the symptom now. To verify that the server is exhibiting the reported RPC performance problem, check the following: See if the MSExchangeIS\RPC latency is higher than 50 ms (this is considered a problem) See if the number of outstanding RPC requests (MSEXCHANGEIS\RPC Requests) is higher than 50. Look at the MSEXCHANGEIS\RPC operations per second. Is it higher than you expect? This number depends on the number of users. At Microsoft, we usually see about 0.20 operations per second per user on the server. If the server isn’t exhibiting a performance issue at the time you are investigating, you may not be able find out what is going wrong. Nonetheless, even if these counters appears to be healthy, if users are complaining, I usually continue to investigate anyway – but keep in mind it’s best to investigate while the server is unhealthy. If the server is healthy, I use the information I gather as a baseline to compare against when the server exhibits poor performance. Now you’re ready for the next step: identifying high or increased load. Identify high load After I find out if the problem is reproing, the next thing I look at is the sources of server load. There are many sources of load such as incoming mail rates, MAPI operations, POP3 requests, or even 3 rd party software running on the server. What you look at first will depend on what you know about the system. For a typical back-end server that hosts Outlook users, I look for high rates of RPC Operations per second. If RPC rates haven’t changed, yet the health of the server has declined, I may also look for other sources of increased load, such as the incoming message rates. If I find an increase in load, I attempt to identify the cause. In the case of high RPC load, I look to see if more than 20-30% of the load is due to a single user, or if the load is distributed across many users. I also check if the number of logons per user is excessive (greater than 4 per user) for each database. Finally, I see if the average mailbox size is high on that server, or if individual folders are excessively large (more than 5000 items in a user’s inbox, sent items, deleted items or calendar folder). Large folders and mailbox growth can lead to increased CPU and I/O load. As I mentioned, at Microsoft, our users average 0.2 ops/sec (MSExchangeIS RPC Operations per second divided by the number of mailboxes on the server) at our peak busy time (around 9-11 am). If the whole server is even 10% higher than that for a sustained period, I suspect we’ve had an increase in load. Normally a 10% fluctuation wouldn’t be noticeable, but we keep a history of this value, so I know what is normal for our load profile. For other systems, I have to guess, which usually works fine too. I tend to get very concerned when RPC rates are higher than 0.4 ops/sec per user, though problems can occur at lower rates. If I don’t have a baseline for the unhealthy server, I compare with other servers in the same company that are healthy. Are the per-user rates higher on the server that is having trouble? If you don’t know your load profile, you just have to guess. You can use 0.20 ops/sec as an approximate baseline for active users. If the rates are high, I run Exmon. Use Exmon to determine if a single user is responsible for more than 20% of the server’s RPC load (on a server with more than 200 users) or 40% of the load (on a server with 200 or fewer users). I usually collect 1 minute of data every 5-10 minutes, to look for users that are consistently consuming a lot of CPU. It’s normal for some common operations to cause a user to hog CPU for a short period of time – ignore this. You’re looking for the guy that is at the top of the list most of the time. This can be more of an art than a science at times… use your best judgment. If the RPC rate is high (for a single user or for everyone), find out if users have desktop search, 3 rd party client plug-ins or blackberry devices. Consider investigating these applications as the source of high load, and trying to reduce the load (by removing plug-ins or verifying that they are being used in an optimal fashion). Note: even though I generally sort by the %CPU usage, this doesn’t mean I am expecting a CPU bottleneck. Actually, disk bottlenecks are the most common bottlenecks that Exchange servers encounter. I look at %CPU usage in Exmon because it is fast, and because high CPU usage will usually translate to high I/O. Some people prefer to sort on the Read Pages and PreRead columns as a more precise way to find out which users are causing the most I/O reads, and the Dirty Pages column to find which users are causing the most I/O writes. Identify resource bottlenecks To find the bottleneck, I usually look at most of the performance counters that are described in the “Troubleshooting Exchange Server 2003 Performance” whitepaper, though I always start with CPU and disk. For nearly all cases, I use the thresholds from the whitepaper. Disk bottleneck With disk, I’m mainly looking for read latencies on the databases drives, and write latencies on the storage log. I’m not going to go into all the counters and thresholds because it’s all laid out in the whitepaper. If the latencies are high, or other counters indicate a problem, the server has a disk bottleneck. Processor bottleneck Check if the processor is healthy. Mostly, I check that the CPU is below 80%, and that most of the CPU is coming from the store process (on a back-end machine). If CPU is higher, I know we have a CPU bottleneck. If it’s not coming from store.exe, I find out what process is hogging CPU. Server misconfiguration There are many other things that can impact performance. I always recommend running the ExBPA tool to ensure the server is well configured. No one can remember the thousands of configuration details to check; let ExBPA do it for you. Here are a few other things you may want to check to verify that the server configuration encourages good performance: Are any maintenance tasks still running, or have they run recently? Make sure all maintenance tasks run during non-busy hours. Are the transaction log drives shared with any other resource? Are the database files, temp file, tmp file, SMTP server or system drives shared with any other resource? Is RegTrace enabled? (leaving RegTrace enabled can cause performance issues) Is there less than 10% free disk space on any drive used by the Exchange Server? Is there less than 20MB free on any drive used by the Exchange Server? Hardware problem Occasionally, hardware is unhealthy, and that is cause of a resource bottleneck. You’re just going to have to make that judgment based on the individual circumstances. Are disk latencies high even though the throughput is low? Maybe something just isn’t performing well. If you suspect hardware isn’t living up to spec, swap it out if you can. Resolutions Once I know what’s going on, I can start working on suggested resolutions. I usually get the whole picture before making any changes, because most systems will exhibit many problems simultaneously. It is easy to focus on the first problem that is found, and miss another bigger problem. So, don’t act on these resolutions until you can answer both of these questions: 1) Is the problem caused by increased load? 2) Which resources are my bottlenecks? Resolving performance problems due to increased load If a performance problem is due to increased load, you have a couple options. First, if you have identified the source of the high load, you might be able to reduce the load – perhaps by asking users to install fixes for some of their client applications, or to stop using certain expensive applications. That’s the most obvious, but it’s not always an optoin. Next, you may want to restrict mailbox sizes, and instruct users to archive items out of folders – this also reduces load. Finally, you may decide to spread the load between servers by moving users. For example, if some users have a lot of email-intensive applications, you may want to avoid putting them on the same database or same server. On the other hand, sometimes you may do the opposite – move the extra heavy users to their own server and let them duke it out between themselves for server resources. Either way, the original server, and the rest of the users, are happier. Resolving performance problems due to a bottleneck If you can’t reduce the load on a server, your options are to improve the capacity of the resource that is bottlenecked, modify the configuration of the server when applicable, replace malfunctioning hardware, or move users off to another server. Increasing the capacity of a resource usually means adding more hardware. Sometimes you can increase the capacity by offloading some server work to another server, such as removing optional applications that are running on the server. Resolving a disk bottleneck: If any of the disks are unhealthy, and there hasn’t been an increase in load, I first check to see if the disks are used by anything else (are they shared with another Exchange server, or disk-intensive application like SQL?). Do performance problems only occur when the disks are also being accessed by the other server/program? Exchange doesn’t do well when disks are bottlenecked, and in my experience, disks that are shared often get bottlenecks. The transaction log drives, in particular, should not be shared by any other resource. If disks are unhealthy, you have a few basic choices: move users to another server or to another database hosted on a different drive system, or increase the number of spindles for the current disk array. Resolving a CPU bottleneck: The resolutions for a CPU bottleneck are simple: increase the processor capacity by increasing the number of CPUs or turning on hyper-threading when applicable, move users to another server, or remove any optional applications on the server that are consuming CPU. Resolving a memory bottleneck: If the kernel memory is unhealthy, and the number of logons per user is high, I recommend removing users from the server or, if this is an option, reducing the number of logons per user. You can reduce logons per user by turning off 3 rd party plugins, or reducing the number of client applications per user. The process in a nutshell… There are many more details that I’ve left out of this blog, but I’ve covered the basics. Here is the process in a nutshell: Identify the symptoms. Find out when the problem occurs. Identify if there has been an increase in load. Try to identify the cause of the increase. Reduce the load if you can. Identify the hardware bottlenecks. Check if the bottleneck is due to a misconfiguration. Fix the misconfiguration if present. Check if the bottleneck is due to malfunctioning hardware. Replace the hardware if necessary. Finally, if 2-5 don’t resolve the problem, remove the resource bottleneck Increase the capacity of the bottlenecked hardware or Move users to another server that doesn’t have a bottleneck. Sometimes you will have to iterate through the step 3-6 a few times, as resolving one bottleneck may expose another. And yeah, I hear your pain - a lot of resolutions do involve moving users to other servers. It’s just a fact of life that if the user load increases, you got to move over to make room for it. In this case, it often means more hardware, unless you can convince your users to stop any excessive activity. Unfortunately, as I mentioned at the start of this blog, there are billions and billions of specific details (this is my Carl Sagan impression) that I have left out, but hopefully this has provided a little structure to the troubleshooting-and-problem-resolution process. Finally, I’ll take a moment to plug my latest project – I mean, I’d like to mention that there is a new tool that is designed to take away some of the tedium of troubleshooting performance. Look for the Exchange Performance Troubleshooter Analyzer (EXPTA) 1.0 release in a few months. The tool is based on the same technology as ExBPA, and will walk you through the steps of identifying high load and bottlenecks. - Nicole AllenDeprecating support for SmartScreen in Outlook and Exchange
What are we announcing? On November 1, 2016, Microsoft will stop generating updates for the SmartScreen spam filters in Exchange Server 2016 and earlier (2013, 2010, 2007), Outlook 2016 for Windows and earlier (2013, 2010, 2007) and Outlook 2011 for Mac. The SmartScreen spam filter will be removed from future versions of Exchange Server and Outlook for Windows. (SmartScreen is not available in any other version of Outlook). This announcement does not affect the SmartScreen Filter online protection features built into Windows, Microsoft Edge and Internet Explorer browsers. While branded similarly, those features are technically distinct. These SmartScreen Filters to help people to stay protected from malicious websites and downloads. What is SmartScreen? What does it provide customers today? In Exchange and Outlook, SmartScreen is a spam content filter. It evaluates each message and returns an overall Spam Confidence Level (SCL). Items that are rated as spam are sent to Outlook’s Junk folder. Microsoft provides periodic updates to the filters, and administrators and users can download and install the updates to improve their junk email protection. For more details, see articles describing how this was done for Exchange Server and Microsoft Outlook. In Windows, Microsoft Edge and Internet Explorer browsers, the SmartScreen Filter online protection feature helps consumers to stay protected from malicious websites and downloads. This feature is not affected and it is not subject of the today’s announcement. Why is Microsoft deprecating support for SmartScreen in Outlook and Exchange? SmartScreen spam filters in Outlook and Exchange Server have become obsolete and have been replaced by Exchange Online Protection (EOP), a more effective cloud-based email filtering service. EOP is built into all Office 365 and Outlook.com accounts and available for purchase to protect on-premises Microsoft Exchange Server. This spam filtering technology was first released in 2003, which provided Outlook and Exchange a content filter able to identify spam campaigns and direct them to the Junk folder. As spammers have evolved and increased the volume and sophistication of their attacks, this type of spam prevention is no longer a useful way to prevent spam. For example, spammers now routinely randomize their campaigns and use reputation hijacking from legitimate sending domains to trick content filters. Spam attacks no longer take days and weeks em; they often complete or significantly morph within minutes. To be effective, filters should be real-time, always tapping into the intelligence of email campaigns happening within recent minutes or hours. Further, SmartScreen often conflicts with EOP (or other 3 rd -party cloud filtering solutions). This is especially painful when emails declared legitimate by upstream filters or administrator policies (e.g. IP Allow Lists, ETR s) are actually junked by SmartScreen, because SmartScreen is unaware of the upstream settings. Microsoft developed Exchange Online Protection to protect Office 365 and Outlook.com mailboxes and remove the need for SmartScreen. Most customers using Exchange Server (on-premises) have either added EOP or use a 3 rd -party filtering service or appliance to sanitize their mail flow. What is Exchange Online Protection (EOP)? Microsoft Exchange Online Protection (EOP) is a cloud-based email filtering service that helps protect end users and organizations against spam and malware, and includes features to safeguard organization from messaging-policy violations. EOP is backed by a modern spam filtering stack, where content filters have a lesser role and sending IP and domain reputation, authentication, campaign detection, and spammer hosting infrastructure reputation are now responsible for filtering. For more details, review the EOP documentation on TechNet and this help article on Office 365 email anti-spam protection. What will happen on November 1, 2016? Microsoft will stop producing new spam definition updates to the SmartScreen filters in Exchange and Outlook. The existing SmartScreen filter and definitions will be left in place, and continue to provide a basic level of protection. The current definition will continue to junk some obvious spam emails, with an effectiveness that will degrade over time. As stated above, most users are protected by superior spam filtering arrangements and should not experience any change in their email experience or see an increase in spam. Again, this change does not affect the SmartScreen Filter online protection feature built into Windows, Microsoft Edge and Internet Explorer browsers. Will the deprecation of SmartScreen have any impact on users using Outlook with Office 365 or Outlook.com? No. Customers using Outlook with Office 365 (for work email) or Outlook.com (for personal email) already have the advanced spam and malware protection found in Exchange Online Protection built into those services. These customers don't need to take any action. Note: Some Office 365 customers may have replaced EOP with a 3 rd -party filtering solution. Those customers will also continue to be protected by those solutions and do not need to take any action. Will the deprecation of SmartScreen have any impact on users using Outlook with Exchange Server (on-premises)? Most likely not. Due to the reasons stated earlier, SmartScreen has stopped being a useful tool for combatting spam. The vast majority of customers using Exchange Server have either added Exchange Online Protection or use a 3 rd party filtering service or appliance to sanitize their mail flow. Customers using Exchange Server should ensure they have their spam protection solution properly configured before November 1, 2016. Customers not using a separate antispam solution today can purchase Exchange Online Protection for $1/user/month. Will the deprecation of SmartScreen have any impact on users using Outlook with Gmail, Yahoo or other online email solutions? Customers using Gmail, Yahoo or other online email solutions will be protected by the spam and malware protection found in those services. How does this deprecation impact Outlook’s Junk Email Options? Outlook’s Junk Email Options stay the same. Since the existing SmartScreen filter and definitions will be left in place, the Options tab will continue to control the SmartScreen protection level. As noted above, those definitions will continue to junk some obvious spam emails, with an effectiveness that will degrade over time. The other tabs are user driven settings and not related to SmartScreen. They will be unaffected by this change (e.g. items in your Safe/Blocked Senders list will still be filtered per your settings). Check out this help article for more on how the Junk Email options work. Will I still have a Junk folder in Outlook? All customers will continue to have a Junk folder. For customers using Outlook with their mailbox in Office 365 or Outlook.com, emails landing in the Junk folder will be determined by Exchange Online Protection (or a 3 rd -party solution) or by Outlook’s Blocked Senders list. Customers using Outlook with other email servers or services will benefit from upstream email filtering (such as EOP or 3 rd party solutions) in their respective environments. These email providers and filters will send items to Outlook’s junk folder. Does this affect the SmartScreen technologies in Windows, Edge and Internet Explorer? As stated earlier in this article, deprecating SmartScreen in Exchange and Outlook does not impact the SmartScreen Filter online protection feature built into Windows, Microsoft Edge and Internet Explorer browsers to protect users from malicious websites and downloads. Those protection tools will remain in place. For more information on how these SmartScreen technologies provide protection, see this link for Windows and this for Edge. The Exchange TeamThe Autodiscover Song
Update! The actual song is now available for download as the attachment to this post. Enjoy! So, how exactly do you encourage developers to adopt good development practices when such practices are not explicitly enforced? We thought about embroidered socks and matching hats, but that was universally frowned upon. Whitepapers and technical articles are good options, but in the end it was decided that Autodiscover was too important of a development practice to treat like any other feature in Exchange. It needed something different and unique. It needed.. a song. Yes, a song - to immortalize the virtues of Autodiscover in verse, hoping that in the end, everyone would use Autodiscover in their applications and would rise up in chorus and sing this song together as a unified front. Ok, maybe all developers singing in unison is a stretch. But one day while working on some changes in Autodiscover for the next version of Exchange, the chorus melody and words came to me and wouldn't leave me alone. I started to tinker on the piano and the rest of the song took shape. I didn't think much of it, but after discussing the idea with some other team members, it was decided that I should record and master it and make it available to the Exchange development community. So here it is: Chorus: >>Autodiscover >>There is no other >>Way to decide >>Where your mailbox is stored >>Autodiscover >>Your sister and brother >>Exchange Admin and mother >>Will be proud if you do You may be tempted to wing it Use a hardcoded link submit it But performance will suffer When you're left to your druthers Should have Autodiscovered Then all would be well >>Chorus Call it once for each mailbox of interest Pair the link and mailbox for each request If you encounter errors Refresh once again For more information search M-S-D-N for >>Chorus Exchange won't exist in a bubble If you think that it will you're in trouble Add a site, one or two And your perf will be through Unless you step up and decide that your app will just... >>Chorus Autodiscover, Autodiscover, Just AutoDiscover for me. It's just plain XML David Sterling Senior Software Development Engineer - EWSGenerate some advanced ActiveSync Server usage stats using IIS Logs
As you all would expect, Exchange Server ActiveSync (EAS) requests get recorded in the IIS logs by default at “%windows%\system32\LogFiles\W3SVC1”. What you may not know is that EAS does extend the standard log by adding some custom fields. These fields are provided with the goal of enabling usage analysis. These statistics include a per-folder count of each add/change/delete/fetch generated by the client, and a companion count for each add/change/delete generated by the server. The format of the appended data is as follows: &Log=V[Protocol Version]T[Folder DataType]S[Sync Type]C:#A#C#D#FS:#A#C#DP:#C#I#S#R Let’s see what this mean in plain English. The parameter V indicates the protocol version of the request. It can be one of three values {1, 2, 3, NA}: 1 indicates v1.0 2 indicates v2.0 3 indicates v2.1 NA if unknown/invalid. T indicates the Type of the data being synchronized: Em for E-mail Co for Contacts Ca for Calendar NA for none S indicates the type of sync operation being performed: normal sync, recovery sync, or first sync. Note that this field only applies to the Sync command. All other commands will set this field to N: N indicates this is not a sync command S indicates that this is a normal sync R indicates that this is a recovery sync F indicates that this is a first sync You may wonder what normal sync, recovery sync and first sync mean…this is really another topic altogether but in a nutshell. First sync: This is the first time you synchronize your device with Exchange. Normal sync: This is the standard sync operation that happens every time you manually sync from your device or when schedule synchronization kicks in. Recover sync: This is when a request from a client or a response from the server gets lost (yes…there are still some unreliable networks!). C indicates that the following statistics are from the client. A is number of client adds C is number of client changes D is number of client deletes F is number of client fetches S indicates that the following statistics are from the server A is number of server adds C is number of server changes D is number of server deletes S is number of server soft deletes P indicates that the next few parameters are all related to performance. Note that in the event that no operations are performed, the count will read 0. C is number of connections initiated to Exchange I is number of I/Os to Exchange S is number of bytes sent to Exchange R is number of bytes received from Exchange Now that we understand how the log string can be parsed, let’s look at a real scenario. Here is an example of an entry in the IIS log: 2005-03-08 09:00:08 157.54.6.21 POST /Microsoft-Server-ActiveSync User=johndoe&DeviceId=09310B008B41CF1128000050BF3F5173&DeviceType= PocketPC&Cmd=Sync&&Log=V2TCoSSC:0A0C0D0FS:1A0C0D0SP:1C2I105992S109982R 200 1122 438 657 Microsoft-AirSync/3.0 The part that provides usage statistics comes after &Log: &Log=V2TCoSSC:0A0C0D0FS:1A0C0D0SP:1C2I105992S109982R This indicates that the protocol version was “2.0” (V=2); the syncType was “normal sync” (S=S); and we were syncing a contact folder (T=Co). Next, we start reading what the client was sending to the server: 0 Client Adds, 0 Client Changes, 0 Deletes, and 0 Fetches. This seems to indicate that the client was just doing a regular scheduled sync or the user was manually synchronizing with the server. If we look at the server side than we can see the following: 1 Server Add, 0 Server Changes, 0 Server Deletes and 0 Server Soft Deletes. This means that the user just got a new email! Finally let’s look at the performance data: there was 1 connection initiated to the Exchange server, 2 I/O operations to the Exchange back-end, 105992 bytes sent to the client and 109982 bytes received from the Exchange back-end server. If you write your own script to parse the Log data make sure that you keep the parsing logic flexible because we may add new fields in the next version. The parsing logic should look for and parse out only the fields that it understands. If you take such approach, new fields will get ignored and won’t break the parser. As we move forward we are looking at adding some great ways to generate better reporting on how devices are synchronizing with your Exchange servers. Please let us know what you’d like to see in the next release! Cheers! - Max Ciccotosto and Selva NalliahWhen, if and how do you modify Outlook Providers?
As mentioned in my previous post on the subject of Autodiscover Service and Outlook Providers - I wanted to follow-up with the discussion of modifying the providers. What is the impact when we change the Outlook provider configuration? In what scenarios the Outlook provider should be modified? Relatively few changes will need to be made in the Outlook Providers. Please also note that depending on the modification that was made, Autodiscover might stop working or prevent the client to connect to the Exchange server, so this should not be done lightly! The cmdlet Set-OutlookProvider allows modifying related settings. As we can see in the table below, the parameters Server and CertPrincName only apply to Outlook EXPR provider - Outlook Anywhere clients. By default both values are set to $null. Parameter Optional Description Type Identity N If RPC is used (Outlook Anywhere not selected in GUI), protocol is EXCH. If RPC/HTTP is selected, protocol is EXPR. String Server Y The value here specifies the name of the mail server to use for RPC/HTTP. String CertPrincName Y This value is only used for EXPR types. It specifies the SSL certificate principal name required when connecting externally from the Exchange topology and using SSL. For example, if SERVER were specified as "fourthcoffee.com" and CERTPRINCNAME were left blank the default value of CERTPRINCNAME would be "msstd:fourthcoffee.com". String TTL Y The value here specifies the time to live in hours that these settings are valid for. After that time has elapsed (from the time the settings were retrieved), the settings should be rediscovered via Autodiscover again. A value of 0 indicates that no rediscovery will be required. If no value is specified, the default will be a TTL of 1 hour. Integer We will consider a few scenarios and the impact if you the change Outlook provider configuration: Scenario 1: Multiple AD sites where both CAS servers are Internet-Facing and Outlook Anywhere is enabled. Each Client Access Server has its own certificate installed. The User1 mailbox is located on a Mailbox server on AD site 1, and User2 mailbox on a Mailbox server on AD site 2. Both Outlook clients are on the Internet, thus they will connect through Outlook Anywhere. Note: Autodiscover is configured properly on the Internet as Autodiscover.fourthcoffee.com. When the User1 connect to Autodiscover.fourthcoffee.com server, the Autodiscover service will identify the request comes from an Outlook client and then will return both InternalURLs and ExternalURLs. In this scenario we will explain the importance of not changing Outlook providers. As the parameters Server and CertPrincName are $null. The Service Discovery will return to the client the best CAS for Outlook Anywhere, in this case mail1.fourthcoffee.com. The same behavior will happen when the User2 connects to Autodiscover.fourthcoffee.com. The Service Discovery will return to the client the best CAS for Outlook Anywhere, in this case mail2.fourthcoffee.com. As the parameters Server and CertPrincName are set to $null, they will be populated with the same value as ExternalHostName. Remember that the Outlook provider is a global setting in Active Directory. What would happen if you have modified the parameter Server to mail1.fourthcoffee.com? Set-OutlookProvider EXPR -Server mail1.fourthcoffee.com This setting will force all Outlook Anywhere clients, User1 and User2 to connect to the same CAS server mail1.fourthcoffee.com no matter where the user mailbox is located, preventing the Service Discovery to provide the best CAS. Another issue could result if you decided to change Outlook Anywhere ExternalHostName to Outlookanywhere.fourthcoffee.com. The setting on the EXPR Outlook provider set to mail1.fourthcoffee.com will prevent Outlook Anywhere to connect since the mail1.fourthcoffee.com is not longer available. Scenario 2: Consider the same scenario as the above, however a wildcard certificate was deployed across the Client Access Servers - *.fourthcoffee.com. No change was made to the ExternalUrls and Outlook Anywhere ExternalHostName is set to mail1.fourthcoffee.com. As the parameters Server and CertPrincName are $null. The Service Discovery will return to the client the best CAS for Outlook Anywhere, in this case mail1.fourthcoffee.com, and will configure the Certificate Principal Name to msstd:mail1.fourthcoffee.com. Given that the Certificate Principal Name setting does not match to the wild certificate installed on the CAS, it is required to modify the parameter CertPrincName. Set-OutlookProvider EXPR -CertPrincipalName msstd:*.fouthcoffee.com With this new setting the Service will always return to the Outlook client the CertPrincName set in the EXPR provider. See: When Outlook Anywhere clients connect to Exchange 2007 and a wildcard certificated are deployed across Exchange Client Access servers. Scenario 3: Once the Outlook 2007 client has successfully created a profile, it will update by default every hour according to the parameter TTL set. This configuration can be modified. Set-OutlookProvider -Identity msExchAutoDiscoverConfig -TTL 2 See: Duration that the auto-discovery settings are valid for the Outlook Provider. I hope you have found this useful! - Vandy RodriguesWindows Management Framework 3.0 on Exchange 2007 and Exchange 2010
Update 9/19/13: A side-by-side coexistence of PowerShell 3.0 and PowerShell 2.0 is supported starting with Exchange 2010 SP3. Note that Exchange 2010 will only use PowerShell 2.0 so PowerShell 2.0 is still a required component. Recently, Windows Update began offering the Windows Management Framework 3.0 as an Optional update. This includes all forms of update distribution, such as Microsoft Update, WSUS , System Center Configuration Manager and other mechanisms. The key bit here is that the Windows Management Framework 3.0 includes PowerShell 3.0. Windows Management Framework 3.0 is being distributed as KB2506146 and KB2506143 (which one is offered depends on which server version you are running - Windows Server 2008 SP2 or Windows Server 2008 R2 SP1). What does that mean to you? Windows Management Framework 3.0 (specifically PowerShell 3.0) is not yet supported on any version of Exchange except Exchange Server 2013, which requires it. If you install Windows Management Framework 3.0 on a server running Exchange 2007 or Exchange 2010, you will encounter problems, such as Exchange update rollups that will not install, or the Exchange Management Shell may not run properly. We have seen Exchange update rollups not installing with the following symptoms: If rollup is installed through Microsoft Update, the installation might error with error code 80070643 If rollup is installed from a download, the error displayed is Setup ended prematurely because of an error. In both cases, Event ID 1024 may be logged in the Application event log with the error error code “1603”. For example, if you try to install update roll 5 for Exchange 2010 SP2, you may see the following description in event ID 1024: Product: Microsoft Exchange Server - Update 'Update Rollup 5-v2 for Exchange Server 2010 Service Pack 2 (KB2785908) 14.2.328.10' could not be installed. Error code 1603. Windows Installer can create logs to help troubleshoot issues with installing software packages. Use the following link for instructions on turning on logging support: http://go.microsoft.com/fwlink/?LinkId=23127 Our guidance at this time is that Windows Management Framework 3.0 should not be deployed on servers running Exchange 2007 or Exchange 2010, or on workstations with the Exchange Management Tools for either version installed. If you have already deployed this update, it should be removed. Once you remove the update, functionality should be restored. Ben Winzenz 12/17/2012: Added details about Event ID 1024.Outlook clients receive error 0x8004010f when downloading the Offline Address Book
EDIT 10/01/2008: To read the updated (and more comprehensive) guidance for troubleshooting errors 0x8004010f, please go here. There are a multiple reasons for why an Outlook client can receive the 0x8004010f sync error. Unfortunately, 0x8004010f is just a generic MAPI error and will show up for a variety of problems. Here is what the error looks like under err.exe (Microsoft Exchange Server Error Code Look-up Tool): C:\WXP\system32>err 0x8004010f # for hex 0x8004010f / decimal -2147221233 ecNotFound ec.h ecAttachNotFound ec.h ecUnknownRecip ec.h ecPropNotExistent ec.h MAPI_E_NOT_FOUND mapicode.h # 5 matches found for "0x8004010f" This is what the error looks like from the sync log from within Outlook: 12:45:53 Synchronizing Mailbox <dgoldman> 12:45:53 Done 12:45:54 Microsoft Exchange offline address book 12:45:54 0x8004010f Some of the most common reasons for Outlook clients to receive the 0x8004010f error with regards to downloading the OAB are listed below. Most of these are documented and I have linked articles to each of these to help everybody out. Please note that these solutions can change a small bit depending on unknown factors in a company's environment. An administrator decommissioned the last Exchange server in a site and never pushes replicas to another Exchange server. A new OAB is created in the active directory and the information store never reads the active directory during its maintenance schedule. This will result in the OAB files never being generated and the Outlook client will fail to download anything. The information store has an invalid EntryID that points to the legacy EX:/ folders. Again there is nothing for the client to download. An outlook client logging in from one domain to another domain and attempting to log in to another users mailbox. The OAB was never generated or some OAB folders are missing from the public folder store. Multiple OAB Version folders exist of the same type. Clients are attempting to download the OAB files from a public folder store that have not received the replicated updates. The offline address book list object has a missing address list. The offline address book list object has an incorrect address list. Send/As changes in the store affect users accounts with no mailbox full rights to another mailbox. If you are seeing this error on an Exchange 2007 server and your OAB is generated by an Exchange 2007 server, please make sure of the following: 1. Make sure that you have added the replicas of OAB to the Exchange 2007 server 2. Make sure public folder replication is working. 3. Make sure the OAB is public folder enabled and you have OAB Version 2, OAB Version 3 and OAB Version 4 checked off so your legacy clients can download the OAB files from the public folder store. 4. Make sure that if you are using an Outlook 2007 client, your OAB is Web Distribution enabled and the OAB files have been replicated over to the Client Access Server. For more information on this process please see this blog: http://blogs.msdn.com/dgoldman/archive/2006/10/23/outlook-client-fails-to-download-the-oab-with-error-0x8004011b.aspx and http://blogs.msdn.com/dgoldman/archive/2006/08/25/How-Exchange-2007-OAB-Files-are-replicated-to-a-Client-Access-Server-for-download.aspx 5. If you are removing your last Exchange 2003 server from the org, please make sure that you follow our documentation on this process. - Dave Goldman
