Introducing: Log Parser Studio
To download the Log Parser Studio, please see the attachment on this blog post. Anyone who regularly uses Log Parser 2.2 knows just how useful and powerful it can be for obtaining valuable information from IIS (Internet Information Server) and other logs. In addition, adding the power of SQL allows explicit searching of gigabytes of logs returning only the data that is needed while filtering out the noise. The only thing missing is a great graphical user interface (GUI) to function as a front-end to Log Parser and a ‘Query Library’ in order to manage all those great queries and scripts that one builds up over time. Log Parser Studio was created to fulfill this need; by allowing those who use Log Parser 2.2 (and even those who don’t due to lack of an interface) to work faster and more efficiently to get to the data they need with less “fiddling” with scripts and folders full of queries. With Log Parser Studio (LPS for short) we can house all of our queries in a central location. We can edit and create new queries in the ‘Query Editor’ and save them for later. We can search for queries using free text search as well as export and import both libraries and queries in different formats allowing for easy collaboration as well as storing multiple types of separate libraries for different protocols. Processing Logs for Exchange Protocols We all know this very well: processing logs for different Exchange protocols is a time consuming task. In the absence of special purpose tools, it becomes a tedious task for an Exchange Administrator to sift thru those logs and process them using Log Parser (or some other tool), if output format is important. You also need expertise in writing those SQL queries. You can also use special purpose scripts that one can find on the web and then analyze the output to make some sense of out of those lengthy logs. Log Parser Studio is mainly designed for quick and easy processing of different logs for Exchange protocols. Once you launch it, you’ll notice tabs for different Exchange protocols, i.e. Microsoft Exchange ActiveSync (MAS), Exchange Web Services (EWS), Outlook Web App (OWA/HTTP) and others. Under those tabs there are tens of SQL queries written for specific purposes (description and other particulars of a query are also available in the main UI), which can be run by just one click! Let’s get into the specifics of some of the cool features of Log Parser Studio … Query Library and Management Upon launching LPS, the first thing you will see is the Query Library preloaded with queries. This is where we manage all of our queries. The library is always available by clicking on the Library tab. You can load a query for review or execution using several methods. The easiest method is to simply select the query in the list and double-click it. Upon doing so the query will auto-open in its own Query tab. The Query Library is home base for queries. All queries maintained by LPS are stored in this library. There are easy controls to quickly locate desired queries & mark them as favorites for quick access later. Library Recovery The initial library that ships with LPS is embedded in the application and created upon install. If you ever delete, corrupt or lose the library you can easily reset back to the original by using the recover library feature (Options | Recover Library). When recovering the library all existing queries will be deleted. If you have custom/modified queries that you do not want to lose, you should export those first, then after recovering the default set of queries, you can merge them back into LPS. Import/Export Depending on your need, the entire library or subsets of the library can be imported and exported either as the default LPS XML format or as SQL queries. For example, if you have a folder full of Log Parser SQL queries, you can import some or all of them into LPS’s library. Usually, the only thing you will need to do after the import is make a few adjustments. All LPS needs is the base SQL query and to swap out the filename references with ‘[LOGFILEPATH]’ and/or ‘[OUTFILEPATH]’ as discussed in detail in the PDF manual included with the tool (you can access it via LPS | Help | Documentation). Queries Remember that a well-written structured query makes all the difference between a successful query that returns the concise information you need vs. a subpar query which taxes your system, returns much more information than you actually need and in some cases crashes the application. The art of creating great SQL/Log Parser queries is outside the scope of this post, however all of the queries included with LPS have been written to achieve the most concise results while returning the fewest records. Knowing what you want and how to get it with the least number of rows returned is the key! Batch Jobs and Multithreading You’ll find that LPS in combination with Log Parser 2.2 is a very powerful tool. However, if all you could do was run a single query at a time and wait for the results, you probably wouldn’t be making near as much progress as you could be. In lieu of this LPS contains both batch jobs and multithreaded queries. A batch job is simply a collection of predefined queries that can all be executed with the press of a single button. From within the Batch Manager you can remove any single or all queries as well as execute them. You can also execute them by clicking the Run Multiple Queries button or the Execute button in the Batch Manager. Upon execution, LPS will prepare and execute each query in the batch. By default LPS will send ALL queries to Log Parser 2.2 as soon as each is prepared. This is where multithreading works in our favor. For example, if we have 50 queries setup as a batch job and execute the job, we’ll have 50 threads in the background all working with Log Parser simultaneously leaving the user free to work with other queries. As each job finishes the results are passed back to the grid or the CSV output based on the query type. Even in this scenario you can continue to work with other queries, search, modify and execute. As each query completes its thread is retired and its resources freed. These threads are managed very efficiently in the background so there should be no issue running multiple queries at once. Now what if we did want the queries in the batch to run concurrently for performance or other reasons? This functionality is already built-into LPS’s options. Just make the change in LPS | Options | Preferences by checking the ‘Process Batch Queries in Sequence’ checkbox. When checked, the first query in the batch is executed and the next query will not begin until the first one is complete. This process will continue until the last query in the batch has been executed. Automation In conjunction with batch jobs, automation allows unattended scheduled automation of batch jobs. For example we can create a scheduled task that will automatically run a chosen batch job which also operates on a separate set of custom folders. This process requires two components, a folder list file (.FLD) and a batch list file (.XML). We create these ahead of time from within LPS. For more details on how to do that, please refer to the manual. Charts Many queries that return data to the Result Grid can be charted using the built-in charting feature. The basic requirements for charts are the same as Log Parser 2.2, i.e. The first column in the grid may be any data type (string, number etc.) The second column must be some type of number (Integer, Double, Decimal), Strings are not allowed Keep the above requirements in mind when creating your own queries so that you will consciously write the query to include a number for column two. To generate a chart click the chart button after a query has completed. For #2 above, even if you forgot to do so, you can drag any numbered column and drop it in the second column after the fact. This way if you have multiple numbered columns, you can simply drag the one that you’re interested in, into second column and generate different charts from the same data. Again, for more details on charting feature, please refer to the manual. Keyboard Shortcuts/Commands There are multiple keyboard shortcuts built-in to LPS. You can view the list anytime while using LPS by clicking LPS | Help | Keyboard Shortcuts. The currently included shortcuts are as follows: Shortcut What it does CTRL+N Start a new query. CTRL+S Save active query in library or query tab depending on which has focus. CTRL+Q Open library window. CTRL+B Add selected query in library to batch. ALT+B Open Batch Manager. CTRL+B Add the selected queries to batch. CTRL+D Duplicates the current active query to a new tab. CTRL+ALT+E Open the error log if one exists. CTRL+E Export current selected query results to CSV. ALT+F Add selected query in library to the favorites list. CTRL+ALT+L Open the raw Library in the first available text editor. CTRL+F5 Reload the Library from disk. F5 Execute active query. F2 Edit name/description of currently selected query in the Library. F3 Display the list of IIS fields. Supported Input and Output types Log Parser 2.2 has the ability to query multiple types of logs. Since LPS is a work in progress, only the most used types are currently available. Additional input and output types will be added when possible in upcoming versions or updates. Supported Input Types Full support for W3SVC/IIS, CSV, HTTP Error and basic support for all built-in Log Parser 2.2 input formats. In addition, some custom written LPS formats such as Microsoft Exchange specific formats that are not available with the default Log Parser 2.2 install. Supported Output Types CSV and TXT are the currently supported output file types. Log Parser Studio - Quick Start Guide Want to skip all the details & just run some queries right now? Start here … The very first thing Log Parser Studio needs to know is where the log files are, and the default location that you would like any queries that export their results as CSV files to be saved. 1. Setup your default CSV output path: a. Go to LPS | Options | Preferences | Default Output Path. b. Browse to and select the folder you would like to use for exported results. c. Click Apply. d. Any queries that export CSV files will now be saved in this folder. NOTE: If you forget to set this path before you start the CSV files will be saved in %AppData%\Microsoft\Log Parser Studio by default but it is recommended that y ou move this to another location. 2. Tell LPS where the log files are by opening the Log File Manager. If you try to run a query before completing this step LPS will prompt and ask you to set the log path. Upon clicking OK on that prompt, you are presented with the Log File Manager. Click Add Folder to add a folder or Add File to add a single or multiple files. When adding a folder you still must select at least one file so LPS will know which type of log we are working with. When doing so, LPS will automatically turn this into a wildcard (*.xxx) Indicating that all matching logs in the folder will be searched. You can easily tell which folder or files are currently being searched by examining the status bar at the bottom-right of Log Parser Studio. To see the full path, roll your mouse over the status bar. NOTE: LPS and Log Parser handle multiple types of logs and objects that can be queried. It is important to remember that the type of log you are querying must match the query you are performing. In other words, when running a query that expects IIS logs, only IIS logs should be selected in the File Manager. Failure to do this (it’s easy to forget) will result errors or unexpected behavior will be returned when running the query. 3. Choose a query from the library and run it: a. Click the Library tab if it isn’t already selected. b. Choose a query in the list and double-click it. This will open the query in its own tab. c. Click the Run Single Query button to execute the query The query execution will begin in the background. Once the query has completed there are two possible outputs targets; the result grid in the top half of the query tab or a CSV file. Some queries return to the grid while other more memory intensive queries are saved to CSV. As a general rule queries that may return very large result sets are probably best served going to a CSV file for further processing in Excel. Once you have the results there are many features for working with those results. For more details, please refer to the manual. Have fun with Log Parser Studio! & always remember – There’s a query for that! Kary Wall Escalation Engineer Microsoft Exchange SupportDeprecating support for SmartScreen in Outlook and Exchange
What are we announcing? On November 1, 2016, Microsoft will stop generating updates for the SmartScreen spam filters in Exchange Server 2016 and earlier (2013, 2010, 2007), Outlook 2016 for Windows and earlier (2013, 2010, 2007) and Outlook 2011 for Mac. The SmartScreen spam filter will be removed from future versions of Exchange Server and Outlook for Windows. (SmartScreen is not available in any other version of Outlook). This announcement does not affect the SmartScreen Filter online protection features built into Windows, Microsoft Edge and Internet Explorer browsers. While branded similarly, those features are technically distinct. These SmartScreen Filters to help people to stay protected from malicious websites and downloads. What is SmartScreen? What does it provide customers today? In Exchange and Outlook, SmartScreen is a spam content filter. It evaluates each message and returns an overall Spam Confidence Level (SCL). Items that are rated as spam are sent to Outlook’s Junk folder. Microsoft provides periodic updates to the filters, and administrators and users can download and install the updates to improve their junk email protection. For more details, see articles describing how this was done for Exchange Server and Microsoft Outlook. In Windows, Microsoft Edge and Internet Explorer browsers, the SmartScreen Filter online protection feature helps consumers to stay protected from malicious websites and downloads. This feature is not affected and it is not subject of the today’s announcement. Why is Microsoft deprecating support for SmartScreen in Outlook and Exchange? SmartScreen spam filters in Outlook and Exchange Server have become obsolete and have been replaced by Exchange Online Protection (EOP), a more effective cloud-based email filtering service. EOP is built into all Office 365 and Outlook.com accounts and available for purchase to protect on-premises Microsoft Exchange Server. This spam filtering technology was first released in 2003, which provided Outlook and Exchange a content filter able to identify spam campaigns and direct them to the Junk folder. As spammers have evolved and increased the volume and sophistication of their attacks, this type of spam prevention is no longer a useful way to prevent spam. For example, spammers now routinely randomize their campaigns and use reputation hijacking from legitimate sending domains to trick content filters. Spam attacks no longer take days and weeks em; they often complete or significantly morph within minutes. To be effective, filters should be real-time, always tapping into the intelligence of email campaigns happening within recent minutes or hours. Further, SmartScreen often conflicts with EOP (or other 3 rd -party cloud filtering solutions). This is especially painful when emails declared legitimate by upstream filters or administrator policies (e.g. IP Allow Lists, ETR s) are actually junked by SmartScreen, because SmartScreen is unaware of the upstream settings. Microsoft developed Exchange Online Protection to protect Office 365 and Outlook.com mailboxes and remove the need for SmartScreen. Most customers using Exchange Server (on-premises) have either added EOP or use a 3 rd -party filtering service or appliance to sanitize their mail flow. What is Exchange Online Protection (EOP)? Microsoft Exchange Online Protection (EOP) is a cloud-based email filtering service that helps protect end users and organizations against spam and malware, and includes features to safeguard organization from messaging-policy violations. EOP is backed by a modern spam filtering stack, where content filters have a lesser role and sending IP and domain reputation, authentication, campaign detection, and spammer hosting infrastructure reputation are now responsible for filtering. For more details, review the EOP documentation on TechNet and this help article on Office 365 email anti-spam protection. What will happen on November 1, 2016? Microsoft will stop producing new spam definition updates to the SmartScreen filters in Exchange and Outlook. The existing SmartScreen filter and definitions will be left in place, and continue to provide abasic level of protection. The current definition will continue to junk some obvious spam emails, with an effectiveness that will degrade over time. As stated above, most users are protected by superior spam filtering arrangements and should not experience any change in their email experience or see an increase in spam. Again, this change does not affect the SmartScreen Filter online protection feature built into Windows, Microsoft Edge and Internet Explorer browsers. Will the deprecation of SmartScreen have any impact on users using Outlook with Office 365 or Outlook.com? No. Customers using Outlook with Office 365 (for work email) or Outlook.com (for personal email) already have the advanced spam and malware protection found in Exchange Online Protection built into those services. These customers don't need to take any action. Note: Some Office 365 customers may have replaced EOP with a 3 rd -party filtering solution. Those customers will also continue to be protected by those solutions and do not need to take any action. Will the deprecation of SmartScreen have any impact on users using Outlook with Exchange Server (on-premises)? Most likely not. Due to the reasons stated earlier, SmartScreen has stopped being a useful tool for combatting spam. The vast majority of customers using Exchange Server have either added Exchange Online Protection or use a 3 rd party filtering service or appliance to sanitize their mail flow. Customers using Exchange Server should ensure they have their spam protection solution properly configured before November 1, 2016. Customers not using a separate antispam solution today can purchase Exchange Online Protection for $1/user/month. Will the deprecation of SmartScreen have any impact on users using Outlook with Gmail, Yahoo or other online email solutions? Customers using Gmail, Yahoo or other online email solutions will be protected by the spam and malware protection found in those services. How does this deprecation impact Outlook’s Junk Email Options? Outlook’s Junk Email Options stay the same. Since the existing SmartScreen filter and definitions will be left in place, the Options tab will continue to control the SmartScreen protection level. As noted above, those definitions will continue to junk some obvious spam emails, with an effectiveness that will degrade over time. The other tabs are user driven settings and not related to SmartScreen. They will be unaffected by this change (e.g. items in your Safe/Blocked Senders list will still be filtered per your settings). Check out this help article for more on how the Junk Email options work. Will I still have a Junk folder in Outlook? All customers will continue to have a Junk folder. For customers using Outlook with their mailbox in Office 365 or Outlook.com, emails landing in the Junk folder will be determined by Exchange Online Protection (or a 3 rd -party solution) or by Outlook’s Blocked Senders list. Customers using Outlook with other email servers or services will benefit from upstream email filtering (such as EOP or 3 rd party solutions) in their respective environments. These email providers and filters will send items to Outlook’s junk folder. Does this affect the SmartScreen technologies in Windows, Edge and Internet Explorer? As stated earlier in this article, deprecating SmartScreen in Exchange and Outlook does not impact the SmartScreen Filter online protection feature built into Windows, Microsoft Edge and Internet Explorer browsers to protect users from malicious websites and downloads. Those protection tools will remain in place. For more information on how these SmartScreen technologies provide protection, see this link for Windows and this for Edge. The Exchange TeamWindows Management Framework 3.0 on Exchange 2007 and Exchange 2010
Update 9/19/13: A side-by-side coexistence of PowerShell 3.0 and PowerShell 2.0 is supported starting with Exchange 2010 SP3. Note that Exchange 2010 will only use PowerShell 2.0 so PowerShell 2.0 is still a required component. Recently, Windows Update began offering the Windows Management Framework 3.0 as an Optional update. This includes all forms of update distribution, such as Microsoft Update, WSUS , System Center Configuration Manager and other mechanisms. The key bit here is that the Windows Management Framework 3.0 includes PowerShell 3.0. Windows Management Framework 3.0 is being distributed as KB2506146 and KB2506143 (which one is offered depends on which server version you are running - Windows Server 2008 SP2 or Windows Server 2008 R2 SP1). What does that mean to you? Windows Management Framework 3.0 (specifically PowerShell 3.0) is not yet supported on any version of Exchange except Exchange Server 2013, which requires it. If you install Windows Management Framework 3.0 on a server running Exchange 2007 or Exchange 2010, you will encounter problems, such as Exchange update rollups that will not install, or the Exchange Management Shell may not run properly. We have seen Exchange update rollups not installing with the following symptoms: If rollup is installed through Microsoft Update, the installation might error with error code 80070643 If rollup is installed from a download, the error displayed is Setup ended prematurely because of an error. In both cases, Event ID 1024 may be logged in the Application event log with the error error code “1603”. For example, if you try to install update roll 5 for Exchange 2010 SP2, you may see the following description in event ID 1024: Product: Microsoft Exchange Server - Update 'Update Rollup 5-v2 for Exchange Server 2010 Service Pack 2 (KB2785908) 14.2.328.10' could not be installed. Error code 1603. Windows Installer can create logs to help troubleshoot issues with installing software packages. Use the following link for instructions on turning on logging support: http://go.microsoft.com/fwlink/?LinkId=23127 Our guidance at this time is that Windows Management Framework 3.0 should not be deployed on servers running Exchange 2007 or Exchange 2010, or on workstations with the Exchange Management Tools for either version installed. If you have already deployed this update, it should be removed. Once you remove the update, functionality should be restored. Ben Winzenz 12/17/2012: Added details about Event ID 1024.Outlook clients receive error 0x8004010f when downloading the Offline Address Book
EDIT 10/01/2008: To read the updated (and more comprehensive) guidance for troubleshooting errors 0x8004010f, please go here. There are a multiple reasons for why an Outlook client can receive the 0x8004010f sync error. Unfortunately, 0x8004010f is just a generic MAPI error and will show up for a variety of problems. Here is what the error looks like under err.exe (Microsoft Exchange Server Error Code Look-up Tool): C:\WXP\system32>err 0x8004010f # for hex 0x8004010f / decimal -2147221233 ecNotFound ec.h ecAttachNotFound ec.h ecUnknownRecip ec.h ecPropNotExistent ec.h MAPI_E_NOT_FOUND mapicode.h # 5 matches found for "0x8004010f" This is what the error looks like from the sync log from within Outlook: 12:45:53 Synchronizing Mailbox <dgoldman> 12:45:53 Done 12:45:54 Microsoft Exchange offline address book 12:45:54 0x8004010f Some of the most common reasons for Outlook clients to receive the 0x8004010f error with regards to downloading the OAB are listed below. Most of these are documented and I have linked articles to each of these to help everybody out. Please note that these solutions can change a small bit depending on unknown factors in a company's environment. An administrator decommissioned the last Exchange server in a site and never pushes replicas to another Exchange server. A new OAB is created in the active directory and the information store never reads the active directory during its maintenance schedule. This will result in the OAB files never being generated and the Outlook client will fail to download anything. The information store has an invalid EntryID that points to the legacy EX:/ folders. Again there is nothing for the client to download. An outlook client logging in from one domain to another domain and attempting to log in to another users mailbox. The OAB was never generated or some OAB folders are missing from the public folder store. Multiple OAB Version folders exist of the same type. Clients are attempting to download the OAB files from a public folder store that have not received the replicated updates. The offline address book list object has a missing address list. The offline address book list object has an incorrect address list. Send/As changes in the store affect users accounts with no mailbox full rights to another mailbox. If you are seeing this error on an Exchange 2007 server and your OAB is generated by an Exchange 2007 server, please make sure of the following: 1. Make sure that you have added the replicas of OAB to the Exchange 2007 server 2. Make sure public folder replication is working. 3. Make sure the OAB is public folder enabled and you have OAB Version 2, OAB Version 3 and OAB Version 4 checked off so your legacy clients can download the OAB files from the public folder store. 4. Make sure that if you are using an Outlook 2007 client, your OAB is Web Distribution enabled and the OAB files have been replicated over to the Client Access Server. For more information on this process please see this blog: http://blogs.msdn.com/dgoldman/archive/2006/10/23/outlook-client-fails-to-download-the-oab-with-error-0x8004011b.aspx and http://blogs.msdn.com/dgoldman/archive/2006/08/25/How-Exchange-2007-OAB-Files-are-replicated-to-a-Client-Access-Server-for-download.aspx 5. If you are removing your last Exchange 2003 server from the org, please make sure that you follow our documentation on this process. - Dave GoldmanThe Autodiscover Song
Update! The actual song is now available for download as the attachment to this post. Enjoy! So, how exactly do you encourage developers to adopt good development practices when such practices are not explicitly enforced? We thought about embroidered socks and matching hats, but that was universally frowned upon. Whitepapers and technical articles are good options, but in the end it was decided that Autodiscover was too important of a development practice to treat like any other feature in Exchange. It needed something different and unique. It needed.. a song. Yes, a song - to immortalize the virtues of Autodiscover in verse, hoping that in the end, everyone would use Autodiscover in their applications and would rise up in chorus and sing this song together as a unified front. Ok, maybe all developers singing in unison is a stretch. But one day while working on some changes in Autodiscover for the next version of Exchange, the chorus melody and words came to me and wouldn't leave me alone. I started to tinker on the piano and the rest of the song took shape. I didn't think much of it, but after discussing the idea with some other team members, it was decided that I should record and master it and make it available to the Exchange development community. So here it is: Chorus: >>Autodiscover >>There is no other >>Way to decide >>Where your mailbox is stored >>Autodiscover >>Your sister and brother >>Exchange Admin and mother >>Will be proud if you do You may be tempted to wing it Use a hardcoded link submit it But performance will suffer When you're left to your druthers Should have Autodiscovered Then all would be well >>Chorus Call it once for each mailbox of interest Pair the link and mailbox for each request If you encounter errors Refresh once again For more information search M-S-D-N for >>Chorus Exchange won't exist in a bubble If you think that it will you're in trouble Add a site, one or two And your perf will be through Unless you step up and decide that your app will just... >>Chorus Autodiscover, Autodiscover, Just AutoDiscover for me. It's just plain XML David Sterling Senior Software Development Engineer - EWS
