Upcoming changes to Exchange Web Services (EWS) API for Office 365
Update: For latest information related to basic authentication in Exchange Online, please see Basic Authentication and Exchange Online – September 2022 Update. Over the last few years, we have been investing in services that help developers access information in Office 365 in a simple and intuitive way, specifically through Microsoft Graph. Microsoft Graph and the use of OAuth 2.0 provide increased security and seamless integration with other Microsoft cloud services and is rapidly expanding developer access to the rich data sets behind Microsoft applications. As we make progress on this journey, we have continued to evaluate the role of Exchange Web Services (EWS). Today we are sharing our plans to move away from Basic Authentication access for EWS over the next two years, with support ending Oct. 13, 2020. These plans apply only to the cloud-based Office 365/Exchange Online products; there are no changes to EWS capabilities of on-premises Exchange products. Exchange Web Services will not receive feature updates Starting today, Exchange Web Services (EWS) will no longer receive feature updates. While the service will continue to receive security updates and certain non-security updates, product design and features will remain unchanged. This change also applies to the EWS SDKs for Java and .NET as well. While we are no longer actively investing in it, EWS will still be available and supported for use in production environments. However, we strongly suggest migrating to Microsoft Graph to access Exchange Online data and gain access to the latest features and functionality. For more information and details on how to make the transition, please refer to the following articles: Overview of Microsoft Graph Overview of Outlook mail API on Microsoft Graph While EWS and Graph have mostly overlapping functionality, there are some differences. If you rely on an EWS API that does not have a Graph counterpart, please let us know via UserVoice of features needed for your app scenarios. Basic Authentication for EWS will be decommissioned Exchange Web Services (EWS) was launched with support for Basic Authentication. Over time, we've introduced OAuth 2.0 for authentication and authorization, which is a more secure and reliable way than Basic Authentication to access data. Please refer to the following article for more information: Getting started with OAuth2 for Microsoft Graph Today, we are announcing that on October 13th, 2020 we will stop supporting and fully decommission the Basic Authentication for EWS to access Exchange Online. This means that new or existing apps will not be able to use Basic Authentication when connecting to Exchange using EWS. Next Steps The deprecation of these APIs follows our service deprecation policies. We understand changes like this may cause some inconvenience, but we are confident it will ensure more secure, reliable, and performant experiences for our customers. We're here to help if you need it. If you have questions, please let us know in Stack Overflow with the [MicrosoftGraph] tag. Thank you in advance for updating and opening your apps to a wider range of useful and intelligent features on Microsoft Graph. We are extremely excited about the growing opportunities that Microsoft Graph offers to our customers, and we remain fully committed to continue our journey to empower developers to access Office 365 data with the most modern features and tools. Frequently Asked Questions Q: Will my application stop working when you make this change? A: It might, yes, it depends on the app itself and how it was coded. If it’s using EWS, and if it’s using Basic authentication then yes, on October 13th 2020 it will fail to connect. However, if the app is using Modern Auth/OAuth, then no, it will keep working as it did before. Q: Why October 13th 2020? Why that date? A: Starting October 13, 2020, Office 365 ProPlus or Office perpetual in mainstream support will be required to connect to Office 365 services. This announcement is posted here Office 365 ProPlus Updates. This change requires that Office 2013/Office 2016 are also required to use Modern Auth. Please see this. Q: Our in-house team created an app for meeting room scheduling, how do we go about changing that over to Graph and OAuth2.0? A: Don’t forget you can keep using EWS if you want to, so then really, it’s just the question of authentication. To get a better understanding of how to use OAuth 2.0 take a look here. Q: How does this impact my On-Premises Exchange deployment? A: It does not. This change only affects Exchange Online. Q: We require Modern Authentication + Multi Factor Auth for all our Outlook users connecting to O365, how do apps work when I have that set as a requirement? A: Applications can be written so they are treated as ‘trusted applications’. That way they can bypass the MFA requirement, more details are here. Q: How do I know which of my apps use Basic authentication to EWS? A: If you only use Outlook to connect to Exchange Online then you don’t need to worry, as long as you are using Office 2019 or Office 2019 Pro Plus you’ll be fine come October 2020. However, if you also have integrated apps into your Office 365 tenant you’ll need to check with the application developers to verify how it authenticates to Exchange Online if you aren’t’ sure. We are investigating how we can share this information with tenant admins, but have nothing available at the time of writing this blog. Q: What features does EWS have that Graph can’t provide? A: Graph is constantly evolving and adding features and functionality to provide the richest set of experiences we can. To see the latest features we’ve added to Graph, go here Overview of Outlook mail API on Microsoft Graph Q: Will this affect my Exchange Hybrid configuration? Exchange On-Premises calls into Exchange Online using EWS doesn’t it? A: Yes, it does. But it doesn’t use Basic Authentication, it uses token-based authentication, and it’s described in this blog post. How Hybrid Authentication Really Works. The Exchange TeamRemoving "Reply All" Functionality for Outlook Users Who Participate in Reply-All Storms, via Group Policy: what to do
Just recently I was speaking to a Microsoft Premier Field Engineer (PFE) who had just returned home from a rather arduous onsite dispatch. While onsite he was tasked with unwinding a phenomenon commonly referred to as a "Reply-All" storm (defined below). This task had been so tedious that he half-jokingly mentioned that he was thinking about submitting a formal Design Change Request to the Exchange/Outlook Product Groups with a proposal to remove Reply-All functionality from within Outlook. Well this got me thinking... "could the removal of Reply-All functionality actually be done with the tools at an Administrator's immediate disposal today"? Eventually curiosity got the best of me and I started performing some basic research on the issue. Throughout the course of this research I found a jumbled set of different articles and blog posts that appeared rather half baked in scope to me. That being said, what did become abundantly clear was that there were a significant number of administrators (as well as standard users) who have expressed interest in this functionality and have actually been looking for it for some time. Presto: Blog idea. To be clear there are a number of different approaches for achieving the goal of disabling Reply-All in Outlook. The most common technique would appear to be the creation and deployment of custom Outlook toolbars. Far more infrequent are macros in Outlook as well as custom Group Policy Objects (the primary topic of this post). Additionally, while future versions of Exchange will include features to deal with this problem much better (read about Exchange 2010 Conversation View here) - I wanted something that people can use now, with versions of software they currently have. As previously noted, this post is going to focus on the use and deployment of a custom Group Policy to disable the Reply-All functionality in Outlook. This post will not focus on disabling Reply-All in OWA (which would require modifications to the ASP Pages) and will also not cover disabling this functionality in Windows/Outlook Mobile. My hope is that you will find this post interesting; that it helps clear up some of the confusion on the subject; and that you learn some new skills along the way. Should and when you choose to deploy a scenario like this, I hope this document serves to aid you in increasing the stability of your messaging environments and networks as a whole. As you will soon see, this post has a little bit of everything, Exchange, Active Directory, Group Policies, Outlook and even some "old/new school" VBA. I hope you enjoy it and learn something new in the process! Back Story: Nearly every user within a corporate messaging environment has at one time or another been the unfortunate chance victim of a "Reply-All Storm". It starts out innocently enough: Typically a formal communication it sent out to a large internal distribution list. A member of the distribution list strays off course or asks to be "taken off the distribution list". A series of people start replying "ME TOO" via a Reply-All. Another member of the list, "Replies All" to the request, and asks everyone to "PLEASE DO NOT REPLY ALL !!!!@!!@!!!". Cycle perpetuates itself over and over. Users come into the office and notice literally hundreds of emails all with the same subject sitting in their inbox. They read the most recent thread, whisper to themselves (geez...not this again), then proceed to delete the entire thread and possibly miss the actual intention of why the original mail was sent out in the first place. Not only is this extremely annoying, but obviously has larger corporate and financial implications as well: Loss of productivity for client base. Potential service interruptions for a variety of different reasons. Storage implications on the Exchange Back-End. Puts unnecessary strain on Exchange, Active Directory and network links. A host of Administrative implications ranging from increased backup times, etc. Loss of credibility (if say an external contact was a member of the list or even your immediate manager or "higher ups", etc.) Upon first read this may seem funny but it is actually a very serious problem. For a blow by blow of the infamous "BEDLAM DL3" issue that occurred internally here at Microsoft several years ago I suggest a review of the following post: http://msexchangeteam.com/archive/2004/04/08/109626.aspx. The issue is definitely real and varying degrees of the problem are most likely occurring somewhere in the world at this very second. Before Getting Started: I want to stress that the steps outlined below are not going to resolve an immediate Reply-All storm. However, it is very possible that they may help reduce or mitigate a future problem from occurring once your user base learns (either by word of mouth or "the hard way") that there are potential consequences for perpetuating one. Again, I'm talking about helping to bring about positive educational change in your user base over time. The steps outlined below will effectively remove a user's ability to use the Reply-All functionality in Outlook via creation and deployment of a custom Group Policy Object (GPO). The examples presented below will target a dedicated Active Directory Organizational Unit. This OU will ultimately become the target for the deployment of the GPO as well as the container where user objects that have Reply-All removed will be homed in the Active Directory. That being said, the techniques presented here could be adapted to a broader environmental context based upon the discretion/desire of the administrator. Thus to perform the steps here you will need at your immediate disposal: Access to the Active Directory. The ability to create a dedicated Organizational Unit. The rights to move members of your user base to said Organizational Unit. The ability to modify and/or create a new custom GPO. The ability to link/deploy said GPO. Hypothetical Scenario and Steps: You come into the office and your immediate manager mentions that the messaging group has been tasked with removing Reply-All ability for all users who participated in a recent Reply-All storm that impacted a large internal distribution list. Upper management has made a decision to not implement bulk message restrictions to the Distribution List itself (e.g. limiting who can send or reply to messages on the DL), but to simply remove Reply-All ability for all users who played a part in perpetuating the storm. The removal of this functionality will be accomplished via deployment of a GPO as opposed to leveraging other potential options. Step-1: Obtain and review the most recent thread in the Reply-All sto rm and build a list of all individual user accounts that either caused or played a part in perpetuating the storm. Build a master list of these users and denote the current location of their user objects within the Active Directory. Once done, receive "buy-off" from management that these users can and will eventually be moved into a dedicated Organizational Unit that will have a GPO applied that prevents the Reply-All button from being active within Outlook. Step-2: Once you have obtained buy-off, build a new Organizational Unit with an obvious clear name. To do so, go into Active Directory Users and Computers, right-click a user container, then select New, Organizational Unit, then provide it with a clear name. In the examples to follow my OU will be titled: "Users who have Reply All Removed" Step-3: Move the users into the new" Users who have Reply All Removed" organizational unit via Active Directory Users and Computers (or your favorite LDAP tool) created in Step-2. Step-4: If not already available proceed to download the appropriate Office Administrative Template files. In my scenario all my users can be assumed to be using Outlook 2007 Service Pack 2. The template files can be downloaded here: http://www.microsoft.com/downloads/details.aspx?FamilyId=73d955c0-da87-4bc2-bbf6-260e700519a8&displaylang=en Legacy Office Administrative Template Files can be downloaded from Office Online. So if your client base is running Outlook 2003, go here: http://www.microsoft.com/office/orkarchive/2003ddl.htm Step-5: Extract the Administrative Template files by running AdminTemplates.exe. Once extracted you should have 3 folders created (ADM, ADMIN, and ADMX): Step-6: Build a new Group Policy that links users of the "Users who have Reply All Removed" organizational unit. To do so: In Active Directory Users and Computers get Properties of the "Users who have Reply All Removed" organizational unit. Select the Group Policy Tab. On the Group Policy Tab, select New and give the policy a clear and applicable name such as "Remove Reply-All Ability" Once named, select Edit. The Group Policy Editor (GPE) opens. Step-7: Add the Office/Outlook Administrative Templates In Group Policy Editor: Navigate to User Configuration\Administrative Templates. Right-Click Administrative Templates and select Add/Remove Templates. On the Add/Remove Templates page select: Add Browse to the Outlook Administrative Template and Select Open (in my case I uncompressed the file to my desktop so the path was: c:\Documents and Settings\ericnor\Desktop\OLK2K7 Admin Templates\ADM\en-us\outlk12.adm): If performed successfully you should now see that the Microsoft Office Outlook 2007 Administrative Template files have been added in Group Policy Editor: Step-8: Once you have successfully added the Outlook Administrative Template, you need to properly define and enable the necessary policies. To successfully block "Reply All" functionality we need to enable and define two specific policies in the Administrative Template: A policy to disable the actual command button as well as the menu option. A policy to disable the keyboard shortcut. To accomplish: In Group Policy Editor navigate to the following path: User Configuration\Administrative Templates\Microsoft Office Outlook 2007\Disable Items in User Interface\Custom. Once here you will notice 2 policies that will currently show as "Not Configured": Disable Command Bar Buttons and Menu Items. Disable ShortCut Keys. To disable the Command Bar Reply-All button we need to select the Disable Command Bar Buttons and Menu Items Policy. Once the policy is open set the State of the policy to Enabled then select the "Show..." button. Any and all Outlook Command Bar Buttons that are currently disabled will be present here in a numerical context (although presumably this would be the first time working with the template so you would not see any numerical values present). Click Add, and in the "Enter the Item to Be Added" field, type 355. Once done click OK. You will be returned to the Show Contents page which should now resemble the following. Click OK: (It's worth making a note here: All command buttons and menu options are referenced by their numerical Command IDs. I will show you how to determine these values in a follow-up post, so that you can apply what you have learned to other command buttons and short-cuts should you desire). Back on the Disable Command Bar Buttons and Menu Items policy page, select Apply and OK. The "state" of the policy should now show as Enabled. Now we need to disable the keyboard shortcut for Reply All. To do so, we need to open the Disable Shortcut Keys policy, configure our settings and enable it. To do so: Select and open the"Disable Shortcut Keys" policy. Select Enable, and then click the "Show..." button. This will bring up the Show Contents page. Select Add. In the Add Item Window type the following: 82,12 Click OK which will shift the focus back to the Show Contents pane. Your value should look like this: (Short-cut keys are always distinguished via the decimal representation of their ASCII character value. In addition the keyboard modifier (e.g. the keyboard combination used to activate the short-cut) needs to be converted to decimal notation and referenced here. As previously noted, I will detail how these values are calculated and converted in a follow-up post). Back on the Disable Shortcut Keys property pages, select Apply and OK. Back in Group Policy Editor, both policies should now show as Enabled: At this point both of your policies are enabled and configured. Close Group Policy Editor. You will be returned to the Group Policy tab for the "Users who have Reply All Removed" Organizational Unit. Click Apply then Close. Step-9: Provide "The List" to Help Desk. You should inform the Help Desk that there is a very high probability that users on "the list" will be calling in to complain that their Reply-All functionality no longer works. Your Help-Desk personnel should know why. Step-10: Create a Distribution List then and add all members of "the List". Send an informational to all users informing them that their "Reply-All" functionality has been removed due to misuse and for "x" amount of time. Once the GPO has been applied to the user workstation, these users will notice that the "Reply to All" button is disabled. If one of these users were to hover their mouse over the disabled button they will be presented with the following information: They will also notice that the menu option for Reply-All is now disabled: If the user attempts to issue the keyboard shortcut for Reply-All (CTRL + SHIFT + R), nothing will happen. I will cover additional related "Techy stuff" on this process in a follow-up post! Happy Trails! - Eric NorbergAdventures in querying the EventHistory table
Beginning with Exchange 2007 the Exchange database has had an internal table called EventHistory. This table has been used to track the events upon which several of the assistants are based and for other short term internal record keeping. The way to query the table hasn’t been publicized before but it has a number of uses: It may tell you the fate of a deleted item (for situations where Audit logging or store tracing was not in place at the time of the delete) It can list accounts who have recently touched a mailbox It can show you the clients that have touched a mailbox Events are kept in the EventHistory table for up to 7 days by default. You can check what your retention period is for all databases by running: Get-mailboxdatabase | fl name,event* Name : MainDB EventHistoryRetentionPeriod : 7.00:00:00 There are a number of approaches to querying the table. Let’s start with a script (please review my caveats before actually running the script) and review the data that is displayed. The script is: Add-PSSnapin Microsoft.Exchange.Management.Powershell.Support $db = (get-mailbox <user alias>).database $mb=(get-mailbox <user alias>).exchangeguid Get-DatabaseEvent $db -MailboxGuid $mb -resultsize unlimited | ? {$_.documentid -ne 0 -and $_.CreateTime -ge “<mm/dd/yyyy>”} | fl > c:\temp\EventHistory.txt For the CreateTime specify the day of the event you are looking for. By default a maximum of 7 days are tracked. Depending on the date range selected and the activity in the mailbox the resulting file size starts at about 5KB and I have seen it rise to nearly 1GB. You can also replace the “| fl > c:\temp\EventHistory.txt” with “| export-csv c:\temp\EventHistory.csv”. I am using the FL output because it is easier for illustration purposes. Inside the EventHistory.txt file will be events like this one (this one is a bulk delete of emails using OWA): Counter : 15328155 CreateTime : 1/28/2013 9:46:16 PM ItemType : MAPI_MESSAGE EventName : ObjectMoved Flags : None MailboxGuid : d05f83c1-255c-42ae-b74f-1ac3329b306a ObjectClass : IPM.Note ItemEntryId : 000000008CFDF3C2BA873648866A1C17D0E3F1AB0700BC9C9BA42124CD4F896E8915C86B2BD00000006027C20000BC9C9BA4 2124CD4F896E8915C86B2BD0000041B6E6570000 ParentEntryId : 000000008CFDF3C2BA873648866A1C17D0E3F1AB0100BC9C9BA42124CD4F896E8915C86B2BD00000006027C20000 OldItemEntryId : 000000008CFDF3C2BA873648866A1C17D0E3F1AB0700BC9C9BA42124CD4F896E8915C86B2BD00000006027BF0000BC9C9BA4 2124CD4F896E8915C86B2BD0000041B6D6260000 OldParentEntryId : 000000008CFDF3C2BA873648866A1C17D0E3F1AB0100BC9C9BA42124CD4F896E8915C86B2BD00000006027BF0000 ItemCount : 0 UnreadItemCount : 0 ExtendedFlags : 2147483648 ClientCategory : WebServices PrincipalName : Contoso\TestUser PrincipalSid : S-1-5-21-915020002-1829042167-1583638127-1930 Database : Mailbox Database 1858470524 DocumentId : 10876 The EventName shows what was done with the object. End user deletes will be listed as moves. When you delete an item it is moved to either Deleted Items or to the Recoverable Items subtree I highlighted the ItemEntryID because that ties directly to the Item you need to locate. The subject and other human readable properties are not included in this table. The ItemEntryID is the database engine’s way of uniquely identifying each item. You can use this to search the mailbox in MFCMAPI and get properties like Subject, From, To, etc. The ParentEntryID is the folder in which the item presently resides. The OldItemEntryID is the previous ItemEntryID before the item was deleted. The OldParentEntryID is the folder it used to reside in. Flags will often show values like SearchFolder. Many events flagged as being related to search folders or folders are not going to be interesting to your investigations. If you are researching the fate of a deleted item they can be ignored. ClientCategory is the type of client that requested the operation. In this case webservices means that OWA was used to remove the item as part of a bulk operation conducted against a 2010 mailbox. If it was deleted individually then Exchange 2010 would list OWA here. The way ClientCategories are tracked in Exchange 2013 is a little different; you should see OWA for all End User deletes through that tool. PrincipalName and PrincipalSid give you the identity of the account that was passed to the information store when the operation was requested. At the time of writing these are not displayed by Exchange 2013. So – we have an output file. What do we do with it? The easy uses for the file (once it is imported into your favorite data analysis tool) at this time are: List of all accounts that have caused an event to be logged in the time period you specified Get a summary of operations (deletes, moves, new items, etc.) conducted on the days you specified Get a list of client types that have changed something in the mailbox Search the records returned for a particular ItemEntryID In our output the ItemEntryID is not immediately useful. To find out what the ItemEntryID in each record actually is we need to use MFCMAPI (steps related to MFCMAPI are at the end of this blog). Once you are in MFCMAPI you can go to the Tools menu, select “Entry ID” and then “Open given entry ID”. In the dialog that appears paste in the ItemEntryId or the OldItemEntryId that you want to investigate. When you click OK MFCMAPI will take you to the item you specified (if it is still in the mailbox). Once MFCMAPI takes you to the mail item you will see the Subject, From, To, Creation date and other meaningful properties. You will also see there is a property called PR_ENTRYID. PR_ENTRYID is the MAPI name for ItemEntryID. This field is our link between the representation of the data in our PowrShell cmdlet and in the more human readable presentation in MFCMAPI. Pulling ItemEntryIDs from the PowerShell output and looking them up one at a time in MFCMAPI may be a little too tedious for most Exchange administrators. If you have mo re than a handful of items you want to check (to see if they are useful and meaningful) it will take a long time to locate them all. The alternative is to start in MFCMAPI. If you can find the item you want there by looking at the subject line, date or other properties you can use the content of the PR_ENTRYID field in MFCMAPI to modify the Get-DatabaseEvent query to pull up the history for just that item. To do this you need access to either a restored copy of the mailbox in a lab or the item of interest must still be in the mailbox (possibly in deleted items or recoverable items). Here is a sample of how the get-databaseevent cmdlet would be used if you have the PR_ENTRYID: Get-DatabaseEvent $db -MailboxGuid $mb -resultsize unlimited | ? {$_.ItemEntryID -eq “000000008CFDF3C2BA873648866A1C17D0E3F1AB0700BC9C9BA42124CD4F896E8915C86B2BD00000006027C20000BC9C9 BA42124CD4F896E8915C86B2BD0000041B6E6570000” –or $_.OldItemEntryId –eq “000000008CFDF3C2BA873648866A1C17D0E3F1AB0700BC9C9BA42124CD4F896E8915C86B2BD00000006027C20000BC9C9 BA42124CD4F896E8915C86B2BD0000041B6E6570000”} | export-csv c:\temp\SingleItemEventHistory.txt Sometimes I have not been able to locate an item using this technique. If that happens it is useful to note that the PR_ENTRYID contains the ID of the mailbox, the folder and the item. For example here is the PR_ENTRYID of an item in the Inbox followed by the PR_ENTRYID of the Inbox itself: 000000006064986ABA58DF40A86C0C67E716264807004885B50069B1D04994374C02417D45A100000000324E00003DEF8F7 FFC1E3448B9D276F022E0E42D0000396D1B280000 000000006064986ABA58DF40A86C0C67E716264801004885B50069B1D04994374C02417D45A100000000324E0000 For the sake of comparison here are the PR_ENTRYIDs of two more folders in the same mailbox: 000000006064986ABA58DF40A86C0C67E716264801004885B50069B1D04994374C02417D45A10000000032510000 - deleted items folder 000000006064986ABA58DF40A86C0C67E716264801004885B50069B1D04994374C02417D45A100000000324B0000 - ipm_subtree folder From this you should be able to get an idea of how the field is divided up by looking at where the repeated digits end. For the purpose of tracking down an individual item that may be in a different folder (because of multiple moves) we want to be able to isolate the portion of the PR_ENTRYID that is specific to the item and modify our PowerShell statement appropriately. The final statement would look like this: Get-DatabaseEvent $db –MailboxGuid $mb -resultsize unlimited | ? {$_.ItemEntryID -like “*3DEF8F7FFC1E3448B9D276F022E0E42D0000396D1B280000” –or $_.OldItemEntryId –like “*3DEF8F7FFC1E3448B9D276F022E0E42D0000396D1B280000”} | export-csv c:\temp\SingleItemEventHistory.txt At this point if we still can’t find the item we want then our last chances are to remove the $_.MailboxGuid from the conditions (meaning we will search all mailboxes in the database – a very expensive operation please review the caveats) or to search other databases in the organization (databases containing delegates of the current user would be the ones to start with). If the data still can’t be found you have either made an error or the records are no longer present. If the records are present you should see all actions taken on the item recently. Caveats: At the time of writing Exchange 2013 is not reporting the account information in the EventHistory records. You can use the technique – you just won’t get any account names or SIDs from it. You can change the length of time items stay in the EventHistory table with Set-MailboxDatabase -EventHistoryRetentionPeriod. You can choose a period from 1 second up to 30 days. I don’t recommend setting a time that is too short as I have not tested how Event based assistants would react to that. For the full syntax of Set-MailboxDatabase please check the TechNet article for your Exchange version. If you choose to direct your output to a variable instead of a text file you should make sure you are running the PowerShell cmdlets from a workstation with the management tools installed. The variable (and the PowerShell session) are likely to consume a substantial amount of memory. These queries of the EventHistory table are expensive to run. Use good judgment in when you choose to run them based on the demands of your environment. In the labs I use all these queries take a second or two, but on a busy server with large databases you can easily be looking at 20-30 minutes per query. There will also be an I/O impact, but I don’t have a way to estimate that for you in advance. You can make the operation less expensive by lowering the number of records returned by Get-DatabaseEvent. We are already including the database and mailbox to look for. You can also add the EventNames and the StartCounter. The latter of these might be a little tricky. The StartCounter is an internal number that is specific to this table in the current database. You probably won’t know what counter value to use until you have already run a query and noted the counter values. This means StartCounter is mostly useful for reducing the impact of your second and subsequent queries of the same table in the same database. Assuming you know a relevant StartCounter value here is an example of doing this: Get-DatabaseEvent $db -MailboxGuid $mb –EventNames objectmodified, objectdeleted –StartCounter 15328155 -resultsize unlimited | ? {$_.documentid -ne 0 -and $_.CreateTime -ge “<mm/dd/yyyy>”} | fl > c:\temp\EventHistory.txt The example above searches a mailbox on a particular database for the event types specified and ignores any rows with a lower counter value than specified. This smaller dataset is then passed to the PowerShell pipeline for additional filtering and is ultimately saved to a CSV file that you can import into your favorite analysis tool. If you prefer to conduct your analysis in PowerShell you also have the option of assigning the result of Get-DatabaseEvent to a PowerShell variable (just remember the variable and the PowerShell session will consume memory proportional to the resultset returned). So how do you find the PR_ENTRYIDs I mentioned above in MFCMAPI? You can download MFCMAPI from https://mfcmapi.codeplex.com. 1. We need an Outlook profile for the mailbox we are searching. That profile should NOT be configured for Cached mode. If you are doing this from your machine make sure you have Full Access to the mailbox of the user. You can then create a profile for that specific user. 2. Once you have the profile open MFCMAPI and Log on 3. Select the profile you created for Step 1. You will see a screen like this one: 4. Double-click the mailbox which will open a window showing you the mailbox details. 5. If you already know the ItemEntryID you want to open and inspect you can locate it with this menu option: 6. If you don’t have the ItemEntryID expand the Root Container, Recoverable Item and Top of Information Store. If you are trying to locate details on a deleted item look in the Deleted Items folder and the Recoverable Items folder (and it’s subfolders) 7. Double-click Deleted Items to open a window that looks like this one: 8. Click the item to fill in the lower half of the window with the properties 9. Locate the PR_EntryID property and double-click it 10. The Binary box contains the value of the PR_ENTRYID field that you can use to search the EventHistory table in the Store. If you locate this value with MFCMAPI first you can use it to limit the search as I described above. If you don’t have this value you can pull the full history and use the ItemEntryIDs as a basis to search MFCMAPI. Thanks to Jesse Tedoff for the idea! Chris PollittHow to write an Exchange 2013 transport agent
Update 2/5/2013: We have also uploaded a sample that will work on Exchange 2010 servers, you can find it here. What is a Transport Agent? Transport agents allow Microsoft, developers in your organization and third-party vendors to hook into the Exchange transport pipeline with their code to process messages (e.g. an antivirus scanner for incoming email messages). Transport agents can process email messages that pass through the transport pipeline in many ways. An agent is a .Net assembly that has to be installed on the Exchange Client Access or Mailbox server. The agent is then loaded by the Exchange Transport service and invoked in the transport pipeline on the specified event. In Microsoft Exchange Server 2013, the transport pipeline is made of three different processes: Front End Transport service: This service runs on all Client Access servers and acts as a stateless SMTP proxy to route messages to and from the Transport service on a Mailbox server. Transport service: This service runs on all Mailbox servers and is virtually identical to the Hub Transport server role in previous versions of Exchange. Unlike previous versions of Exchange, the Transport service never communicates directly with the mailbox store. That task is now handled by the Mailbox Transport service. The Transport service routes messages between the Mailbox Transport service, the Transport service, and the Front End Transport service. Mailbox Transport service: This service runs on all Mailbox servers and consists of two separate services: Mailbox Transport Submission and Mailbox Transport Delivery. Mailbox Transport Delivery receives SMTP messages from the Transport service, and connects to the mailbox database using an Exchange remote procedure call (RPC) to deliver the message. Mailbox Transport Submission connects to the mailbox database using RPC to retrieve messages, and submits the messages over SMTP to the Transport service. Like the previous version of Exchange, Exchange 2013 transport provides extensibility that is based on the Microsoft .NET Framework version 4.0 and allows third parties to implement the following predefined classes: SmtpReceiveAgent RoutingAgent DeliveryAgent Note: This Article will concentrate mainly on how to implement and build a SmtpReceiveAgent. The SmtpReceiveAgentFactory and SmtpReceiveAgent classes provide support for the extension of the Microsoft Exchange Server 2013 Edge Transport behavior. You can use these classes to implement transport agents that are designed to respond to messages coming into and going out of the organization. The following list explains the requirements for using transport agents in Exchange 2013. The Transport service fully supports all the predefined classes in Exchange 2010, which means that any transport agents written for Hub Transport server role in Exchange 2010 should work in the Transport service in Exchange 2013. The Front End Transport service only supports the SmtpReceiveAgent class that was available in Exchange 2010, and third party agents can't operate on the OnEndOfData SMTP event. You can't use any third party agents in the Mailbox Transport service. Exchange 2013 updates to Transport Agent Management Due to the updates to the transport pipeline, the transport agent cmdlets need to distinguish between the Hub Transport service and the Front End Transport service, especially if Client Access server and Mailbox server are installed on the same physical server. All transport agent cmdlets now have the TransportService parameter. The values you can specify are Hub for the Hub Transport service and FrontEnd for the Front End Transport service. For example, to view the manageable transport agents in the Hub Transport service, run the command: Get-TransportAgent -TransportService Hub. To view the manageable transport agents in the Front End Transport service, run the command: Get-TransportAgent -TransportService FrontEnd. The TransportService parameter is available in all transport agent cmdlets: Disable-TransportAgent Enable-TransportAgent Get-TransportAgent Install-TransportAgent Set-TransportAgent Uninstall-TransportAgent StripIncomingLinkReceiveAgent This Transport Agent was designed and implemented to illustrate various Exchange 2013 transport agent functionality as well as stripping all the hyperlinks from the message body. This agent will illustrate the following functionality: Setting up the Visual Studio Environment to code and build the agent Adding Agent Event Handlers such as EndOfData and EndOfHeader to strip all the Hyperlink from the incoming message body Enumerate through all the message body part and select the text body part and then stripping the link. Removing any existing attachment from the message Skipping Health messages sent by the Exchange Process MimePart of the body Error Handling Implementing test unit to test various part of agent before installing on Exchange 2013. Setting up the Environment Visual Studio can be utilized to build and implement a transport agent. This lesson will show you how to build a transport agent to remove all HTML links from an incoming SMTP messages as well as illustrating on how to use the agent in the Exchange 2013 Front End Transport Service which was introduced in Exchange 2013. 1. In Visual Studio 2012, Create a new project using Templates->Visual C#->Windows->Class Library and name the project StripIncomingLinkAgent 2. Create a folder name Exchange under \StripIncomingLinkAgent\StripIncomingLinkAgent and copy the following DLLs from C:\Program Files\Microsoft\Exchange Server\Public Microsoft.Exchange.Data.Common.dll Microsoft.Exchange.Data.Transport.dll 3. In Visual Studio 2012, Go to Solution Explorer, Right click the References and select “Add Reference…” 4. Select Browse from the Reference Manager dialog and navigate to \StripIncomingLinkAgent\StripIncomingLinkAgent\Exchange 5. Select both dll and click add and OK 6. These two DLL will be used to integrate the agent to MSExchangeFrontEndTransport and MSExchangeTransport process. 7. First step is to rename the Class1.cs generated by Visual Studio to a meaningful name (e.g. StripIncomingLinkReceiveAgent) 8. Next Step is to create a ReceiveAgent class (naming the class StrinIncomingLinkReceiveAgent) which will inherit from SmtpReceiveAgent and add the proper references: using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; using System.Diagnostics; using System.IO; using Microsoft.Exchange.Data.Transport; using Microsoft.Exchange.Data.Transport.Smtp; using Microsoft.Exchange.Data.Transport.Routing; using Microsoft.Exchange.Data.Transport.Delivery; using Microsoft.Exchange.Data.Mime; using Microsoft.Exchange.Data.Transport.Email; using StripLink; using StripLink.Utilities; using StripIncomingLinkAgent.Configuration; 9. Before adding the business logic to our agent code and setup the agent call back, we need to setup a logging method to log events to the application event log: public static void WriteLog(string message, EventLogEntryType entryType, int eventID, string proccessName) { try { EventLog evtLog = new EventLog(); evtLog.Log = s_EventLogName; evtLog.Source = proccessName; if (!EventLog.SourceExists(evtLog.Source)) { EventLog.CreateEventSource(evtLog.Source, evtLog.Log); } evtLog.WriteEntry(message, entryType, eventID); } catch (ArgumentException) { } catch (InvalidOperationException) { } } 10. Next step is to register our call back with the Transport EventHandlers and create our agent: public sealed class StripLinkReceiveAgentFactory : SmtpReceiveAgentFactory { public override SmtpReceiveAgent CreateAgent(SmtpServer server) { IConfigurationProvider config; config = new StaticConfigProvider { ForceSinglePart = true, ForceTextPlain = true, FilterAuthenticated = true, FilterTNEF = true, HonorAntiSpamBypass = false, SkipInternalMessages = false, AlwaysFilterCAFETraffic = false }; // To use the RegistryConfigProvider class - uncomment the line // below and comment above staticConfigProvider // config = new RegistryConfigProvider() return new StripIncomingLinkReceiveAgent(config); } } public class StripIncomingLinkReceiveAgent : SmtpReceiveAgent { private static string processName = "ExchagneStripIncomingAgent"; private string machineName; private Process currentProcess; private static string FrontEndTransport = "ExchangeStripIncomingLinkFrontEndAgent"; private static RoutingAddress inboundProxy = new RoutingAddress( "inboundproxy@inboundproxy.com"); private bool m_IsTNEF = false; private bool m_IsSummaryTNEF = false; /// <summary> /// Configuration provider. Should be set by constructor. /// </summary> private IConfigurationProvider configProvider; public static void WriteLog(string message, EventLogEntryType entryType, int eventID, string proccessName) { StripLinkHelper.WriteLog("ReceiveAgent: " + message, entryType, eventID, proccessName); } public StripIncomingLinkReceiveAgent(IConfigurationProvider config) { configProvider = config; this.OnEndOfData += new EndOfDataEventHandler(StripIncomingLinkEndOfDataHandler); this.OnEndOfHeaders += new EndOfHeadersEventHandler(StripIncomingLinkEndOfHeadersHandler); currentProcess = Process.GetCurrentProcess(); machineName = Environment.MachineName; if (currentProcess.ProcessName.ToLower().Contains("frontend")) { processName = FrontEndTransport; } } 11. Now we can implement the Event Handler logic for End of Header event handler. We are keeping the Header Event handler simple. The code checks for health messages generated by Exchange and skip processing of those messages and it adds a marker to the header to notify the backend agent that header has been processed by frontend agent: private bool IsHealthMessage(MailItem mailItem) { string domainName = mailItem.FromAddress.DomainPart; RoutingAddress adminEmailAddress = new RoutingAddress("Administrator@" + domainName); return (mailItem.Recipients.Contains(inboundProxy) || mailItem.FromAddress.LocalPart.Contains("HealthMailbox") || mailItem.FromAddress.LocalPart.Contains("inboundproxy")); } private void StripIncomingLinkEndOfHeadersHandler(ReceiveMessageEventSource source, EndOfHeadersEventArgs e) { try { if (IsHealthMessage(e.MailItem) && !(currentProcess.ProcessName.ToLower().Contains("frontend"))) return; // Add a header indicating that this was processed // by a Front End Server... StripLinkHelper.MarkAsProcessedByCAFE(machineName, e.Headers); } catch (Exception except) { WriteLog("EndofHeader Exception = " + except.ToString() + processName, EventLogEntryType.Error, 10, processName); } } 12. End of Data Event handler has all the logic to parse the message body, create a single part (based on registry key value), convert the message to plain text (based on registry key value) and finally remove all the hyperlink from the message and save the body using StreamWriter: private void StripIncomingLinkEndOfDataHandler(ReceiveMessageEventSource source, EndOfDataEventArgs e) { try { EmailMessage message = e.MailItem.Message; Body currentBody = message.Body; if(ShouldSkipMessage(e.MailItem, e.SmtpSession)) { // Message skipped, nothing to do. return; } // The following Actions are only valid for // pure mime messages (Not TNEF) if (!m_IsTNEF) { // Do we want to make it single part // (only Valid is no TNEF) // The goal here is to remove all other mime parts // (body types, attachments) to minimize // the surface area where hyperlinks can be found. // There is no point in removing hyperlinks in the body // if there is an HTML attachment in the email. :) if (configProvider.ForceSinglePart) { // Once again, we try to reduce the exposure to hyperlinks. // We do this by trying to get the lowest fidelity // body, and making that the only mimepart on the message. // We hope for Text/Plain body but it could be HTML. MimePart mimepartLowFidelity = StripLinkHelper.GetLowerFidelityBodyPart(currentBody); if (mimepartLowFidelity != null) { // We now remove any branches that do not contain // this node. Another option is to make the // message single part, however, this will // require merging headers of the source // part with the root part. // NOTE: This will break multipart/related messages // because they rely on the other parts // to store the other components (images, documents, etc). // This should not be an issue // if the target part is always text/plain. StripLinkHelper.MakeSingleBranch(mimepartLowFidelity); } else { // TODO: Decide what to do in this scenario... // Probably route to Admin... // This is the case for TNEF messages as there // is no MimePart associated with the body. WriteLog("Failed to get a low fidelity body.” + “Mime tree will not be simplified. Quarantine Message " + processName, EventLogEntryType.Error, 10, processName); source.Quarantine(e.MailItem.Recipients, "Could not find MimePart for the body."); return; } } // Force Plain text... if (configProvider.ForceTextPlain) { StripLinkHelper.ForcePlainTextIfNeeded(message); } } else if(configProvider.ForceSinglePart) { // We do honor the ForceSinglePart for TNEF, we // assume this means we don't want any attachments // so for TNEF we simply remove the message attachments. // A seperate option could be added, TNEFRemoveAttachments // if this needs to be handle independent of // the ForceSinglePart settings. message.Attachments.Clear(); } // We now need to process the message. StripLinkHelper.ProcessEmailBody(message.Body); // TODO: For LegacyTNEF we thought we also needed to // filter the text/plain part that is included // for non-TNEF clients, but it appears modifying the body // also generates a new text/plain part so // the code below is not needed. //if (m_IsTNEF && !m_IsSummaryTNEF) //{ // FilterTNEFTextPart(message); //} StripLinkHelper.MarkAsProcessedByFilteringAgent(message.RootPart.Headers, machineName); } catch (Exception except) { WriteLog("EndofData Exception = " + except.ToString() + processName, EventLogEntryType.Error, 10, processName); source.Quarantine(e.MailItem.Recipients, "StripIncomingLinkAgent - Error occurred: " + except.Message); } } 13. Process Body method uses the regular expression class to remove any hyperlink or website address from the body of the message: public class TextToTextLinkProcessor : IHyperLinkProcessor { private int m_LinkCount = 0; private const string s_RegExBodyString = @"((www\.|(http|https|ftp|news|file)+\:\/\/) [_.a-z0-9-]+\.[a-z0-9\/_:@=.+?,##%&~-]*[^.|\'|\# |!|\(|?|,| |>|<|;|\)])"; private bool m_bChanged = false; private string m_strReplacementText = string.Empty; public bool WasChanged { get { return m_bChanged; } } public void ProcessEmailBody(Body bodyMessage) { m_LinkCount = 0; string savedContent = string.Empty; Stream memStream; Encoding encoding = StripLinkHelper.GetEncodingFromBody(bodyMessage.CharsetName); if (bodyMessage.TryGetContentReadStream(out memStream)) { using (StreamReader streamRead = new StreamReader(memStream, encoding)) { // TODO: May also want to decide on size of message and // whether or not it should be processed if it is too large. savedContent = FilterText(streamRead.ReadToEnd()); } // Now write the new body only if it was changed/filtered. if (m_bChanged) { using (StreamWriter streamWriter = new StreamWriter(bodyMessage.GetContentWriteStream(), encoding)) { streamWriter.Write(savedContent); } } } } public string FilterText(string strText) { Regex rgx = new Regex(s_RegExBodyString, RegexOptions.IgnoreCase); string strFiltered = rgx.Replace(strText, new MatchEvaluator( match => { // If we got a match, mark it as changed. m_bChanged = true; m_LinkCount++; return m_strReplacementText; })); return strFiltered; } 14. GetLowerFidelityBodyPart enumerates through all the available body type and return the plain text body: /// <summary> /// Finds the lowest fidelity MimePart associated to a Body, /// normally Text/Plain. /// </summary> /// <param name="body">The body part that we want /// to find the lowest fidelity MimePart for.</param> /// <returns></returns> public static MimePart GetLowerFidelityBodyPart(Body body) { // Nothing to work with if they are null... // Divided here for logging purposes.. if (body == null) { return null; } else if (body.MimePart == null) { return null; } MimePart bodyPart = body.MimePart; ; // If it is text already, then that's the lowest fidelity... if (body.BodyFormat == BodyFormat.Text) return body.MimePart; // Need to find lower fidelity body type at the same level. // If no parent, there are no children, so this would be the only // part at this level. // If it has a parent but the parent is not multipart then this will also // be the only child. so: // if (bodyPart.Parent == null || !bodyPart.Parent.IsMultipart) // The other option is to simply check that we have no siblings. if (bodyPart.PreviousSibling == null && bodyPart.NextSibling == null) { return bodyPart; } // If we are here then we must have a parent and siblings. Get // the parent, find the lowest fidelity.. IEnumerator<MimeNode> enumer = bodyPart.Parent.GetEnumerator(); MimePart currentPart; while (enumer.MoveNext()) { currentPart = (MimePart) enumer.Current; if (currentPart.ContentType.Equals(s_TextPlainContentType, StringComparison.OrdinalIgnoreCase)) { return currentPart; } } return bodyPart; } 15. The complete Visual Studio project can be foundattached to this blog post. You can build the project by simply selecting build from the “Build” pull down menu using the release version. 16. Once the agent build is completed you can copy the StripIncomingLinkAgent from the \bin\release folder on the exchange server in the following folder: C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\SmtpAgents\StripIncomingLinkAgent 17. Use the following cmdlet to install and enable the agent on the Front End Transport on CAS. Note: if your CAS and Mailbox servers are on separate machine then you need to launch a window powershell to prevent the powershell proxy: Launch a Window Powershell window and execute the following commands: C:\Program Files\Microsoft\Exchange Server\V15\bin> Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn Install-TransportAgent -Name "FrontEndStripIncomingLinkAgent" -TransportAgentFactory "StripIncomingLinkAgent.StripLinkReceiveAgentFactory" -AssemblyPath "C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\SmtpAgents\StripIncomingLinkAgent\StripIncomingLinkAgent.dll" -TransportService FrontEnd Enable-TransportAgent -Identity "FrontEndStripIncomingLinkAgent" -TransportService FrontEnd 18. User the following cmdlet to install and enable the agent on the back end Transport on Mailbox: Install-TransportAgent -Name "StripIncomingLinkAgent" -TransportAgentFactory "StripIncomingLinkAgent.StripLinkReceiveAgentFactory" -AssemblyPath "C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\SmtpAgents\StripIncomingLinkAgent\StripIncomingLinkAgent.dll” -TransportService Hub Enable-TransportAgent -Identity "FrontEndStripIncomingLinkAgent" -TransportService Hub 19. After installing the agent on back end, send an email from the pickup folder with a few link such as (www.msn.com, etc.) 20. The agent will remove all the links from the email body. Please remember that this is a sample only. David Santamaria, Nasir Ali and Nasser Salemizadeh
