So your Exchange Transport rule (ETR) or Data Loss Prevention (DLP) rule is not working as expected...

 

... and troubleshooting this issue can get extraordinarily complex, especially if it´s hard to reinitiate the sent email to check adjustments. For many years this has been a recurring question and feedback we got from you.

 

Let´s look into today's situation, taking this example. With two configured ETRs, as usual, rules have been configured and shortly after that, feedback comes back that it´s not working. In the first steps, the admin revisits the rule and does some sanity checks for typos or the correct ordering. But that seems to be fine.

 

Before we dig deeper into that one thing to be aware of is the pipeline order of processing ETR and DLP rules. ETR is processed before DLP comes into play, so anything made in ETR would be available for DLP. But not the way around.

 

Up until now the only way to see if an ETR is applied to emails was by checking the message trace, which would display matched rules. So, in this example, the Rule which matches would look like this in the message trace

 

If you needed to check which rules have been evaluated for a specific message you had to use the Extended Message Trace data, this needs to be requested by an Exchange Administrator. Also Enhanced summary and Extended reports are prepared using archived message trace data, and it can take up to several hours before your report is available for download (Message trace in the modern EAC in Exchange Online | Microsoft Docs). Which is fairly not a quickly executed task. This is going to be even more complex when dealing with DLP, as using DLP allows you to specify the count on how often sensitivity information type matches and the confidence of the detection before the policy triggers.

 

Using the extended message trace to check for ETR and DLP rule evaluation

 

As mentioned before more information for ETR und DLP evaluation is available in the Extended Trace, so let's get back to my first example but with one more enabled rule, which doesn´t match and therefore doesn´t show up in the simple message trace. Which would give you the chance to use the Extended trace now.

As a reminder, this is how it looks like in the simple trace:

 

Now let´s check the Extended Trace Report. What I found the easiest way to open such a Trace is to import (using From Text/CSV) the CSV into Excel, which helps to structure and sort the data as needed.

 

In our case, we are interested in the “custom_data” field of the “AGENT AGENTINFO” line

 

I use Powershell to split this string in useful junks, for ETR checks we are interested in the “TRA=ETR|” and “TRA=ETRP|” lines. ETRP| shows me all the processed ones, the one without P| is the applied one including the appropriate action. This gives you insides which rules (rule ID values which can be listed using Get-TransportRule -Identity cmdlet ) have been evaluated, but without further detailed information.

 

The same applies to DLP policies, message trace only shows you the executed policies not all the evaluated. In our case, there are two DLP policies that should be checked. Both policies have configurations for “low” and “high” volume detection. So that finally four evaluations will be done.

 

 

But the simple message trace only shows the matching DLP Policy, same as the ETR case.

 

Now let’s investigate the DLP-related lines, with the same steps of preparing the data to review. Get the Custom_data field of the “AGENT AGENTINFO” line and split it. But for DLP we are looking at the “DPA=” lines. In this example, we have four DPA= lines which are matching the expected four evaluations. None of them contains the word “action” there nothing was found.

 

To match the evaluated rules to your DLP policy you need to switch your PowerShell connection to the SCC target using “Connect-IPPSSession” which is part of the Exchange Online management Module. What your need to look for is the mgtRuleId and match this to the ImmutableID of the DlpCompliance Rules using “Get-DlpComplianceRule”. In my example, the first mgtRuleId is the of “low volume”.

 

Next, a trace where DLP rules matched with one of the sensitive information types (SIT). Sensitive info (dcid) with id “50842eb7-edc8-4019-85dd-5a5c1f2bb085” is detected with a Unique count of 12 (ucount) and confidence of 85 which is high and therefore the rule executed the action to block.

 

Which SIT matched can be simply identified by comparing “dcid” number with the ID of the SITs using the same Connect-IPPSSession as before checking for the DLP policy.

 

Let's talk about what you can do in addition to the message trace to get more from Exchange Online

 

Microsoft Information Engineering released a new cmdlet "Test-Message" for all Exchange Online customers which allows you to kick off an ETR/DLP Evaluation of a specific mail. The documentation is available since yesterday in the Exchange Online documentation Test-Message (ExchangePowerShell) | Microsoft Docs.

 

But first, some general information to know before starting to use it. Test message essentially introduces an email into the actual ETR/DLP pipeline to simulate the real behavior

1. Actions such as moderation/Block are taken, and corresponding notifications are sent as configured in the rule.
2. The tracing report email would be visible in the exchange message trace and DLP events would be visible in audit/activity explorer.
3. Today GIR/end-user notification is suppressed to not be sent by DLP. But Engineering would like to get your feedback if even this should be sent in your cases.

The "Test-Message" cmdlet has been used by our Engineering Teams for a long time now to develop, test, and troubleshoot evaluation in ETR/DLP. It enables you to send an EML-Message as a template through the stack and get a detailed report about the evaluation of the rules.

Let's do this with the examples from above:

• Get the EML files to use with the cmdlet, one user-driven quick approach could be to ask them to open OWA, switch to the folder the mail ist stored, open a new mail, and move the needed email to the new mail body. OWA will attach this email as an attachment to the new mail. And allows you to simply download this EML attachment.

 

 

This EML file can now be used with the cmdlet to process the evaluation and get a detailed report about it. Just open your Exchange Online Powershell connection and execute this:

 

 

$data = Get-Content -Path "C:\temp\eml\Check out my new email signature.eml" -Encoding Byte -ReadCount 0
 
Test-Message -Sender LidiaH@M365x613720.OnMicrosoft.com -Recipients AllanD@M365x613720.OnMicrosoft.com -SendReportTo admin@M365x613720.onmicrosoft.com -TransportRules -UnifiedDlpRules -MessageFileData $data

 

 

After that, your "SendReportTo" Recipient will receive such a report email which will contain all the details of rule processing, depending on the parameter you specified this could be ETR only or DLP only, or both. If you choose both, you will receive a separate report for ETL and DLP processing.

 

The ETR report email looks like this. It will tell you all the details which have been used for the evaluation like for the first ETR

Is sent to 'Outside the organization. The evaluation Condition is 'Microsoft.Exchange.MessagingPolicies.Rules.IsInternalPredicate' evaluated as 'Not Match'. and it didn´t match.

 

The second ETR with the condition Message.From property value evaluated as rule condition: 'System.Collections.Generic.List`1[System.String]'property is a collection of values: 'LidiaH@M365x613720.OnMicrosoft.com' matched and applied the configured Action to apply disclaimer.

 

Looking into the DLP report for my second example with the blocked message based on the detected amount of credit cards, the report exposes the discovered sensitivity type and the rule action in this case Evaluating rule action High volume of content detected U.S. Financial.

 

Keep in mind this also works with examples of external senders or recipients, so no more need to ask external partners to resend a message for finetuning or troubleshoot ETR/DLP rules!

 

Therefore, we are hoping that this new capability provided by Microsoft Information Engineering will help you to find answers to your ETR and DLP questions in the future more quickly and more understandable.

 

Also be sure to check out the Powershell documentation which is now available here Test-Message (ExchangePowerShell) | Microsoft Docs

 

Big thanks go to AakashMalhotra  and the whole PM team for helping me with reviews, feedback, and processes to get this published.