Microsoft Defender for Office 365 enables users and administrators to submit suspicious items for analysis (email and Teams messages, files, or URLs) to enhance detection and prevention. Your submissions allow Microsoft to determine the nature of the item, update filtering decisions, and offer you actionable insights. We're often asked what happens after you submit an item to Microsoft, so here's a brief overview of what happens behind-the-scenes.
How to submit items to Defender for Office 365
You can submit items to Microsoft Defender for Office 365 in diverse ways, depending on your role and the source of the item. For example, you can submit items from:
- Outlook, after configuring the user reported settings and the reported message destination to Microsoft only or Microsoft and a reporting mailbox.
- Microsoft Defender XDR Submissions, Quarantine pages, or the Take Action wizard (for organizations with Microsoft 365 E5 or Microsoft Defender for Office 365 Plan 2 licenses) or using Real-time detections (for organizations with Defender for Office 365 Plan 1 licenses). You can create allow or block entries while submitting items to resolve false positives and false negatives immediately.
- Message trace in the Exchange admin center (EAC).
TIP: There’s no difference between user reported items and admin submissions from a feedback point of view. They’re just ways for different personas to report items.
We strongly recommend configuring the message destination as either Microsoft only or Microsoft and reporting mailbox in user reported settings. This configuration reinsures that admins don't need to resubmit user reports. When user reported settings are configured to send messages only to the reporting mailbox, security teams should actively submit user reports via admin submissions.
What happens after you submit an item for analysis
After an item is submitted to Defender for Office 365 by users or admins, it goes through the following steps:
Phish simulation check
- We check if the email is either first-party Attack simulation training or third-party phishing simulation training configured via the Advanced delivery policy for third-party phishing simulations. If it is, the User reported view on the Submissions page will show “Phish simulation” in the Result column.
- Alerts aren’t generated, and Automated investigation and response (AIR) investigations aren’t performed for user reported items of the “Phish simulation” type.
- The items aren’t analyzed any further.
Authentication check
- Whether the message passed or failed email authentication checks: Sender Policy Framework (SPF), DomainKeys Identified Mail (DKIM), Domain-based Message Authentication Reporting and Conformance (DMARC), and composite authentication.
Policy hits
- The Microsoft Defender for Office 365 filtering verdict was overridden due to admin overrides (Exchange mail flow rule (transport rule), anti-spam policy allow/blocks, anti-spam policy settings, or the Tenant Allow/Block List).
- The Microsoft Defender for Office 365 filtering verdict was overridden due to user overrides (for example user defined Outlook settings like: trust email from my contacts, user Safe Senders list (including domains) user Blocked Sender list (including domains)).
- The Microsoft Defender for Office 365 filters detected the message, but the admin configuration prevented action (for example, impersonation protection in anti-phishing policies is turned off).
Defender for Office 365 filters check the latest verdict (rescan)
Sometimes, Microsoft Defender for Office 365 has already caught up with the Indicator of Compromise (IOC; URL/Attachment/Sender/IP) associated with the submission. This updated decision could be due to changes in Sender/IP reputation or detection of URLs and files involving delayed weaponization (initially clean, but malicious after a delay).
- The item is rescanned by various engines and systems (like antivirus, antispam, and machine learning).
- If the rescan results in an updated verdict, the submission result shows the latest verdict.
- The item is not analyzed any further since the Microsoft Defender for Office 365 filters have started classifying it correctly.
Grading
The submitted item is analyzed and classified by a mix of our state of the art automated and human graders. The graders look at the messages, examine URLs and attachments, QR codes, and all metadata associated with the submitted item.
Based on the combination of these 5 steps, you get the result, result details, and recommended steps. The complete set of results can be found here.
In U.S. Government organizations (Microsoft 365 GCC, GCC High, and DoD), admins can submit items to Microsoft for analysis, but the items are analyzed for phishing simulation, authentication, and policy hits only. Rescan and grader analysis aren't done for compliance reasons (data isn't allowed to leave the organization boundary).
Why does Microsoft validate submitted items? Why does Microsoft sometimes disagree with the submission?
While we know most customers have done their due diligence before submitting items to Microsoft, trusting every submission can be a critical security loophole. Some email verdicts can be subjective (spam for one user can be acceptable to another users), and some submissions may involve human errors. Hence, it’s critical for Microsoft to confirm the submission before using it to update filters.
In cases where Microsoft’s final grade is different from what was reported, we ensure we have a strong signal for disagreement. For example, if the email was submitted as clean but our human analysts found convincing evidence of malicious entity/intent, the submission result shows the verdict of the human analyst. Microsoft tries to ensure the instances of incorrect disagreements are low. However, there can be exceedingly rare cases (human error) where customers might feel the disagreement is invalid.
Why does Microsoft sometimes generate the result as Unknown?
This verdict happens due to two reasons:
- Despite our rigorous grading process, sometimes we cannot reach a conclusion for classification. For example, by the time we grade, URLs in the email can be inaccessible. We continuously monitor these samples to research and design techniques to reduce unknowns and classify related items in the future.
- Given the volume of submitted items Microsoft receives, the human analyst is unable to review it in a timely manner. We monitor our analysis coverage rate daily and make investments so we can review each submitted item in a timely manner.
What actions does Microsoft take after analyzing a submission?
If the submitted item is identified as malicious (a false negative), Microsoft takes one or more of the following actions:
- Execute a system-wide zero-hour auto-purge (ZAP) action for messages containing the IOC (URL, file, fingerprint, etc.) ZAP and its requirements are explained here: Zero-hour auto purge (ZAP) in Microsoft Defender for Office 365.
- Add the specific IOC to system-wide Microsoft Defender for Office 365 internal block lists.
- Use the label as input to retrain machine learning models and other heuristics.
- Add manual rules targeting specific patterns that Microsoft Defender for Office 365 missed.
- Invest in durable improvements for attack methods that require using innovative technologies (for example, QR code-based attacks).
If the submitted item is identified as clean (a false positive), Microsoft takes one or more of the following actions:
- Mark the specific IOC as clean across Microsoft Defender for Office 365.
- Use the label as input to retrain machine learning models and other heuristics.
- Update existing manual rules targeting specific patterns that Microsoft Defender for Office 365 is catching.
- Invest in innovative technologies and techniques that cause less legitimate email to be identified as malicious.
For user reported phishing messages, automated investigation and response (AIR) is triggered from the alerts. AIR clusters all related messages, and then analyses them to determine if the original email was malicious. When a user submission cluster is deemed to be malicious, it will recommend remediation action to the SOC team for the entire cluster, increasing SOC team efficiency in remediating threats and responding back with automated feedback to end users.
Why are changes not immediate despite accepting the submission as valid?
Even if your submission is accepted as valid, our technologies are designed for long-term improvements and actions to durably fix the Microsoft Defender for Office 365 filtering stack (bulk/phishing/spam/malware/clean) is not straightforward in all instances. Hence, several reasons can influence the immediate nature of changes, including:
- Taking the easy route of just adding a sender or IP to allow/block lists globally as a fix can have devastating effects if done incorrectly. This carries high risk especially at the scale at which Microsoft operates (some email filtering solutions in the market might accept this risk). For example:
- Legitimate domains that were hacked and used to send malicious campaigns. Marking the domain as bad for extended periods of time can have significant impact on legitimate domain owners.
- There could be attacker-owned domains which initially build a good reputation to get their emails submitted as clean. They could use this as a "free pass" to send threat campaigns, which could have a devastating impact for numerous Microsoft customers.
- Attackers often used shared infrastructure which can be used by both legitimate senders and bad actors.
- Machine learning (ML) models are trained on large volumes of data. Often, one submission from one customer might not be enough to cause the behavior to change. While this is not an excuse, it helps highlight the trade off with these ML models (catch at the cost of explainability).
- Email verdict (for spam, bulk) can be subjective at times. What is spam for one user can be acceptable or desirable for another user. Outlook end user allow/blocks enables personalization.
- Given the scale of submissions, manual rules written by security researchers come into the picture only for submissions that are part of a big cluster (email messages with same IOC).
For all the previously explained reasons, even though we have several ongoing investments to improve the process of learning from submissions, some customers might perceive no immediate change, even after a submission. We recommend using the Tenant Allow/Block Lists actions available during submission (in Take action wizard or Submission pages) for immediate relief and let Defender for Office 365 manage the expiry of those depending on the time to learn.
Your submissions help improve email security
Submissions are the most critical source of information for Defender for Office 365 to improve. We continue to encourage performing submissions to fix FP/FN issues and resort to Support tickets only when you feel the need for additional intervention. Defender for Office 365 utilizes every single submission, even if one might not perceive any immediate change. Apart from contributing to the collective security intel by submitting items as spam, phish, malware, or clean, you are also indirectly helping Defender for Office 365 get better and be on top of attackers. In turn, submissions benefit your organization and other customers across the service by reducing the number of unwanted email messages you receive and ensures that legitimate email messages aren't mistakenly flagged.
More information
- Learn more about reporting a suspicious email or file to Microsoft.
- Learn more about order and precedence of email protection.
