Microsoft Secure Tech Accelerator
Apr 03 2024, 07:00 AM - 11:00 AM (PDT)
Microsoft Tech Community
Evaluate Defender for Office 365 in your environment!
Published May 31 2022 09:00 AM 8,578 Views

Email is ubiquitous, so securing it is complex

 Despite the recent surge in chat-based and video conferencing tools in the workplace, email remains the primary method of communication used by most organizations. As a result, email is in use 24 hours a day, by just about everyone in your organization. And with email representing the primary attack vector for cybercrime1, email security is a top concern from security teams and CISOs around the world.


But mastering email security can be complex. Email users face a variety of types of attacks, from simple malware attachments, to intricate credential phishing pages, and even payload-less business email compromise attacks. The tools that prevent these attacks provide many layers of protection, offer a variety of configurations, and typically require tuning to achieve the correct balance between productivity and security.


Given the ubiquity of email and the role it plays in business, customers typically want to evaluate email security solutions without impacting production environments and the day-to-day operation of their business.


Challenges with comparing email security solutions

Of course, customers have the right to choose the solution that fits their organization best. Unfortunately, all email security solutions are not created equal. Advanced threat protection offerings across the industry are built differently, and typically use different syntax and controls that are hard to compare or translate.


Effective evaluations require real senders and real recipients

Effectively comparing these solutions seems simple, but doing so in a sufficient way requires a deeper dive into the nuts and bolts of the underlying technology. Today’s security tools make use of a variety of signals, from threat intelligence (things we’ve seen before and know are malicious or suspicious), to machine learning and artificial intelligence (identifying malicious or suspicious content based on patterns).


If we take Defender for Office 365 as an example, the diagram below shows the full protection stack that Microsoft uses to evaluate the contents and reputation of a message and it’s sender.

Figure 1: Effective evaluation of email security products requires a real sender and a real recipient.Figure 1: Effective evaluation of email security products requires a real sender and a real recipient.


As is indicated by the yellow and red designations in the diagram, without a real sender and a real recipient, many of these capabilities are either bypassed or are rendered ineffective.


This illustrates why synthetic emails used to evaluate Microsoft Defender for Office 365 are not recommended. Synthetic messages or traffic lacking a real sender or real recipient is missing important metadata that allows us to provide accurate verdicts on that email.



Evaluation Mode makes evaluating Defender for Office 365 easier than ever

Through our conversations with customers over the past few years, we’ve noticed an opportunity to simplify the evaluation process. As a result, we’re thrilled to release Evaluation Mode in Microsoft Defender for Office 365.


Given the importance of proper evaluation, we’ve worked to develop a tool that allows customers to both a) evaluate Defender for Office 365 using real world traffic, and b) have the option to do so without impacting production environments. We’ve enabled these capabilities in two modes: blocking mode and audit mode, allowing customers to select whether or not they would like to apply Defender for Office 365 protections as they evaluate.


Both methods of evaluating Microsoft Defender for Office 365 allows customers to take advantage of the full set of protection capabilities in the product, and evaluate all features of Defender for Office 365 (including Attack simulation training). For the 90 day evaluation period, customers can evaluate the service, getting hands on with the Threat protection status report, investigate and hunt via Threat Explorer and the Email entity page, and run phishing simulation with Attack simulation training.


Evaluate in Blocking Mode

Regardless of where your mail exchanger (MX) records point, customers can choose to evaluate Microsoft Defender for Office 365 capabilities in blocking mode. In this case, Defender for Office 365 automatically applies the standard protection Preset security policy to evaluation traffic.

Throughout the evaluation period, you can choose at any time to opt in to a higher protection template (our strict Preset security policy) or create your own custom policies that suits your needs.


Evaluate in Audit Mode

If you choose to evaluate in audit mode, Defender for Office 365 will not take blocking action on messages that we determine to be harmful. These threats will be logged and available for your review through the Threat protection status report, giving you detailed information on the types of threats that were caught, who they were targeting, and so much more. Audit mode may be a valuable tool for customers who have an MX record pointing to another solution. This additional “catch” that is identified by Defender for Office 365 will serve as an indication of the additional protection capabilities available over standard Exchange Online Protection capabilities, or that of any other third-party email gateway that customers are using. Once customers are satisfied with their evaluation experience and are ready to use Microsoft Defender for Office 365 to protect their email and collaboration workloads, they can switch their MX record to point to Office 365.


Figure 2: Enable Evaluation Mode in either blocking or audit mode.Figure 2: Enable Evaluation Mode in either blocking or audit mode.


How can I use Evaluation Mode without changing my MX record?

If your organization is using a third-party email security solution, or you are using on-prem appliances to route mail traffic, there is no prerequisite to make changes to your MX record in order to use Evaluation mode.


How does this work?

As part of Evaluation mode, Office 365 uses the inbound connector that routes email from an external service or server to our environment. When you evaluate Defender for Office 365 in audit mode, we recommend enabling Enhanced Filtering for Connectors. This step is included as part of the setup process and allows Microsoft to accurately see the original sending IP address and verify the reputation of the sending domain of the incoming traffic. This improves the accuracy of our protection stack and machine learning models, and provides accurate verdicts for incoming email to the Office 365 environment. You can learn more about the importance of Enhanced Filtering for Connectors here.


Convert your evaluation to Standard protection

If you’re loving the capabilities offered by Defender for Office 365 and want to adopt this higher level of protection in your environment, you can covert your evaluation to Standard protection. Converting means you’ll enable preset security policies for the users you specify. Evaluation Mode will be turned off and mailflow will be impacted by these policies. This conversion can be completed in just a few clicks!


Figure 3: Convert an evaluation to Standard protection with a few clicks.Figure 3: Convert an evaluation to Standard protection with a few clicks.



Get started today with Defender for Office 365

To get started, you can kick off an evaluation today. For more information, review our documentation and let us know what you think! You can get the most out of a trial or evaluation of Defender for Office 365 using the trial playbook.



Do you have questions or feedback about Microsoft Defender for Office 365? Engage with the community and Microsoft experts in the Defender for Office 365 forum.



12020 IC3 Report, Federal Bureau of Investigation






Version history
Last update:
‎May 31 2022 09:34 AM
Updated by: