Phishing Filter - M365 ATP - false positives blocked or clear spam messages get into inbox.

[StephanGee]

Hi everyone,

we are having problems with our filter. We used Sophos UTM before but switched our MX now to M365 as there are 99% of the mailboxes.

But we don't really get the logic behind the filtering service.

Yesterday a mail got through that was clearly send from outside of Office365 with the "from" mailadress of our CEO which got straight to inbox. We have setup SPF!

Also those fake Office365 always get through. Even with dynamite phishing.

Are there best practices available for setting this up?

Some examples which should go straight to junk or quarantine.

Do you other IT Pros have similar experiences?

Best regards

Stephan

[#2]

Have you verified from the mail that was sent out from the CEO?
have you checked headers to see where it was sent and why it might have passed your spam filter?

I assume you haven't whitelisted your own domain?


I highly recommend using the ORCA module to verify that your ATP is setup according best practices.
https://www.powershellgallery.com/packages/ORCA/1.6.3

[#3]

@Thijs Lecomte 

Hi. Thanks for your quick email. 

I checked the message header and it was received from an internet provider (not ours) from Germany. So it should have been blocked.

We did not add our domain on the allowed list ( i checked ;) ) and the sender ip is not in our allowed IP list in the mail rule.

Thanks for the tip with ORCA. I will check this tool.

---

Cannot install it on my Admin machine though

Name : ConsoleHost
Version : 5.1.14409.1018

Can be installed on my Win 1909 machine (but from there i am not allowed to perform this ;) )

Error:

WARNING: Source Location 'https://www.powershellgallery.com/api/v2/package/ORCA/1.6.3' is not valid.
PackageManagement\Install-Package : Package 'ORCA' failed to download.

[#4]

Best thing to do here is open a support case and work with the engineer to find out why the message was not blocked.

[#5]

@Vasil Michev 

Yes i think i will do that. Because what Message Analyzer says - i would say 100% Junk. MS says -> straight to Inbox

[#6]

@Vasil Michev 

We implemented all best practices (if suitable).

Now "internal mails" to external are filtered as SPAM (highest number 9).

How can internal mail be marked as spam? They are normal responses to mails.

These are filtered - but mails like these still coming through:

At this moment our Sophos UTM was a better filter than Office ATP - as we needed about 20% of the resources to manage it.

[#7]

@Thijs Lecomte@StephanGee

Is there any possibility to check how the SPAM score was calculated?

eg.

Score starts at 5

SPF ok -1 points

DKIM passed -1 Point 

20 mails in the past 1h +1

= 4

?

[#8]

Not as far as I know
Such things probably aren't published in order to protect the inner workings of ATP

[#9]

@Thijs Lecomte 

Thanks for your fast answer. 

We have so many cases with false positives that we might just make more internal trainings and set the spam recognition lower.

Current case:

Normal mails from customer is received fine

Teams meetings are sent to quarantine. No special configuration (no picture, no disclaimer, no custom help page)

Customer uses M365 like us

One difference is, that a special character in the name of the customer is recognized in normal mail and in Teams meeting not.

[#10]

Could you check the spam header why it's being blocked?
https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/anti-spam-message-header...

[#11]

@Thijs Lecomte 

This are the information is from the message analyzer (blocked teams meeting):

You can see in the senders display name the wrong character:

When she is sending mail - the display name is ok. I now asked her if she is creating the teams meetings from the teams client or the outlook addin.

[#12]

There is no way to know from the headers what is causing is.
Best you open a case for this, they might be able to check in the backend

[#13]

I have to push it back up.

Literally any newsletter is landing now in quarantine due to Phish. This is an authentication result of one of them:

spf=pass (sender IP is 185.71.127.155) smtp.mailfrom=u106878.rmh1.net; mycompany.com; dkim=pass (signature was verified) header.d=rmh6.net;mycompany.com; dmarc=fail action=none header.from=amcham.de;compauth=fail reason=001

SPF right, DKIM pass. 

Why not send this mail into "Junk" but into "Quarantine" where to user has to release it?

Spam Level is "5" for this. 

This takes up a lot of time ... my colleague also went through some support calls but this is just a waste of time.

"It is what it is"