Microsoft Secure Tech Accelerator
Apr 03 2024, 07:00 AM - 11:00 AM (PDT)
Microsoft Tech Community

Phishing attack simulator incorrectly emails people the message, "Because you were recently phished"

Copper Contributor

Hi folks,

 

* I am evaluating Microsoft Phishing Attack Simulator with a 4 user pilot

* None of the 4 users were phished in any of the 3 simulations that I actioned

* At the end of each simulation, users are correctly being emailed a message with a link to phishing traning

* However, the email with the link to the phishing training contains this wording:

 

"Because you were recently phished, we require you to take training(s) to recognize phishing attacks in future."

 

* The wording I quote is troublesome since it:

 

a) Is inaccurate; none of the users were phished

b) Presents me (and potentially my colleagues in IT Support), negatively, since it makes us look like we aren't in control of the simulation technology (ie it looks like we don't understand the reality of how each user responded in the simulations)

c) Risks alienating us from our users

 

My questions thus are:

 

1) Is anyone else impacted with this issue?

2) Is there a way for the wording I refer to, to be constructively edited?

 

Any help is always appreciated.

 

Regards,

Steve

19 Replies

You can edit the message under

 

  • Attack Simulation Training > End-user notifications > Tenant Notification 
  • Choose the notification (e.g. "Microsoft default simulation notification")
  • On the Define Content section you can choose the language you want to edit 
  • Edit the content & Save

I like there's different level of triggering and education.  If the email is opened, Microsoft considers that phished.  There's also different trainings for a) if the link is clicked and b) if credentials were supplied or file was downloaded.  

 

I hope it helps.  I think the tool is still lacking a lot of features still, but for us it's better than nothing.  User reporting is weak and I can't find the life of me how to remind users to take the training (if that's even possible).  

 

To take your points in order:

(a) in your pilot test, did you specify training for all recipients, those who clicked the payload or those who were fully compromised? What notification option did you choose? One problem with the simulator (as I last used it in December) is that a lot of the minor settings are not recorded in the simulation list. If for example you want a record of what training you have assigned, you are going to have to keep a manual record.

(b) is a danger. You have to recognise that the MS phishing simulator is a product being continuously improved, and if all of the changes are being announced in the O365 message center then I'm missing some of them. I use the simulator quarterly, and every time last year there were some new changes to take into account. I have even seen changes arrive in mid-simulation. Your only option is to test in advance, and once more just before you launch. Unless your security stack is MS from top to bottom, you need to do that anyway in case another security vendor has suddenly decided your chosen phishing URL is malicious.

(c) is a danger with any attack simulator. Weeks before you launch, you might consider sending out a general mail reminding recipients of the dangers of phishing and that regretfully the organisation has no choice but to conduct simulated tests for all staff. Explain that this is something all proactive organisations are adopting, and that the object is to train rather than catch people out. They will face the same dangers with their personal addresses. Be constructive and helpful; I use payloads of varying "difficulty" so no-one feels bad about falling for the trickier ones.

Check the Settings tab on the Attack Simulator page of the Security portal. The default setting for reminders is Off.

In answering myatkaw, I have just seen the End-user notifications tab on the same portal. It seems that it is now possible to edit a copy of the offending Microsoft default simulation notice, but in starting a test simulation I only saw the option to choose a variant positive reinforcement notice.
Thanks Myatkyaw for your constructive reply,

"If the email is opened, Microsoft considers that phished"

* This comment is very interesting
* I remark so because I encourage our users to report phishing emails using the feature to do so in Outlook
* From what you are saying, every person who reports one of the phishing emails in the simulation will be marked by the simulation tool as having been phished?

Regards,
Steve
Definitely not, Steve. The simulator records if the e-mail was delivered (and it's not a conventional delivery; the simulator stuffs the phish directly into the recipient mailbox), whether the recipient clicked on the phishing link and whether a credential was supplied. It also records whether the phish was reported and whether training was done. It might do more for some of the other payload types, but at the moment I mostly work with drive-bys and harvesters.

Use the View Users link in the simulation report to download a worksheet of the tasty details.
I think it's a matter of documentation in an evolving product. Features are getting added faster than there's education and documentation. I'm still trying to wrap my head around Defender products renaming and regrouping (ATP is now Identity, MCAS is Defender something). Nowhere in Attack Sim documents on Docs.Microsoft does it tell you how to edit the response files. It's just something I stumbled on. Luckily, there's a community here that can help each other out.
""If the email is opened, Microsoft considers that phished"... sorry, allow me to elaborate.

I think it is a good feature, but wordings could be better by Microsoft. Opening and reading the email is a level of susceptibility. I think traditional definition of "phished" is credentials were stolen or a malware file was clicked. I think Microsoft considers phished at 3 levels: 1) if an email is opened - i could be wrong on this 2) if an embedded link was clicked 3) if credentials were supplied or file was executed. Depending on susceptibility, customized education would be generated and sent. I hear what you're saying though.... Phished in my vocab before is compromised.
If I recollect correctly only the drive-by URL is instant death; the other payload types only count as full compromise if the recipient completes the chain of actions required by the payload. Note that the credential harvester does not verify that the password given is correct (a service Microsoft are in a unique position to offer).
Hi ExMSW4319,

Thanks for your reply :)

1) Please pardon me all for focusing very closely on a point of jargon semantics, as this point underpins so much of what I understand and what I am confused about.

If a user opens the phishing email, does Microsoft report this as the user being phished?

2) From the above question, I've looked at how I direct users to respond to phishing emails.
* We are a small humanitarian NGO, so we get free licenses for web M365 (thanks Microsoft :) )
* We have though approx 40 users who need the additional functionality in locally installed M365
* Hence we in IT Support, support M365 delivered to the users via browser and via local install
* I put together an intranet page with instructions for users on how to respond to phishing emails
* There are separate instructions for reporting phishing emails, based on whether using web M365 or locally installed M365
* I now see my instructions on the web are sub-optimal (I am keen to rectify this quickly)
* In both cases, the instructions direct the user to use the Report Phishing feature available after the phishing email is opened
* However, I now see that in web M365, right clicking the unopened message produces a menu that leads to an option to report the phishing email
* Is there a way to report an unopened phishing email using locally installed M365?

3) Point 2. feeds back into point 1. If MS Attack Simulator interprets an opened email as phished, all our users who have locally installed M365 will be interpreted by Attack Simulator as phished if they open a phishing email as a mandatory step as part of the process to report a phishing email. It would be great if there's a way to report a phsihing email without needing to open the phishing email 1st in locally installed M365.

Any help is always appreciated :)
Steve, to clarify my earlier answers, the simulator records if a simulated phishing mail is delivered but it does not record if it is opened. It does record an initial click-through of the phishing link in the mail but as far as I know it only records this as being "phished" (full compromise) for payloads of the the drive-by URL type.

As I said in an earlier reply, the simulator is being developed at a pace and is therefore subject to change. If you still have doubts, send a series of test simulations to yourself and see what happens in each case when you just open a simulation, open and click the initial link and in the final case complete the whole sequence of recipient actions.

That will also give you a chance to test landing pages, indicators and all the new groovy reinforcement mails that are being added to the simulator. In my last test, these didn't arrive. :(
Hi ExMSW4319,

Thanks for your reply :)

* Reasons I sought clarification are:
1) "the simulator records if a simulated phishing mail is delivered but it does not record if it is opened."
This is different to the reply @myatkyaw was kind enough to offer

2) I'm in a situation at the moment where I am struggling to trust the reports in the Simulator tool
* I logged a separate post for this particular issue

https://answers.microsoft.com/en-us/msoffice/forum/all/microsoft-phishing-attack-simulation-not-repo...

I'm hoping Microsoft will give me some help on that.

Regards,
Steve

@SteveCRF I think @ExMSW4319 is correct on reporting delivery, and not what I said about opening the email.  The supporting logic is reasonable, and more probable.  

 

Hi @myatkyaw

 

* I have endeavoured to follow you proposed steps to edit the wording in the email that gets sent to users re training

"

  • Attack Simulation Training > End-user notifications > Tenant Notification "

* After I go to 'Attack Simulation Training', there doesn't appear to be an 'End-user notifications' option

 

attack simulation.PNG

 

* Text search on 'end-' and 'notifications' doesn't find anything

* Any further suggestions please?

* Any help finding a way forward from anyone is appreciated :)

@SteveCRF - you might want to check your roles in that case. Here's what I see, and I am not a global admin either:

 

End User Notifications.PNG

Thanks @ExMSW4319 for your very enlightening screenshot.

* I've quite recently been made a Global Admin
* I suspect stuff I don't see compared to what you see when looking in the phishing attack simulator may be due to licensing

https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/attack-simulation-traini...

"If your organization has Microsoft 365 E5 or Microsoft Defender for Office 365 Plan 2, which includes Threat Investigation and Response capabilities, you can use Attack simulation training in the Microsoft 365 Defender portal to run realistic attack scenarios in your organization."

* Rather than the licenses mentioned in the above quote, we have community licenses for NGOs.
* If the other post I've mentioned in this thread leads to a resolution, it may be an appropriate time to try this:

https://docs.microsoft.com/en-gb/microsoft-365/security/office-365-security/about-defender-for-offic...

Thanks both of you for having helped on this, it's been very constructive.

Regards,
Steve
I suspect you're right about the licensing -- We have E5.
I'm assigned the following roles:
Attack Simulation Administrator
Security Operator
Security Administrator
I have discovered this exact annoyance myself. You can create a custom Simulation Notification to remove the troublesome wording of 'You have been Phished' but then there is no way of setting that as your simulation notification in the attack.