@JamianP Rubbish isn't it... we have done the same over the last year and moving everything to the cloud... trouble with tech (I'm the IT guy at my company) :(
Just had a phone call about it and they did recommend to change or update the phishing policy as a few people have reported the same issue. They suggested to make a new phishing policy/ turn off existing ones (not the greatest fix, might just need to hold on until it fixes itself but its below for you):
As discuss this are the steps how to disable the phishing settings.
In the Security & Compliance Center, go to Threat management > Policy > ATP anti-phishing.
On the Anti-phishing page, click Create.
The Create a new anti-phishing policy wizard opens. On the Name your policy page, configure the following settings:
Name: Enter a unique, descriptive name for the policy.
Description: Enter an optional description for the policy.
When you're finished, click Next.
On the Applied to page that appears, identify the internal recipients that the policy applies to.
