Meetings and Calls notifications muted but still displaying

We've tried using the new feature in Notifications for Meetings and Calls to block notifications while in a meeting or a call. We've turned this on then tried calling while in a meeting (created in Outlook). The status in Teams changes to In a Meeting. When making a test call, the notification pops up and the call rings despite having Mute notifications during meetings and calls turned on. 

Is there something more we need to do to get this feature to work?