Microsoft Teams Call on Azure VDI with no Audio selected for the call still come out speakers on MAC

I join a MS Teams call with no audio option as I want to join audio from my phone but then on MAC Microsoft App client I can still hear the audio coming out my local Mac speakers. This then required I completely lower my mac volume level which then causes other problems.