Restrict communication of a guest only to the members of the team he was added to

We have noticed that a guest of a Team can chat with users of the tenant he is a guest in, even if they are not members of the team he was added to. Once he types the email of any user he can chat with them.
The result is that when adding a guest - he is able to communicate with all company members and not only the team he is a guest in.
Is there an option to restrict this