Need to limit chat access to only subset of meeting internal participant is actively dialed in for

I have the following situation:  

I organize multi-day exec meetings for which there are guest participants (mostly internal) who typically dial in for a single session then leave the meeting.

My requirement is for these guest participants to only have access to the chat while they are actively in the meeting (ideal) or not at all.  Guest participants should not have access to chat that happens around any topics prior to them joining nor after them leaving.

I would be happy to manually remove the guest participants from the chat after their session has ended, but it seems the only option is to "remove from meeting" which still allows them access to chat.