Avatar Trigger Shared State Issue

My use case is to make 'break-out rooms'. By entering a collider the user gets a discussion topic and question to discuss with the other avatars present in the same collider. In mesh there are two ways to handle trigger collider logic. Avatar triggers happens locally, while on trigger enter gets networked to all avatars. I don't want to use on trigger enter because only the avatar that enters should see the hidden content. Not all avatars in the scene.

However, using avatar trigger allows the avatar who enters any trigger collider to see all hidden content. Putting conditions, such as limiting the interaction to a specific trigger collider, does nothing as it activates all available avatar trigger component directly. Which means that all avatar trigger components share the same state and when one is set to True then all of them get set to True and their interaction happens.  This makes me unable to get a workaround through code. Is there something I have missed?