Could you elaborate a bit more? You can't add permissions to a group in Stream, so I'm assuming you added someone to the actual Office 365 group which has vidoe's in the attached group in Stream you don't want them to access? You will need to remove them from the actual group to remove them from that permission.
Let me know a bit more about the scenario and what you can't get working and we'll see what we can do.