Adding users to a folder but with limited access to subfolders

I need to add five users to a specific folder(which has 9 subfolders), but some users should not have access to first three subfolders. How do I control the access to individual subfolders? I've tried user group but i didn't find anywhere how to connect group with folder. I cant change the folder structure, it must be the way it is right now.

Note: they are all external emails and not part of my organisation.