Group Permissions

Background:

Step 1:  Created permissions level by coping Member and taking off "Delete" options.... Called it "Restricted Users"

Step 2:  Started creating user groups and set each user group permission as "Restricted Users"

Step 3:  Library "Working Files" there are several different folders and under each folder are several different files.  Depending on the folder / file, it depends on who can have the access. 

Step 3A:  So went to each folder and to "Manage Access" and stopped sharing with all except owners

Step 3B:  Added back the appropriate groups to each folder and file 

Issue:

A:  Went to add people to the individual groups and as soon as a person was added the group automatically went to "Limited Access" / "Restricted users" permissions and the added users could not access the site

B:  When I went to the folders to check on permissions, even though the group was set up with "Restricted Users" permissions, when they were added to the folder / file they are coming across with "Contributor" permissions.

What I "Want" to do is have groups so I can manage via that way instead of having to go to each folder / file and manage by person and still have to go in and change them from contributor to restricted user.

Really, any help is greatly appreciated.  Right now I have a total of 24 folders under 3 different librarys (total of 72 folders) and 83 files per library for a total of 249 files with 28 different people who will be accessing.  Some will have access to all, some to only a couple items, etc. 

Was hoping to manage this through groups as people can change and easier to add them to a group vs go to every folder / file they need access to and add them individually.

However, no reason to add groups if I have to change the permissions of the group to "Restricted User" each time it is added to a file / folder (might as well do it by person)

Any help is greatly appreciated.