@Bryan123 The "Edit" and "Read only" group are present in the scope of Site collection and there is only one copy of it. There is no different copy of the groups for the subsite or the list. So if users are added/removed from any of these 2 groups they get/loose access from both the subsite and the list (even though permissions are broken). In your situation, you have two options:
1. Grant users access directly to List: You can grant users read/ edit access directly to the list (without using groups). This is good if you have very less number of users and each list has its own unique permissions (i.e you dont have to replicate the same permissions in any other resource in SharePoint).
2. Create groups for each unique resource: Instead of creating just 1 edit & read only group, you need to create "Subsite Edit", "List Edit", "Subsite Readonly", "List read only". Grant the Susbite groups access to the subsite and List groups access to list. In this way when you add a user to Subsite readonly or edit group, they will not get access to List. You can rarely go wrong if you choose this method.
