How to set Different Policy set for Different Apple Devices with Endpoint/InTune?

So, to make sure I understand you correctly (just making things up here, it's about the structure and most how things are assigned). 

Policy Set "Manager"

Assigned to the virtual "All devices" group.

• Configuration Profile "Manager"
  Assigned to "Managers" ("All users") group
• Compliance Policy "Manager"
  Assigned to "Managers" ("All users") group

Policy Set "Staff"

Assigned to the virtual "All devices" group.

• Configuration Profile "Staff"
  Assigned to "Staff" ("All users") group
  
• Compliance Policy "Staff"
  Assigned to "Staff" ("All users") group

You are already assigning the Configuration Profiles and Compliance Policies to the groups directly (which answers my question ).

I don't think you even need Policy Sets right now, so I suggest you remove them from the equation to reduce complexity. As you already removed the separate items from the Policy Sets and they're still not working, start troubleshooting them one by one, starting with the most simple setup.

Finally, just a little afterthought: are you sure your Apple devices are enrolled with user affinity? If not, you can't assign anything to users.