Can't add users to site through group permissions

All the owners of one Sharepoint site in our environement get an error when adding users to a site. We are clicking invite users, add members to group then we get the error. The error reads: Sorry, we couldn't display all the group members. Please try again. There are no special groups setup just owners, members, visitors.