Visitors are displaying as visitors and members

I had a very similar issue and have found a solution. Hope it works for you!

We have a public sharepoint site that we would like owners to have full permissions, members have edit perms, visitors have read perms, and external users no access at all. We also have a document library within this site that we have titled Private Docs and would like to keep that library only visible to owners and members, but NOT visitors.

Unfortunately, Sharepoint seems to default to a setting that assumes everyone is a member except external users.

To fix this, I had to visit the advance permissions settings for "members" and remove the setting "Everyone except external users". In addition, to enable site visitors to have read access, I wen't into the "visitors" setting and added the correct setting by searching "everyone except external users" and adding it.

To keep the document library private, even from visitors, I went into "permissions for this document library", used unique permissions, then removed visitors.