Mark-Kashman I've done that multiple times over the last month. I've added everyone by inviting them as you describe. I've tried adding everyone directly to the visitor group through advanced site permissions. I've even added them directly at the site level without a group at all. Sometimes it lasts a few days, maybe a week but inevitably, the day comes when I get an access request and when I check the site, everyone is gone. All that's left is the O365 group in members. Today I had to add everyone 3 times before lunch.
Just for kicks, I've added an individual to the visitor group and his access has been stable. It's only "everyone except external" that gets booted from the visitor group.
My IT raised a ticket with "Microsoft" (whatever that means) and supposedly "Microsoft" told them it's because my O365 group is private and I need to make it public. First, that makes no sense. Why would the status of my group affect visitors? But more importantly, I don't want a public Team. I want a public site with a private Team so that when people request access to the Team, I get a notification and can send them a Welcome to the User Group message.