I worked with 365 Support on this today and they recommended I create a post here, hope it is in the right place.
I have allowed Yammer to link with Sharepoint Groups in the Network Admin tab. After this I've created an Internal Yammer Group which then creates a Sharepoint Linked site and others connected resources like Azure AD, Planner, OneNote, etc. Within Azure AD this group is visible so I add already created Guest Users to the group. They are able to access Sharepoint, Planner, and OneNote Group resources but not Yammer. Yammer still shows only myself as the member and I am unable to invite them.
The only way to invite them is if I go into Yammer Settings > Network Admin > Invite Guests. The trouble with this is that if you invite them this way it will add them to the "All Company" Yammer Group and give them the ability to join any other Yammer Group.
My expectation of how this should all work is that since Yammer seems to be connected to every other aspect of Office 365 like Sharepoint, OneNote, Planner, Azure AD, etc that is should bring that user into Yammer.
If I am forced to add a user to Yammer through the Invite Guests feature then I should be able to Invite them only to specific groups in Yammer.
It should be noted that even if I add in Azure AD a non Guest, so a user in my Office 365 Domain, it still doesn't get applied to Yammer but does get applied to every other 365 connected service.