I cannot invite a external Guests to a external Yammer group. It only works through the admin panel.
Just keep in mind that the guests you're trying to add cannot be "consumer" addresses (ie. outlook.com, gmail.com, yahoo.com, etc). If you're still having trouble, might want to raise a support ticket.