First, I recommend that you restrict access to the Azure AD portal for ANY user. And yes, Guest users can also access the Azure AD Portal and see a list of all users if you haven't restricted this.
You can do this by browsing the Azure Portal > clicking on User settings > and choosing to Restrict access to the Azure AD administration portal.
To answer your question, can you verify that someone is logged into the Azure Portal? To do this, browse the Azure Portal > Choose Sign-in logs and create a filter > Application contains Azure Portal. Good luck!