401 unauthorized as guest user using Microsoft Graph

Our company is using SharePoint as a client portal and our clients are set up as guest users in our Azure AD. When the client logs into their SharePoint site there's a web part that shows a picture of an employee within our company. I've tried using msGraphClientFactory and aadHttpClientFactory and they both seem to work but only when people within our company log in. When a client logs into their SharePoint site they can't see the picture and it displays the following error even though SharePoint API access has been granted to User.ReadBasic.All in the SharePoint admin area.

Is there something I'm missing or does Microsoft Graph not work for guest users? Note: the guest user email I am testing with ends in @outlook.com but the client's email will be @company.com. Not sure if that makes a difference.

EDIT: https://docs.microsoft.com/en-us/graph/permissions-reference#user-permissions and it says Microsoft accounts don't have permission. I'm guessing that's what's causing the issue since I'm logging in with an @outlook.com account?

Error:

https://graph.microsoft.com/v1.0/users/email_of_our_employee/photo/$value 401 (Unauthorized)

Here is the code I added to package-solution.json:

    "webApiPermissionRequests": [
      {
        "resource": "Microsoft Graph",
        "scope": "User.ReadBasic.All"
      },

Here is the code added to get the picture:

Context.aadHttpClientFactory
                .getClient("https://graph.microsoft.com")
                .then((client: AadHttpClient): void =>
                {
                    client
                        .get("https://graph.microsoft.com/v1.0/users/" + EmailAddress + "/photo/$value", AadHttpClient.configurations.v1)
                        .then((res: HttpClientResponse) =>
                        {
                            res.blob().then((result: any) =>
                            {
                                const url = URL.createObjectURL(result);
                                resolve(url);

                            });
                        }, (error: any): void =>
                        {
                            reject("GetUserImageURL Error: " + error);
                        });
                });