InMemoryTokenCache for Microsoft Graph API

Hello everyone,

I am working on a project that involves Angular SPA calling a protected ASP.NET Core Web API. The frontend is using MSAL.js for user authentication and authorization, and the backend implements the logic to validate the token and for other security purposes. 
builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddMicrosoftIdentityWebApi(options =>
{
builder.Configuration.Bind("AzureAd", options);
options.Events = new JwtBearerEvents()
{
OnAuthenticationFailed = c =>
{
Console.WriteLine(c.Exception.Message);
return System.Threading.Tasks.Task.CompletedTask;
}
};
}, options => { builder.Configuration.Bind("AzureAd", options); })
.EnableTokenAcquisitionToCallDownstreamApi(options => builder.Configuration.Bind("AzureAd", options))
.AddMicrosoftGraph(builder.Configuration.GetSection("DownstreamAPI"))
.AddInMemoryTokenCaches();

Now, I was trying to use Microsoft Graph API and call some of the basic delegate apis (User.Read) that don't require admin consent. Locally, everything is working fine but when I deploy the web api to azure app service it throws the following exception:
An error occurred while calling the downstream API IDW10502: An MsalUiRequiredException was thrown due to a challenge for the user. See https://aka.ms/ms-id-web/ca_incremental-consent.
I know this exception is thrown when we are trying to call the api that requires admin consent. But I am not sure why it is throwing that exception for basic API when deployed to azure app service.
Does it have something to do with InMemoryTokenCahce()?
Thanks for any help in advance!