SPFX call gets error "interaction required" after turning on function's System Assigned Identity

I have an spfx web part that is connected an Azure Function. All was working until I turned on the System Assigned Identity for the function. Below is my configuration while it was working:

Function app: contoso-fn

• App Service authentication enabled
• Unauthenticated requests get a 302 HTTP code
• Microsoft is the identity provider, with app registration "contoso-fn-auth"
• It allows token audiences from api://<id-contoso-fn-auth> and from <my-tenant>.sharepoint.com
• Allowed client applications include 08e18876-6177-487e-b8b5-cf950c1e598c and <id-contoso-fn-auth>
• CORS: https://learn.microsoft.com/en-us/answers/questions/4374948/spfx-making-call-to-azure-function-gets-error-inte and https://<my-tenant>.sharepoint.com

App registration "contoso-fn-auth" (used above when setting up identity provider)

• Authentication > Web > Redirect URI: https://learn.microsoft.com/en-us/answers/questions/4374948/spfx-making-call-to-azure-function-gets-error-inte
• ID tokens issued
• only accounts in the tenant can access this API
• API permissions > contoso-fn-auth > user_impersonation ; Delegated; Status: granted for tenant
• Expose an API: Scopes api://<id-contoso-fn-auth>/user_impersonation

The code in the SPFX to request the function is something like this:

this.context.aadHttpClientFactory 
.getClient('<id-contoso-fn-auth')
.then((client: AadHttpClient): void => { client.get(<url-to-the-fn>/api/upload)...});

In the SharePoint Admin Center, under Advanced > API Access, the user_impersonation permission is approved for contoso-fn.

I then changed the Function app identity, turning on "system assigned", as I needed the function to authenticate against the Azure Blob Client, and now I'm getting the following error:

interaction_required: Seamless single sign on failed for the user. This can happen if the user is unable to access on premises AD or intranet zone is not configured correctly

I read something about turning on the function's system assigned identity affecting how I requests tokens using the aadHttpClientFactory, but I haven't been too successful in finding how to solve the issue (including internet search and asking several LLMs).

I don't need my function to have user context (I could send that from the spfx) but I need this to work without any redirects. I would appreciate help to be able to call the function from the spfx seamlessly, while keeping the function secured with Microsoft identity.

(porting from a https://learn.microsoft.com/en-us/answers/questions/4374948/spfx-making-call-to-azure-function-gets-error-inte, as requested)