AVD Conditional Access random failure

Dear Community,

For some time now we have been working to solve a strange problem that occurs randomly.

Situation:

User logs into a Cloud Only Windows 11 workstation using the office365 UPN.

Then the user logs in to the Azure Virtual Desktop environment via the Remote Desktop Client (SSO) The AVD environment is a Hybrid Azure AD Joined environment.

This is where things occasionally go wrong. The user enters a loop and cannot continue logging into the AVD environment. Only option to get the user out of the loop is to completely reset his MFA configuration.

In the Sign-in log the following error is displayed:

Sign-in error code: 50206

The user or administrator has not consented connecting to the target-device: identifier. Send an interactive authorization request for this user and target-machine.

We have had a ticket of this with Microsoft for some time but we are not really getting anywhere.

What have we tried?

1- Configured Conditional Access based on Microsoft learn:

   https://learn.microsoft.com/en-us/azure/virtual-desktop/set-up-mfa

2- Legacy per user Multi Factor Authentication disabled.

3- Azure Windows VM sign-in app excluded from Conditional Access.

So far without success, users are stuck in the loop (random)

Hope you guys can help us further with tips / suggestions.

Thanks in advance!