Azure Virtual Desktop and VPN problems?

Hello,

I'm experimenting with AVD if it would be a viable solution to use instead of physical machines.

One problem I have is that when I enable the VPN from the 3rd party VPN Client, it disconnects me from the VM. This is because all traffic is routed through the tunnel, so it's quite expected behavior.

Now, I'm looking how to prevent this from happening and I have found a solution. Instead of using the 3rd party VPN client, I configured the VPN connection using the Windows' own VPN client, because that allows me to enable split tunneling for the VPN profile. This isn't quite enough though. The second thing I have to do is create static routes for the public IP addresses that AVD uses, so that all traffic to those get routed through the VM default GW. I got the IP's from this document:

Download Azure IP Ranges and Service Tags – Public Cloud from Official Microsoft Download Center

After that everything seems to work. I can enable the VPN without getting disconnected and work like I would on a physical machine. Great success!

BUT, if the session times out or if I close the browser, for some reason I can't connect back to the VM...

What happens is that while connecting it asks for my credentials as normal, but then gets stuck on the "Establishing secure connection" part.

Could I still be missing some necessary IP's from my configuration?