AAD joined session hosts can be RDP'ed to directly

We've recently setup AVD for a client and AAD joined their session hosts. When testing one of the user found an interesting issue - AAD Joined hosts require Virtual Machine User Login RBAC assigned at the resource group level to allow them to login. This also allows them to directly RDP from an on-prem network that has Site-to-Site connection outside of AVD brokers. This allows users to find the IP of a machine and just log in to it, which would essentially throw the algorithms. This sounds like a gap in the solution. This doesn't happen on AD joined machines unless an Administrator tries to RDP.

Can someone let me know if we're missing anything here?

Thanks!