Limit access to AVD by IP

We are using AVD for a vendor, they will be using an Azure AD account and the AVD is not connected back to our AD or network. I need to limit the IP space that can connect to AVD. With a standard az...