Unable to access AVD servers from on prem via direct RDP

We have created an AVD solution and created a site to site VPN so the AVD's can connect into the network shares which is working a dream.   However the issue we are having is we cant ICMP or RDP to...