Secure a VM-based web server sitting behind Azure Front Door + WAF from the internet

Hi All

I have a web server that's running on a VM inside Azure. The server is exposed to the internet through an NGINX reverse proxy and a public IP address.

For added security, I've set up an Azure Front Door, incorporating an Azure Web Application Firewall (WAF), which works fine so far.

My issue is that I can't see how I'm supposed to stop internet traffic connecting directly to my internet-facing NGINX proxy. Without putting restrictions in place (I'm not sure what these would be), there's no reason for anyone to access the server via AFD as they could just go straight to the server. This completely defeats the purpose of AFD + WAF (apart from load balancing, etc. features which I'm not using).

Has anyone worked this out? It doesn't appear as though Microsoft has thought this through???

Cheers,

Josh