This is where Front Door (Microsoft's Dynamic Site Acceleration offering) comes in.
What you find is actually your site is performing OK, and see descent times in your origin logs. That's where looking at the wholistic HTTP request is required and points you in the direction of where the time is spent. I'm using a handy tool, https://tools.keycdn.com/speed, to illustrate the full request. The key here is the Browser takes time to do a few things before it can hit your server:
TLS Handshake (SSL)
And the "further" you go from your origin, and further isn't necessarily miles/kilometers physically away, but it's how many hops do you need to traverse the public internet to your origin. So when we break down a simple request for an image without Front Door (German => Central US):
Notice that ~60% of the total time the client is waiting for the Image is just done in the upfront TLS parts (ref Transport Layer Security (TLS) handshake). The several roundtrips between client and your origin can be extremely costly.
With Front Door, you greatly reduce those upfront interactions, since the client Browser will be interacting with the "closest" POP (Point of Presence) that is advertised thru BGP by the Front Door Service. Front Door will be using the Microsoft Global Network to traverse across the planet to the closest location to your origin from where the client's request enters. That normalizes your latency (you can see the average published here: Azure Network Latency).
So now we look at with Front Door:
DNS in this example is faster, but we aren't changing the DNS service you are using. The important 2 to look at are Connect and TLS. The rest of the request (Send, Wait, Receive) can also be helped, but isn't the focused. Front Door will create a pool of persistent connections to your origin which optimize aspects of each request. By acting as a proxy, it safely uses those pooled connections to your origin. This reduces the overhead on your origin of establishing and cleaning up individual connections per client.
To get started, here is a Quick Start to create your first Front Door!