Microsoft
MicrosoftFonsecaSergio Thanks for the blog.
Hopefully you will do more of these connectivity blogs. We need one that discusses outbound HTTPS from spark-pool-executors to private resources (REST API). This is virtually an impossible task today, since it involves a very wide assortment of expensive components (load balancer, NAT VM, private link service, etc). Given how common REST API's are these days, it seems like it should be a priority to allow our Spark pools to reach an API on the private VNET. Our Spark pools run inside of a "managed vnet", but my understanding is that the configuration is very normal.
I suspect Microsoft has a technology that is supposed to make this easier some day (called "Azure Application Gateway Private Link"). But it is in preview, and I'm told it can take many, many months for the Synapse Analytics workspaces to introduce a compatible MPE, so that we are able to reach out to a new type of private link.
