Introduction
A large enterprise customer set out to build a generative AI application using Azure OpenAI. While the app would be hosted on-premises, the customer wanted to leverage the latest large language models (LLMs) available through Azure OpenAI. However, they faced a critical challenge: how to securely access Azure OpenAI from an on-prem environment without private network connectivity or a full Azure landing zone.
This blog post walks through how customers overcame these limitations using Application Gateway as a reverse proxy in front of their Azure Open AI along with other Azure services, to meet their security and governance requirements.
Customer landscape and challenges
The customer’s environment lacked:
- Private network connectivity (no Site-to-Site VPN or ExpressRoute). This was due to using a new Azure Government environment and not having a cloud operations team set up yet
- Common network topology such as Virtual WAN and Hub-Spoke network design
- A full Enterprise Scale Landing Zone (ESLZ) of common infrastructure
- Security components like private DNS zones, DNS resolvers, API Management, and firewalls
This meant they couldn’t use private endpoints or other standard security controls typically available in mature Azure environments.
Security was non-negotiable. Public access to Azure OpenAI was unacceptable. Customer needs to:
- Restrict access to specific IP CIDR ranges from on-prem user machines and data centers
- Limit ports communicating with Azure OpenAI
- Implement a reverse proxy with SSL termination and Web Application Firewall (WAF)
- Use a customer-provided SSL certificate to secure traffic
Proposed solution
To address these challenges, the customer designed a secure architecture using the following Azure components:
Key Azure services
- Application Gateway – Layer 7 reverse proxy, SSL termination & Web Application Firewall (WAF)
- Public IP – Allows communication over public internet between customer’s IP addresses & Azure IP addresses
- Virtual Network – Allows control of network traffic in Azure
- Network Security Group (NSG) – Layer 4 network controls such as port numbers, service tags using five-tuple information (source, source port, destination, destination port, protocol)
- Azure OpenAI – Large Language Model (LLM)
NSG configuration
- Inbound Rules: Allow traffic only from specific IP CIDR ranges and HTTP(S) ports
- Outbound Rules: Target AzureCloud.<region> with HTTP(S) ports (no service tag for Azure OpenAI yet)
Application Gateway setup
- SSL Certificate: Issued by the customer’s on-prem Certificate Authority
- HTTPS Listener: Uses the on-prem certificate to terminate SSL
- Traffic flow:
-
- Decrypt incoming traffic
-
- Scan with WAF
-
- Re-encrypt using a well-known Azure CA
-
- Override backend hostname
- Custom health probe: Configured to detect a 404 response from Azure OpenAI (since no health check endpoint exists)
Azure OpenAI configuration
- IP firewall restrictions: Only allow traffic from the Application Gateway subnet
Outcome
By combining Application Gateway, NSGs, and custom SSL configurations, the customer successfully secured their Azure OpenAI deployment—without needing a full ESLZ or private connectivity. This approach enabled them to move forward with their generative AI app while maintaining enterprise-grade security and governance.
Microsoft