Integrate private access to your Azure Open AI Chatbot
Published Nov 30 2023 04:03 PM 14.3K Views
Microsoft

Do you want to create your own private (non-Internet exposed) chatbot using Azure Open AI (AOAI) services with your company-specific data?  In this blog post, we will walk you through the steps to integrate the solution into your Azure environment and secure from public exposure.

InternalAOAI.jpg

Figure 1: Reference Architecture for internal chatbot using Azure AI Services

For a high-level reference of the Azure Landing zone integration and resource details see Azure OpenAI Landing Zone reference architecture (microsoft.com)

 

This post is based on the ChatGPT + Enterprise data with Azure OpenAI and Cognitive Search Demo, which demonstrates how to build a chatbot using Azure AI services. See the link for more details and instructions on how to run the demo. The diagram above details the infrastructure components needed to accomplish this, a hub connectivity network and application spoke that host the AOAI app.

 

The solution achieves the following goals:  

  • Accessing the Azure Open AI chat bot only via the corporate network, without exposing it to the internet. 
  • Protecting your enterprise data from public access. 
  • Guidance on integrating the Azure Open AI solution following Azure best practices.

 

Prerequisites

Azure

  • Azure Account. If you don't have an Azure subscription, create a free account before you begin.
  • Azure subscription with access enabled for the Azure OpenAI service. You can request access with this form. Models are subject to region availability. You can review the list here.
  • The deployment must be started by a user who has sufficient permissions to assign roles, such as a User Access Administrator or Owner.
  • Your Azure account also needs Microsoft.Resources/deployments/write permissions at the subscription level.
  • IP scheme that is not overlapping with on-premises and subnet plan

On-Premises

 

Subnet Planning

The choice of CIDR blocks for your subnets depends on various factors including the size of your network, the number of resources you plan to deploy in each subnet and your future scaling needs. The following are the general guidelines for this architecture however your specific needs may require a different approach.

Component

CIDR Block

Description

Virtual Network Gateway

/27

This subnet covers all SKUs requirements.

DNS Private Resolver Inbound

/28

This subnet satisfies the minimum requirements.

 

DNS Private Resolver Outbound

/28

This subnet satisfies the minimum requirements.

 

Application Gateway

/24

This subnet ensures that Application Gateway v2 allows for autoscaling.

Web App Services Inbound

/24

This subnet allows for scaling out.

Web App Services Outbound

/26

This subnet covers issues with capacity.

APIM

/28

This subnet efficiently maximizes Basic and Standard SKU scale-out limits. /27 for Premium SKU

AI Services

/24

This subnet is used for deploying AI services, such as cognitive services, machine learning, etc.

Data

/24

This subnet is used for storing and accessing data, such as databases, storage accounts, etc.

Shared

/24

This subnet is used for shared resources, such as Key Vault, jump box, etc.

 

Connectivity between On-Premises and Azure

There are two options for connecting Azure to on-premises networks: Azure ExpressRoute private peering and VPN Gateway. Both solutions allow you to send encrypted traffic between Azure and on-premises locations over a private or public connection. However, there are some key differences that you should consider when choosing the best option for your organization.

  1. Azure ExpressRoute private peering extends on-premises networks into the Microsoft cloud over a private connection. This option provides high bandwidth, low latency, and predictable performance. Steps to setup ExpressRoute
    jsheerin_1-1701372903530.png
    Figure 2: ExpressRoute Setup
  2. VPN Gateway sends encrypted traffic between an Azure virtual network and an on-premises location over the public Internet. You can also use VPN Gateway to send encrypted traffic between Azure virtual networks over the Microsoft network. Steps to setup S2S VPN

jsheerin_2-1701372903536.png

Figure 3: Site-to-site VPN Configuration

 

Comparison and Recommendations

You can use either ExpressRoute or VPN Gateway to connect Azure to on-premises networks, depending on your requirements and preferences. Here is a side-by-side comparison of the two options based on some common criteria:

Criteria

ExpressRoute

VPN Gateway

Bandwidth

Up to 100Gbps

Up to 10Gbps

Latency

Low and consistent

Higher and variable

Availability

99.95% SLA

99.9% SLA

Cost

Higher

Lower

Complexity

Higher

Lower

Security

High

High

For a more comprehensive comparison, see About Azure VPN Gateway.

Generally, you should choose ExpressRoute if you need high performance, reliability, and security for your connectivity. You should choose VPN Gateway if you need lower cost, simplicity, and flexibility for your connectivity. You can also use both options together to create a hybrid solution that meets your specific needs.

 

Azure Application Gateway with Web Application Firewall

Application Gateway is a layer-7 (application layer) load balancer that manages traffic to web applications. It distributes application traffic to the app services backend pools; provides Web Application Firewall integrated capabilities, SSL/TLS termination, and session affinity; and helps to ensure that all users of the Azure OpenAI APIs get the fastest response and highest throughput for model completions. Steps to setup Application Gateway

jsheerin_3-1701372903546.png

Figure 4: Application Gateway functions

Azure Application Gateway works together with Azure Web Application Firewall to help protect web applications from common exploits and vulnerabilities. Web Application Firewall is a service that monitors and filters incoming and outgoing web traffic based on rules and policies. Steps to setup Web Application Firewall (WAF)

Considerations:

  • It is recommended to store the TLS Certificate in Key Vault. Note: Specifying Azure Key Vault certificates that are subject to the role-based access control permission model is not supported via the portal. See here for PowerShell example.
  • A critical issue with any Network Security Firewall products like WAF is efficiently analyzing WAF block actions and analyzing potential false positives. AI integration with WAF can assist with detecting false positives and mitigating them using predictive rule guidance.
  • Use Troubleshoot - Azure Web Application Firewall | Microsoft Learn to understand how to identify false positives and tune WAF policies and leverage the WAF Triage Workbook to more easily tune your WAF ruleset.

 

Azure Private Endpoint: A Secure and Efficient Way to Connect to PaaS Services

Azure Private Endpoint is a network interface card (NIC) inside a subnet that attaches to an Azure service. Private endpoints are needed in this configuration to limit public access to PaaS services. Private Endpoints offer support for inbound connectivity (and its respective return traffic) only. For each service in this architecture, below are the steps and resources that you need to set up private endpoints.

By using private endpoints, you can:

  • Enhance the security of your data and network by preventing unauthorized access to the internet or other networks.
  • Reduce data exposure and latency by keeping the traffic within your virtual network.
  • Optimize the performance and reliability of your applications by using the closest and fastest Azure service instances.

You can find the full list of Azure services that support private link at What is a private endpoint? - Azure Private Link | Microsoft Learn

 

DNS Configuration

Azure Private DNS zones are a feature that enables you to use custom domain names for your Azure services within your virtual network. By using Azure Private DNS Zones, you can:

  • Improve the security of your network by preventing DNS queries from leaving your virtual network or being exposed to the internet.
  • Simplify the management of your DNS records by using the same domain names for your Azure services across different regions and subscriptions.
  • Ensure consistent resolution of your Azure services by using the same DNS zone for both public and private endpoints.

Private DNS Zones are global services where best practice is to deploy them in the Hub Virtual Network. For a list of Private DNS Zones, visit here.

The following table shows the private DNS zone for each service domain name. You can also find more details and screenshots for setup steps in this article: Quickstart - Create an Azure private DNS zone using the Azure portal | Microsoft Learn.

Service

Service Domain Name

Azure App Services

azurewebsites.net

Azure Storage Account

blob.core.windows.net

Azure Open AI

openai.azure.com

Azure Key Vault

vaultcore.azure.net

Azure AI Services

search.windows.net

Azure AI Document Intelligence

cognitiveservices.azure.com

Azure APIM Services

azure-api.net

 

DNS for Custom Domain Name

A custom domain is used to access the chat bot app to give the user a friendly URL. Outlined in the following article is the DNS Zone setup for custom domains in Azure: Map existing custom DNS name - Azure App Service | Microsoft Learn  

jsheerin_4-1701372903551.png

 

On Premises DNS Server

DNS Private Resolver simplifies private DNS resolution from on-premises to the Azure private DNS service and vice versa.

  1. Setup DNS Private Resolver in Azure
    1. Choose between a Distributed vs. Centralized DNS Private Resolver implementation. Consider Distributed if your spoke virtual networks will require auto-registration for Virtual Machine resources.
  2. Configure on-premises Windows DNS conditional forwarder
  3. For redundancy add another DNS Private Resolver in the secondary region

jsheerin_1-1701387216333.png

Figure 5: Azure DNS Private Resolver architecture

An Azure Private DNS resolver ensures a secure and customizable DNS resolution service within your virtual network, guaranteeing that queries are resolved accurately to the appropriate URL.

 

NSG Rules

Network security groups (NSGs) are used to filter network traffic to and from Azure resources in an Azure virtual network. A network security group contains security rules that allow or deny inbound network traffic to, or outbound network traffic from, several types of Azure resources. For more information, see Best practices for network security - Microsoft Azure | Microsoft Learn

Application security groups (ASG) allow you to group resources based on their application and define network security policies based on those groups. In this scenario, it is used to separate out the PaaS service endpoints based on their front-end and back-end functionality.

Each subnet has default rules that allow communication within the subnet and with the Internet but block any other traffic. You can learn more about these rules here: Azure network security groups overview | Microsoft Learn.  Further, subnets in Azure are able to honor NSGs rules for private endpoints when the network policy is enabled. Make sure to enable the network policy for each subnet in your environment.

Below is the configuration for the subnets outlined in this scenario:

  • Web Tier Egress Subnet | nsg-app: This subnet handles the outbound traffic from the web application.

Direction

Source/ Destination Port(s)

Service tags Source/ Destination

Protocol

Purpose

Outbound

* / 443

VirtualNetwork / The APIM Subnet’s range private endpoint

TCP

Allows outbound access to the APIM from App Service Integration subnet.

Outbound

* / *

VirtualNetwork / AzureMonitor

 

Any

Allows outbound access to the Azure Monitor from App Service Integration subnet.

Outbound

* / *

VirtualNetwork / VirtualNetwork

 

Any

Denies outbound traffic from the Virtual Network

  • APIM Subnet | nsg-apim: This subnet hosts the Azure API Management service.

Direction

Source/ Destination Port(s)

Service tags Source/ Destination

Protocol

Purpose

Inbound

* / 3443

ApiManagement / VirtualNetwork

 

 

TCP

Allows traffic from APIM instance to Virtual network hosting Azure AI services

Inbound

* / 6390

AzureLoadBalancer / VirtualNetwork

 

TCP

Allow Azure Infrastructure Load Balancer

Inbound

* / 8080

VirtualNetwork / VirtualNetwork

 

Any

Denies Virtual network traffic

Outbound

* / 443

VirtualNetwork Storage

 

TCP

Allows outbound traffic to storage private endpoint

Outbound

* / 443

VirtualNetwork / AzureKeyVault

 

TCP

Allows outbound traffic to Key Vault private endpoint

Outbound

* / *

VirtualNetwork / VirtualNetwork

 

Any

Denies outbound traffic from the Virtual Network

Outbound

 

* / *

Any / Internet

 

Any

Denies outbound traffic to the Internet

  • Shared Tier Subnet | nsg-shared: This subnet contains the shared resources for the chat app, such as the Azure Key Vault.

Direction

Source/ Destination Port(s)

Service tags Source/ Destination

Protocol

Purpose

Outbound

* / *

VirtualNetwork / VirtualNetwork

Any

 

Denies outbound traffic from the Virtual Network

Outbound

* / *

Any / Internet

 

 

Any

Denies outbound traffic to the Internet

  • Data Tier Subnet | nsg-data: This subnet connects to the data services, such as Azure Storage

Direction

Source/ Destination Port(s)

Service tags Source/ Destination

Protocol

Purpose

Inbound

 

* / 443

 

IP Addresses 20.3.165.95 / VirtualNetwork

Any

 

Allow Document intelligence Studio IP

 

Outbound

* / *

Any / Internet

 

 

Any

Denies outbound traffic to the Internet

  • Web Tier Ingress Subnet | nsg-fe: This subnet handles the inbound traffic to the chat app via HTTPS

Direction

Source/ Destination Port(s)

Service tags Source/ Destination

Protocol

Purpose

Inbound

 

* / 443

 

Your Application Gateway Subnet's Private IP Address Range / The explicit IP address of your App Service's private endpoint

TCP

 

Allows inbound access to the App Service from the Application Gateway.

Inbound

* / 80

AzureLoadBalancer / VirtualNetwork

 

TCP

Allow internal health traffic

 

Inbound

* / *

VirtualNetwork / VirtualNetwork

Any

 

Denies inbound traffic from the Virtual Network

Outbound

* / 443

 

The explicit IP address of your App Service's private endpoint / Your Application Gateway Subnet's Private IP Address Range

TCP

 

Allow HTTPs to App Gateway

Outbound

* / *

Any /Any

 

 

Any

Denies outbound traffic

  • AI Tier Subnet | nsg-ai: This subnet hosts the Azure AI Services that provide capabilities for the chat app.

Direction

Source/ Destination Port(s)

Service tags Source/ Destination

Protocol

Purpose

Inbound

 

* / 443

 

The explicit IP address of APIM Subnet / ASG-ai

TCP

 

Allows inbound access to the AI Service from APIM

Inbound

* / 443

ASG-ai /

ASG-data

 

TCP

Allow AI services to Data

Inbound

* / 443

IP address of Management VM / ASG-ai

ASG-data

 

TCP

Allow Open AI Studio management VM to AI services and Data

Inbound

 

* / 443

 

IP Addresses 20.3.165.95 / ASG-ai

Any

Allow Document intelligence Studio IP

 

Inbound

* / *

VirtualNetwork / VirtualNetwork

Any

Denies inbound traffic from the Virtual Network

Outbound

* / 443

ASG-ai /

ASG-data

 

TCP

Allows outbound traffic from AI services to data tier

Outbound

* / 53

VirtualNetwork / DNS Resolver IP

UDP

Allows DNS

Outbound

* / 443

 

ASG-ai / Your Application Gateway Subnet's Private IP Address Range

TCP

 

Allow AI Services to App Gateway

Outbound

* / *

VirtualNetwork / VirtualNetwork

Any

 

Denies outbound traffic from the Virtual Network

Outbound

* / *

Any / Internet

 

 

Any

Denies outbound traffic to the Internet

 

Authentication and Authorization

Monitoring

Resources 

Contributors

Thanks to everyone for their contributions to the development of this reference architecture: Alejandra Palacios, Alexandra Miller-Browne,  Andre Murrell, Charles Shea, Jen Sheerin, Mauricio Rojas Martinez, Sowmya Mahadevaiah, Simona Kovatcheva

 

 

4 Comments
Version history
Last update:
‎Nov 30 2023 04:42 PM
Updated by: