Multi Model Deployment with Azure AI Foundry Serverless, Python and Container Apps
Intro Azure AI Foundry is a comprehensive AI suite, with a vast set of serverless and managed models offerings designed to democratize AI deployment. Whether you’re running a small startup or an 500 enterprise, Azure AI Foundry provides the flexibility and scalability needed to implement and manage machine learning and AI models seamlessly. By leveraging Azure’s robust cloud infrastructure, you can focus on innovating and delivering value, while Azure takes care of the heavy lifting behind the scenes. In this demonstration, we delve into building an Azure Container Apps stack. This innovative approach allows us to deploy a Web App that facilitates interaction with three powerful models: GPT-4, Deepseek, and PHI-3. Users can select from these models for Chat Completions, gaining invaluable insights into their actual performance, token consumption, and overall efficiency through real-time metrics. This deployment not only showcases the versatility and robustness of Azure AI Foundry but also provides a practical framework for businesses to observe and measure AI effectiveness, paving the way for data-driven decision-making and optimized AI solutions. Azure AI Foundry: The evolution Azure AI Foundry represents the next evolution in Microsoft’s AI offerings, building on the success of Azure AI and Cognitive Services. This unified platform is designed to streamline the development, deployment, and management of AI solutions, providing developers and enterprises with a comprehensive suite of tools and services. With Azure AI Foundry, users gain access to a robust model catalog, collaborative GenAIOps tools, and enterprise-grade security features. The platform’s unified portal simplifies the AI development lifecycle, allowing seamless integration of various AI models and services. Azure AI Foundry offers the flexibility and scalability needed to bring your AI projects to life, with deep insights and fast adoption path for the users. The Model Catalog allows us to filter and compare models per our requirements and easily create deployments directly from the Interface. Building the Application Before describing the methodology and the process, we have to make sure our dependencies are in place. So let’s have a quick look on the prerequisites of our deployment. GitHub - passadis/ai-foundry-multimodels: Azure AI Foundry multimodel utilization and performance metrics Web App. Azure AI Foundry multimodel utilization and performance metrics Web App. - passadis/ai-foundry-multimodels github.com Prerequisites Azure Subscription Azure AI Foundry Hub with a project in East US. The models are all supported in East US. VSCode with the Azure Resources extension There is no need to show the Azure Resources deployment steps, since there are numerous ways to do it and i have also showcased that in previous posts. In fact, it is a standard set of services to support our Micro-services Infrastructure: Azure Container Registry, Azure Key Vault, Azure User Assigned Managed identity, Azure Container Apps Environment and finally our Azure AI Foundry Model deployments. Frontend – Vite + React + TS The frontend is built using Vite and React and features a dropdown menu for model selection, a text area for user input, real-time response display, as well as loading states and error handling. Key considerations in the frontend implementation include the use of modern React patterns and hooks, ensuring a responsive design for various screen sizes, providing clear feedback for user interactions, and incorporating elegant error handling. The current implementation allows us to switch models even after we have initiated a conversation and we can keep up to 5 messages as Chat History. The uniqueness of our frontend is the performance information we get for each response, with Tokens, Tokens per Second and Total Time. Backend – Python + FastAPI The backend is built with FastAPI and is responsible for model selection and configuration, integrating with Azure AI Foundry, processing requests and responses, and handling errors and validation. A directory structure as follows can help us organize our services and utilize the modular strengths of Python: backend/ ├── app/ │ ├── __init__.py │ ├── main.py │ ├── config.py │ ├── api/ │ │ ├── __init__.py │ │ └── routes.py │ ├── models/ │ │ ├── __init__.py │ │ └── request_models.py │ └── services/ │ ├── __init__.py │ └── azure_ai.py ├── run.py # For Local runs ├── Dockerfile ├── requirements.txt └── .env Azure Container Apps A powerful combination allows us to easily integrate both using Dapr, since it is natively supported and integrated in our Azure Container Apps. try { const response = await fetch('/api/v1/generate', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ model: selectedModel, prompt: userInput, parameters: { temperature: 0.7, max_tokens: 800 } }), }); However we need to correctly configure NGINX to proxy the request to the Dapr Sidecar since we are using Container Images. # API endpoints via Dapr location /api/v1/ { proxy_pass http://localhost:3500/v1.0/invoke/backend/method/api/v1/; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; Azure Key Vault As always all our secret variables like the API Endpoints and the API Keys are stored in Key Vault. We create a Key Vault Client in our Backend and we call each key only the time we need it. That makes our deployment more secure and efficient. Deployment Considerations When deploying your application: Set up proper environment variables Configure CORS settings appropriately Implement monitoring and logging Set up appropriate scaling policies Azure AI Foundry: Multi Model Architecture The solution is built on Azure Container Apps for serverless scalability. The frontend and backend containers are hosted in Azure Container Registry and deployed to Container Apps with Dapr integration for service-to-service communication. Azure Key Vault manages sensitive configurations like API keys through a user-assigned Managed Identity. The backend connects to three Azure AI Foundry models (DeepSeek, GPT-4, and Phi-3), each with its own endpoint and configuration. This serverless architecture ensures high availability, secure secret management, and efficient model interaction while maintaining cost efficiency through consumption-based pricing. Conclusion This Azure AI Foundry Models Demo showcases the power of serverless AI integration in modern web applications. By leveraging Azure Container Apps, Dapr, and Azure Key Vault, we’ve created a secure, scalable, and cost-effective solution for AI model comparison and interaction. The project demonstrates how different AI models can be effectively compared and utilized, providing insights into their unique strengths and performance characteristics. Whether you’re a developer exploring AI capabilities, an architect designing AI solutions, or a business evaluating AI models, this demo offers practical insights into Azure’s AI infrastructure and serverless computing potential. References Azure AI Foundry Azure Container Apps Azure AI – Documentation AI learning hub CloudBlogger: Text To Speech with ContainersHow to build a data streaming pipeline for real-time enterprise generative AI apps
How to build a data streaming pipeline for real-time enterprise generative AI apps using Azure Event Hubs + Azure OpenAI + Pathway’s LLM App+Streamlit. The source code is on GitHub: https://github.com/pathway-labs/azure-openai-real-time-data-app/tree/main Real-time AI app needs real-time data to respond with the most up-to-date information to user queries or perform quick actions autonomously. For example, a customer support team wants to improve its customer support by analyzing customer feedback and inquiries in real-time. They aim to understand common issues, track customer sentiment, and identify areas for improvement in their products and services. To achieve this, they need a system that can process large data streams, analyze text for insights, and present these insights in an accessible way. To help them we will build a real-time data pipeline with Azure Event Hubs, Pathway, and Azure OpenAI. This integrated system leverages the strengths of Pathway for robust data processing, LLMs like GPT for advanced text analytics, and Streamlit for user-friendly data visualization. This combination empowers businesses to build and deploy enterprise AI applications that provide the freshest contextual visual data. The new solution can help to multiple teams: Customer Support Team: They can use the dashboard to monitor customer satisfaction and common issues in real-time, allowing for quick responses. Product Development: Insights from customer feedback can inform product development, highlighting areas for improvement or innovation. Marketing and PR: Understanding customer sentiment trends helps in tailoring marketing campaigns and managing public relations more effectively. Implementation Let’s break down the main parts of the application architecture and understand the role of each in our solution. The project source code, deployment automation implementation, and setup guidelines can be found on GitHub. Azure Event Hubs & Kafka: Real-Time Data Streaming and Processing Azure Event Hubs collects real-time data from various sources, such as customer feedback forms, support chat logs, and social media mentions. This data is then streamed into a Kafka cluster for further processing. Large Language Models (LLMs) like GPT from Azure OpenAI: Text Analysis and Sentiment Detection The text data from Kafka is fed into an LLM for natural language processing using Pathway. This model performs sentiment analysis, key phrase extraction, and feedback categorization (e.g., identifying common issues or topics). Pathway to enable real-time data pipeline Pathway gains access to the data streams from Azure Event Hubs, it preprocesses, transforms, or joins them and the LLM App helps to bring real-time context to the AI App with real-time vector indexing, semantic search, and retrieval capabilities. The text content of the events will be sent to Azure OpenAI embedding APIs via the LLM App to compute the embeddings and vector representations will be indexed using KNN (K-Nearest Neighbors). Using the LLM app, the company can gain deep insights from unstructured text data, understanding the sentiment and nuances of customer feedback. Streamlit: Interactive Dashboard for Visualization Streamlit is used to create an interactive web dashboard that visualizes the insights derived from customer feedback. This dashboard can show real-time metrics such as overall sentiment trends, and common topics in customer feedback, and even alert the team to emerging issues (See example implementation of alerting to enhance this project). Here is a short demo of running the app in Azure: Overview of the Azure services the sample project uses Service Purpose Azure AI Services To use Azure OpenAI GPT model and embeddings. Azure Event Hubs To stream real-time events from various data sources. Azure Container Apps Hosts our containerized applications (backend and frontend) with features like auto-scaling and load balancing. Azure Container Registry Stores our Docker container images in a managed, private registry. Azure Log Analytics Collects and analyzes telemetry and logs for insights into application performance and diagnostics. Azure Monitor Provides comprehensive monitoring of our applications, infrastructure, and network. Azure infrastructure with the main components As you can see in the below infrastructural diagram, we use two Azure Container Apps to deploy the Pathway LLM App and Streamlit UI dashboard to a single Azure Resource Group. Simple architecture and reduced costs The current solution with the LLM App simplifies the AI pipeline infrastructure by consolidating capabilities into one platform. No need to integrate and maintain separate modules for your Gen AI app: Vector Databases (e.g. Pinecone/Weaviate/Qdrant) + LangChain + Cache (e.g. Redis) + API Framework (e.g. Fast API). It also reduced the cloud costs compared to other possible implementations using Vector Database/Azure Cognitive Search + Azure Functions (For hosting API and logic code) + Azure App Service. Let’s calculate it using the Azure pricing calculator. For simplicity, we do not count costs for Azure OpenAI. 1. In the current solution with two Azure Container Apps (10 million requests per month) + one Azure Event Hubs (10 million events per month), you can see the total estimated cost per month here which is around 11 USD with the basic setup. See the report: https://azure.com/e/d3f1261757d14dc5a9ea1c414a00069f 2. In the second solution, we use Azure Event Hubs + Azure Function (to ingest real-time events and logic code) + Azure AI Search (vector search) + Container Apps (To host Streamlit UI Dashboard). You will end with an estimated monthly cost of 17 USD per month. We did not count costs for Storage accounts required by Functions. As you can see, the See the report: https://azure.com/e/0c934b47b2b745f596d59121f4020677 Tutorial - Creating the app The app development consists of two parts: backend API and frontend UI. Part 1: Design the Streamlit UI We will start with constructing Streamlit UI and create a simple web application with Streamlit. It interacts with the LLM App backend service over REST API and displays the insights derived from real-time customer feedback. See the full source code in the app.py file. st.title("Customer support and sentiment analysis dashboard") st.subheader("Example prompt") default_prompt = "Provide overall sentiment trends, and common topics and rating over time and sources with counts based on last feedback events and respond only in json without explanation and new line." st.text(default_prompt) placeholder = st.empty() for seconds in range(200): url = f"{api_host}" data = {"query": default_prompt, "user": "user"} response = requests.post(url, json=data) if response.status_code == 200: data_response = response.json() json_data = json.loads(data_response) with placeholder.container(): # Sentiment Trends sentiment_df = pd.DataFrame(list(json_data["sentiment_trends"].items()), columns=['Sentiment', 'Count']) color_map = {"positive": "green", "negative": "red", "neutral": "blue"} fig_sentiment = px.bar(sentiment_df, x='Sentiment', y='Count', title="Sentiment Trends", color='Sentiment', color_discrete_map=color_map) # Rating Over Time rating_data = json_data["rating_over_time"] rating_df = pd.DataFrame(rating_data) rating_df['Date'] = pd.to_datetime(rating_df['date']) fig_rating = px.line(rating_df, x='Date', y='rating', title="Average Rating Over Time", markers=True) # Streamlit layout st.plotly_chart(fig_sentiment, use_container_width=True) st.plotly_chart(fig_rating, use_container_width=True) # Convert the source counts to a DataFrame for visualization sources_df = pd.DataFrame(json_data["common_topics"], columns=['topic', 'count']) fig_sources = px.bar(sources_df, x='topic', y='count', title="Common Topics") st.plotly_chart(fig_sources, use_container_width=True) sources_df = pd.DataFrame(json_data["common_sources"], columns=['source', 'count']) fig_sources = px.bar(sources_df, x='source', y='count', title="Common Sources") st.plotly_chart(fig_sources, use_container_width=True) time.sleep(1) else: st.error( f"Failed to send data to API. Status code: {response.status_code}" ) In the above code, we define a default prompt to instruct the LLM App to respond with all necessary data such as sentiment trends, common topics, ratings over time, and common sources as structured in JSON format. We use Streamlit Charts to visualize different dashboards. The dashboards are updated every second, likely fetching and displaying new data each time. Part 2: Build a backend API Next, we develop the backend logic where the app ingests streaming data from a Kafka topic provided by Azure Event Hubs and uses it to respond to user queries via an HTTP API. The function integrates with Azure OpenAI's Embeddings API for generating embeddings and a ChatGPT model for generating LLM responses. See the full source code in the app.py file. ... def run( *, host: str = "0.0.0.0", port: int = 8080 ): # Real-time data coming from the Kafka topic topic_data = pw.io.kafka.read( rdkafka_settings, topic="eventhubpathway", format="raw", autocommit_duration_ms=1000, ) # Tranform data to structured document transformed_topic_data = transform(topic_data) # Compute embeddings for each Kafka event using the OpenAI Embeddings API embedded_topic_data = embeddings(context=transformed_topic_data, data_to_embed=transformed_topic_data.doc) # Construct an index on the generated embeddings in real-time index = index_embeddings(embedded_topic_data) # Given a user question as a query from your API query, response_writer = pw.io.http.rest_connector( host=host, port=port, schema=QueryInputSchema, autocommit_duration_ms=50, ) # Generate embeddings for the query from the OpenAI Embeddings API embedded_query = embeddings(context=query, data_to_embed=pw.this.query) # Build prompt using indexed data responses = prompt(index, embedded_query, pw.this.query) # Feed the prompt to ChatGPT and obtain the generated answer. response_writer(responses) pw.run() ... Generated vector embeddings are indexed for efficient retrieval in real-time. The user's query is embedded using the same embeddings API endpoint, to enable a vector search against the indexed Kafka data. Finally, a new prompt is constructed using the indexed data and the embedded user query. This prompt is then used to generate a response by querying a model like ChatGPT. What is next As we have seen in the example of the customer feedback analysis app demo, used for businesses looking to harness real-time data for strategic decision-making and responsive customer service. This simplification in the architecture and implementation with Pathway’s LLM App means that your GenAI apps go to market within a short period (4-6 weeks), with lower costs and high security. Consider also visiting another showcase on Use LLMs for notifications. You will see a few examples showcasing different possibilities with the LLM App in the GitHub Repo. Follow the instructions in Get Started with Pathway to try out different demos.
