Announcing GPT‑5.2‑Codex in Microsoft Foundry: Enterprise‑Grade AI for Secure Software Engineering
Enterprise developers know the grind: wrestling with legacy code, navigating complex dependency challenges, and waiting on security reviews that stall releases. OpenAI’s GPT‑5.2‑Codex flips that equation and helps engineers ship faster without cutting corners. It’s not just autocomplete; it’s a reasoning engine for real-world software engineering. Generally available starting today through Azure OpenAI in Microsoft Foundry Models, GPT‑5.2‑Codex is built for the realities of enterprise codebases, large repos, evolving requirements, and security constraints that can’t be overlooked. As OpenAI’s most advanced agentic coding model, it brings sustained reasoning, and security-aware assistance directly into the workflows enterprise developers already rely on with Microsoft’s secure and reliable infrastructure. GPT-5.2-Codex at a Glance GPT‑5.2‑Codex is designed for how software gets built in enterprise teams. You start with imperfect inputs including legacy code, partial docs, screenshots, diagrams, and work through multi‑step changes, reviews, and fixes. The model helps keep context, intent, and standards intact across that entire lifecycle, so teams can move faster without sacrificing quality or security. What it enables Work across code and artifacts: Reason over source code alongside screenshots, architecture diagrams, and UI mocks — so implementation stays aligned with design intent. Stay productive in long‑running tasks: Maintain context across migrations, refactors, and investigations, even as requirements evolve. Build and review with security in mind: Get practical support for secure coding patterns, remediation, reviews, and vulnerability analysis — where correctness matters as much as speed. Feature Specs (quick reference) Context window: 400K tokens (approximately 100K lines of code) Supported languages: 50+ including Python, JavaScript/TypeScript, C#, Java, Go, Rust Multimodal inputs: Code, images (UI mocks, diagrams), and natural language API compatibility: Drop-in replacement for existing Codex API calls Use cases where it really pops Legacy modernization with guardrails: Safely migrate and refactor “untouchable” systems by preserving behavior, improving structure, and minimizing regression risk. Large‑scale refactors that don’t lose intent: Execute cross‑module updates and consistency improvements without the typical “one step forward, two steps back” churn. AI‑assisted code review that raises the floor: Catch risky patterns, propose safer alternatives, and improve consistency, especially across large teams and long‑lived codebases. Defensive security workflows at scale: Accelerate vulnerability triage, dependency/path analysis, and remediation when speed matters, but precision matters more. Lower cognitive load in long, multi‑step builds: Keep momentum across multi‑hour sessions: planning, implementing, validating, and iterating with context intact. Pricing Model Input Price/1M Tokens Cached Input Price/1M Tokens Output Price/1M Tokens GPT-5.2-Codex $1.75 $0.175 $14.00 Security Aware by Design, not as an Afterthought For many organizations, AI adoption hinges on one nonnegotiable question: Can this be trusted in security sensitive workflows? GPT-5.2-Codex meaningfully advances the Codex lineage in this area. As models grow more capable, we’ve seen that general reasoning improvements naturally translate into stronger performance in specialized domains — including defensive cybersecurity. With GPT‑5.2‑Codex, this shows up in practical ways: Improved ability to analyze unfamiliar code paths and dependencies Stronger assistance with secure coding patterns and remediation More dependable support during code reviews, vulnerability investigations, and incident response At the same time, Microsoft continues to deploy these capabilities thoughtfully balancing access, safeguards, and platform level controls so enterprises can adopt AI responsibly as capabilities evolve. Why Run GPT-5.2-Codex on Microsoft Foundry? Powerful models matter — but where and how they run matters just as much for enterprise. Organizations choose Microsoft Foundry because it combines Foundry frontier AI with Azure enterprise grade fundamentals: Integrated security, compliance, and governance Deploy GPT-5.2-Codex within existing Azure security boundaries, identity systems, and compliance frameworks — without reinventing controls. Enterprise ready orchestration and tooling Build, evaluate, monitor, and scale AI powered developer experiences using the same platform teams already rely on for production workloads. A unified path from experimentation to scale Foundry makes it easier to move from proof of concept to real deployment —without changing platforms, vendors, or operating assumptions. Trust at the platform level For teams working in regulated or security critical environments, Foundry and Azure provide assurances that go beyond the model itself. Together with GitHub Copilot, Microsoft Foundry provides a unified developer experience — from in‑IDE assistance to production‑grade AI workflows — backed by Azure’s security, compliance, and global scale. This is where GPT-5.2-Codex becomes not just impressive but adoptable. Get Started Today Explore GPT‑5.2‑Codex in Microsoft today. Start where you already work: Try GPT‑5.2‑Codex in GitHub Copilot for everyday coding and scale the same model to larger workflows using Azure OpenAI in Microsoft Foundry. Let’s build what’s next with speed and security.Open AI’s GPT-5.1-codex-max in Microsoft Foundry: Igniting a New Era for Enterprise Developers
Announcing GPT-5.1-codex-max: The Future of Enterprise Coding Starts Now We’re thrilled to announce the general availability of OpenAI's GPT-5.1-codex-max in Microsoft Foundry Models; a leap forward that redefines what’s possible for enterprise-grade coding agents. This isn’t just another model release; it’s a celebration of innovation, partnership, and the relentless pursuit of developer empowerment. At Microsoft Ignite, we unveiled Microsoft Foundry: a unified platform where businesses can confidently choose the right model for every job, backed by enterprise-grade reliability. Foundry brings together the best from OpenAI, Anthropic, xAI, Black Forest Labs, Cohere, Meta, Mistral, and Microsoft’s own breakthroughs, all under one roof. Our partnership with Anthropic is a testament to our commitment to giving developers access to the most advanced, safe, and high-performing models in the industry. And now, with GPT-5.1-codex-max joining the Foundry family, the possibilities for intelligent applications and agentic workflows have never been greater. GPT 5.1-codex-max is available today in Microsoft Foundry and accessible in Visual Studio Code via the Foundry extension . Meet GPT-5.1-codex-max: Enterprise-Grade Coding Agent for Complex Projects GPT-5.1-codex-max is engineered for those who build the future. Imagine tackling complex, long-running projects without losing context or momentum. GPT-5.1-codex-max delivers efficiency at scale, cross-platform readiness, and proven performance with top scores on SWE-Bench (77.9), the gold standard for AI coding. With GPT-5.1-codex-max, developers can focus on creativity and problem-solving, while the model handles the heavy lifting. GPT-5.1-codex-max isn’t just powerful; it’s practical, designed to solve real challenges for enterprise developers: Multi-Agent Coding Workflows: Automate repetitive tasks across microservices, maintaining shared context for seamless collaboration. Enterprise App Modernization: Effortlessly refactor legacy .NET and Java applications into cloud-native architectures. Secure API Development: Generate and validate secure API endpoints, with `compliance checks built-in for peace of mind. Continuous Integration Support: Integrate GPT-5.1-codex-max into CI/CD pipelines for automated code reviews and test generation, accelerating delivery cycles. These use cases are just the beginning. GPT-5.1-codex-max is your partner in building robust, scalable, and secure solutions. Foundry: Platform Built for Developers Who Build the Future Foundry is more than a model catalog—it’s an enterprise AI platform designed for developers who need choice, reliability, and speed. • Choice Without Compromise: Access the widest range of models, including frontier models from leading model providers. • Enterprise-Grade Infrastructure: Built-in security, observability, and governance for responsible AI at scale. • Integrated Developer Experience: From GitHub to Visual Studio Code, Foundry connects with tools developers love for a frictionless build-to-deploy journey. Start Building Smarter with GPT-5.1-codex-max in Foundry The future is here, and it’s yours to shape. Supercharge your coding workflows with GPT-5.1-codex-max in Microsoft Foundry today. Learn more about Microsoft Foundry: aka.ms/IgniteFoundryModels. Watch Ignite sessions for deep dives and demos: ignite.microsoft.com. Build faster, smarter, and with confidence on the platform redefining enterprise AI.
How to Reliably Gauge LLM Confidence?
a { text-decoration: none; color: #464feb; } tr th, tr td { border: 1px solid #e6e6e6; } tr th { background-color: #f5f5f5; } I’m trying to estimate an LLM’s confidence in its answers in a way that correlates with correctness. Self-reported confidence is often misleading, and raw token probabilities mostly reflect fluency rather than truth. I don’t have grounding options like RAG, human feedback, or online search, so I’m looking for approaches that work in this constraint. What techniques have you found effective—entropy-based signals, calibration (temperature scaling), self-evaluation, or others? Any best practices for making confidence scores actionable?Introducing OpenAI’s GPT-image-1.5 in Microsoft Foundry
Developers building with visual AI can often run into the same frustrations: images that drift from the prompt, inconsistent object placement, text that renders unpredictably, and editing workflows that break when iterating on a single asset. That’s why we are excited to announce OpenAI's GPT Image 1.5 is now generally available in Microsoft Foundry. This model can bring sharper image fidelity, stronger prompt alignment, and faster image generation that supports iterative workflows. Starting today, customers can request access to the model and start building in the Foundry platform. Meet GPT Image 1.5 AI driven image generation began with early models like OpenAI's DALL-E, which introduced the ability to transform text prompts into visuals. Since then, image generation models have been evolving to enhance multimodal AI across industries. GPT Image 1.5 represents continuous improvement in enterprise-grade image generation. Building on the success of GPT Image 1 and GPT Image 1 mini, these enhanced models introduce advanced capabilities that cater to both creative and operational needs. The new image models offer: Text-to-image: Stronger instruction following and highly precise editing. Image-to-image: Transform existing images to iteratively refine specific regions Improved visual fidelity: More detailed scenes and realistic rendering. Accelerated creation times: Up to 4x faster generation speed. Enterprise integration: Deploy and scale securely in Microsoft Foundry. GPT Image 1.5 delivers stronger image preservation and editing capabilities, maintaining critical details like facial likeness, lighting, composition, and color tone across iterative changes. You’ll see more consistent preservation of branded logos and key visuals, making it especially powerful for marketing, brand design, and ecommerce workflows—from graphics and logo creation to generating full product catalogs (variants, environments, and angles) from a single source image. Benchmarks Based on an internal Microsoft dataset, GPT Image 1.5 performs higher than other image generation models in prompt alignment and infographics tasks. It focuses on making clear, strong edits – performing best on single-turn modification, delivering the higher visual quality in both single and multi-turn settings. The following results were found across image generation and editing: Text to image Prompt alignment Diagram / Flowchart GPT Image 1.5 91.2% 96.9% GPT Image 1 87.3% 90.0% Qwen Image 83.9% 33.9% Nano Banana Pro 87.9% 95.3% Image editing Evaluation Aspect Modification Preservation Visual Quality Face Preservation Metrics BinaryEval SC (semantic) DINO (Visual) BinaryEval AuraFace Single-turn GPT image 1 99.2% 51.0% 0.14 79.5% 0.30 Qwen image 81.9% 63.9% 0.44 76.0% 0.85 GPT Image 1.5 100% 56.77% 0.14 89.96% 0.39 Multi-turn GPT Image 1 93.5% 54.7% 0.10 82.8% 0.24 Qwen image 77.3% 68.2% 0.43 77.6% 0.63 GPT image 1.5 92.49% 60.55% 0.15 89.46% 0.28 Using GPT Image 1.5 across industries Whether you’re creating immersive visuals for campaigns, accelerating UI and product design, or producing assets for interactive learning GPT Image 1.5 gives modern enterprises the flexibility and scalability they need. Image models can allow teams to drive deeper engagement through compelling visuals, speed up design cycles for apps, websites, and marketing initiatives, and support inclusivity by generating accessible, high‑quality content for diverse audiences. Watch how Foundry enables developers to iterate with multimodal AI across Black Forest Labs, OpenAI, and more: Microsoft Foundry empowers organizations to deploy these capabilities at scale, integrating image generation seamlessly into enterprise workflows. Explore the use of AI image generation here across industries like: Retail: Generate product imagery for catalogs, e-commerce listings, and personalized shopping experiences. Marketing: Create campaign visuals and social media graphics. Education: Develop interactive learning materials or visual aids. Entertainment: Edit storyboards, character designs, and dynamic scenes for films and games. UI/UX: Accelerate design workflows for apps and websites. Microsoft Foundry provides security and compliance with built-in content safety filters, role-based access, network isolation, and Azure Monitor logging. Integrated governance via Azure Policy, Purview, and Sentinel gives teams real-time visibility and control, so privacy and safety are embedded in every deployment. Learn more about responsible AI at Microsoft. Pricing Model Pricing (per 1M tokens) - Global GPT-image-1.5 Input Tokens: $8 Cached Input Tokens: $2 Output Tokens: $32 Cost efficiency improves as well: image inputs and outputs are now cheaper compared to GPT Image 1, enabling organizations to generate and iterate on more creative assets within the same budget. For detailed pricing, refer here. Getting started Learn more about image generation, explore code samples, and read about responsible AI protections here. Try GPT Image 1.5 in Microsoft Foundry and start building multimodal experiences today. Whether you’re designing educational materials, crafting visual narratives, or accelerating UI workflows, these models deliver the flexibility and performance your organization needs.Azure OpenAI Model Upgrades: Prompt Safety Pitfalls with GPT-4o and Beyond
Upgrading to New Azure OpenAI Models? Beware Your Old Prompts Might Break. I recently worked on upgrading our Azure OpenAI integration from gpt-35-turbo to gpt-4o-mini, expecting it to be a straightforward configuration change. Just update the Azure Foundry resource endpoint, change the model name, deploy the code — and voilà, everything should work as before. Right? Not quite. The Unexpected Roadblock As soon as I deployed the updated code, I started seeing 400 status errors from the OpenAI endpoint. The message was cryptic: The response was filtered due to the prompt triggering Azure OpenAI's content management policy. At first, I assumed it was a bug in my SDK call or a malformed payload. But after digging deeper, I realized this wasn’t a technical failure — it was a content safety filter kicking in before the prompt even reached the model. The Prompt That Broke It Here’s the original system prompt that worked perfectly with gpt-35-turbo: YOU ARE A QNA EXTRACTOR IN TEXT FORMAT. YOU WILL GET A SET OF SURVEYJS QNA JSONS. YOU WILL CONVERT THAT INTO A TEXT DOCUMENT. FOR THE QUESTIONS WHERE NO ANSWER WAS GIVEN, MARK THOSE AS NO ANSWER. HERE IS THE QNA: BE CREATIVE AND PROFESSIONAL. I WANT TO GENERATE A DOCUMENT TO BE PUBLISHED. {{$style}} +++++ {{$input}} +++++ This prompt had been reliable for months. But with gpt-4o-mini, it triggered Azure’s new input safety layer, introduced in mid-2024. What Changed with GPT-4o-mini? Unlike gpt-35-turbo, the gpt-4o family: Applies stricter content filtering — not just on the output, but also on the input prompt. Treats system messages and user messages as role-based chat messages, passing them through moderation before the model sees them. Flags prompts that look like prompt injection attempts like aggressive instructions like “YOU ARE…”, “BE CREATIVE”, “GENERATE”, “PROFESSIONAL”. Flags unusual formatting (like `+++++`), artificial delimiters or token markers as it may look like encoded content. In short, the model didn’t even get a chance to process my prompt — it was blocked at the gate. Fixing It: Softening the Prompt The solution wasn’t to rewrite the entire logic, but to soften the system prompt and remove formatting that could be misinterpreted. Here’s what helped: - Replacing “YOU ARE…” with a gentler instruction like “Please help convert the following Q&A data…” - Removing creative directives like “BE CREATIVE” or “PROFESSIONAL” unless clearly contextualized. - Avoiding raw JSON markers and template syntax (`{{ }}`, `+++++`) in the prompt. Once I made these changes, the model responded smoothly — and the upgrade was finally complete. Evolving the Prompt — Not Abandoning It Interestingly, for some prompts I didn’t have to completely eliminate the “YOU ARE…” structure. Instead, I refined it to be more natural and less directive. Here’s a comparison: ❌ Old Prompt (Blocked) ✅ New Prompt (Accepted) YOU ARE A SOURCING AND PROCUREMENT MANAGER. YOU WILL GET BUYER'S REQUIREMENTS IN QNA FORMAT. HERE IS THE QNA: {{$input}} +++++ YOU WILL GENERATE TOP 10 {{$category}} RELATED QUESTIONS THAT CAN BE ASKED OF A SUPPLIER IN JSON FORMAT. THE JSON MUST HAVE QUESTION NUMBER AS THE KEY AND QUESTION TEXT AS THE QUESTION. DON'T ADD ANY DESCRIPTION TEXT OR FORMATTING IN THE OUTPUT. BE CREATIVE AND PROFESSIONAL. I WANT TO GENERATE AN RFX. You are an AI assistant that helps clarify sourcing requirements. You will receive buyer's requirements in QnA format. Here is the QnA: {$input} Your task is to generate the top 10 {$category} related questions that can be asked of a supplier, in JSON format. - The JSON must use the question number as the key and the question text as the value. - Do not include any description text or formatting in the output. - Focus on creating clear, professional, and relevant questions that will help prepare an RFX. Key Takeaways - Model upgrades aren’t just about configuration changes — they can introduce new moderation layers that affect prompt design. - Prompt safety filtering is now a first-class citizen in Azure OpenAI, especially for newer models. - System prompts need to be rewritten with moderation in mind, not just clarity or creativity. This experience reminded me that even small upgrades can surface big learning moments. If you're planning to move to gpt-4o-mini or any newer Azure OpenAI model, take a moment to review your prompts — they might need a little more finesse than before.
Azure AI Foundry/Azure AI Service - cannot access agents
I'm struggling with getting agents to work via API which were defined in AI Foundry (based on Azure AI Service). When defining agent in project in AI Foundry I can use it in playground via web browser. The issue appears when I'm trying to access them via API (call from Power Automate). When executing Run on agent I get info that agent cannot be found. The issue doesn't exist when using Azure OpenAI and defining assistants. I can use them both via API and web browser. I guess that another layer of management which is project might be an issue here. I saw usage of SDK in Python and first call is to connect to a project and then get an agent. Does anyone of you experienced the same? Is a way to select and run agent via API?
Principle Does not have Access to API/Operation
Hi all, I am trying to connect Azure OpenAI service to Azure AI Search service to Azure Gen 2 Data lake. In the Azure AI Foundry studio Chat Playground, I am able to add my data source, which is a .csv file in the data lake that has been indexed successfully. I use "System Assigned Managed Identity". The following RBAC has been applied: AI Search service has Cognitive Services OpenAI Contributor in Azure Open AI service Azure OpenAI service has Search Index Data Reader in AI Search Service Azure OpenAI service has Search Service Contributor in AI Search Service AI Search Service has Storage Blob Data Reader in Storage account (Data Lake) As mentioned when adding the data source it passes validation but when I try to ask a question, I get the error "We couldn't connect your data Principal does not have access to API/Operation"AMA on Camera - AI Revolution: Azure OpenAI's Game-Changing Enhancements
Discover the latest enhancements to Azure OpenAI Service offering and deployments. Join our engineering experts to learn about Azure OpenAI Data Zones for flexible, multi-regional data processing and compliance. Explore simplified deployment flows, industry-leading performance guarantees, and cost-efficiency at scale. Perfect for anyone looking to leverage adaptable, reliable, and scalable AI solutions for enterprise applications. Please check out this blog post for more information: Accelerate scale with Azure OpenAI Service Provisioned offering | Microsoft Azure Blog Azure OpenAI Global Batch offering is designed to handle large-scale and high-volume processing tasks efficiently. Process asynchronous groups of requests with separate quota, a 24-hour turnaround time, at 50% less cost than global standard. Learn more There will be presentations on Batch, Data Zone Standard and Data Zone Provisioned and we will be taking questions throughout. Please share any questions that you would like our experts to address in the comments below. Questions can be posted anytime in the comments below beforehand, if it fits your schedule or time zone better, though questions will not be answered until the live hour. Questions will be answered both in the live broadcast on video and in text below. This will be a live stream video directly on this event page. NOTE: Please be aware the link sent via the private message to folks will not work due to some platform changes. We are working on redirecting it, but just in case, the new event link (the page you are on) is here: AMA on Camera - AI Revolution: Azure OpenAI's Game-Changing Enhancements | Microsoft Community Hub )
Enhance Large Language Models (LLM)/Azure OpenAI Performance and Cost Efficiency
Basic techniques to optimize performance and cost efficiency of Large Language Models (LLM) in Azure OpenAI. Selection of Optimal Models: Choosing the right model is crucial for achieving the best performance. This involves understanding the capabilities of each model and selecting the one that best suits the task at hand. Factors to consider include the model's language capabilities, its ability to generate creative content, and its performance on tasks similar to the one you're working on. Enhancing Prompts: The quality of the prompts you provide to the model can significantly impact its performance. A well-crafted prompt can guide the model to generate more relevant and useful responses. This might involve specifying the format you want the answer in, or asking the model to think step-by-step or debate pros and cons before settling on an answer. Controlling Call Rates: To optimize costs and performance, it's important to manage the rate at which you make API calls. This involves understanding the rate limits of the API and designing your application to stay within these limits. You might need to implement a queuing system or use exponential backoff in case of rate limit errors. Provisioned Throughput Units (PTUs): PTUs allow you to reserve capacity for your application, ensuring consistent performance even during peak times. By correctly provisioning PTUs, you can balance cost and performance to meet your application's needs. Integration with Networks and Security: Azure OpenAI can be integrated with your existing network and security infrastructure. This might involve setting up Virtual Networks, using Private Link for secure communication, and managing access with Azure Active Directory. Integration with Data Sources and Cognitive Enterprise Search: Azure OpenAI can be used in conjunction with various data sources and Azure's Cognitive Search service. This allows you to build powerful applications that can search through large amounts of data, understand it, and generate human-like text based on it. Efficient Search using Vector Search: Vector search is a method of retrieving information that involves converting text into high-dimensional vectors and searching for similar vectors. This can be much more efficient than traditional search methods, especially for large datasets. Azure OpenAI includes support for vector search, allowing you to build efficient, AI-powered search applications. Utilizing Agent Pools for Different Tasks: Agent pools are a powerful feature that allows you to manage and distribute workloads across different resources. By creating separate agent pools for different tasks, you can ensure that each task has the resources it needs to run efficiently. This can help to optimize performance and reduce costs. Use Clear Meta Prompts and System Messages for Desired Outputs: Meta prompts and system messages play a crucial role in guiding the AI to produce the desired outputs. A meta prompt is a directive that instructs the AI on the format or type of response required. Use AI model combination for simple tasks and LLM for complex tasks: Different AI models have different strengths and capabilities. For simple tasks, such as answering straightforward questions or generating short pieces of text, a combination of smaller, specialized AI models might be sufficient. These models can be faster and more efficient than larger models, and can often produce high-quality results for simple tasks. It's my hope that these 10 methods will deepen your understanding of how to optimize large Language Models, thereby improving performance and cost efficiency.How to build a data streaming pipeline for real-time enterprise generative AI apps
How to build a data streaming pipeline for real-time enterprise generative AI apps using https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-kafka-overview + https://azure.microsoft.com/en-us/products/ai-services/openai-service + https://pathway.com/’s https://github.com/pathwaycom/llm-app+https://streamlit.io/ The source code is on GitHub: https://github.com/pathway-labs/azure-openai-real-time-data-app/tree/main Real-time AI app needs real-time data to respond with the most up-to-date information to user queries or perform quick actions autonomously. For example, a customer support team wants to improve its customer support by analyzing customer feedback and inquiries in real-time. They aim to understand common issues, track customer sentiment, and identify areas for improvement in their products and services. To achieve this, they need a system that can process large data streams, analyze text for insights, and present these insights in an accessible way. To help them we will build a https://github.com/pathway-labs/azure-openai-real-time-data-app with https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-kafka-overview, https://github.com/pathwaycom/llm-app, and https://azure.microsoft.com/en-us/products/ai-services/openai-service. This integrated system leverages the strengths of Pathway for robust data processing, LLMs like GPT for advanced text analytics, and Streamlit for user-friendly data visualization. This combination empowers businesses to build and deploy enterprise AI applications that provide the freshest contextual visual data. The new solution can help to multiple teams: Customer Support Team: They can use the dashboard to monitor customer satisfaction and common issues in real-time, allowing for quick responses. Product Development: Insights from customer feedback can inform product development, highlighting areas for improvement or innovation. Marketing and PR: Understanding customer sentiment trends helps in tailoring marketing campaigns and managing public relations more effectively. Implementation Let’s break down the main parts of the application architecture and understand the role of each in our solution. The project source code, deployment automation implementation, and setup guidelines can be found on https://github.com/pathway-labs/azure-openai-real-time-data-app. Azure Event Hubs & Kafka: Real-Time Data Streaming and Processing Azure Event Hubs collects real-time data from various sources, such as customer feedback forms, support chat logs, and social media mentions. This data is then streamed into a Kafka cluster for further processing. Large Language Models (LLMs) like GPT from Azure OpenAI: Text Analysis and Sentiment Detection The text data from Kafka is fed into an LLM for natural language processing using Pathway. This model performs sentiment analysis, key phrase extraction, and feedback categorization (e.g., identifying common issues or topics). Pathway to enable real-time data pipeline Pathway gains access to the https://pathway.com/developers/user-guide/concepts/welcome/ from Azure Event Hubs, it preprocesses, transforms, or joins them and the https://github.com/pathwaycom/llm-app helps to bring real-time context to the AI App with real-time vector indexing, semantic search, and retrieval capabilities. The text content of the events will be sent to Azure OpenAI embedding APIs via the LLM App to compute the embeddings and vector representations will be indexed using https://pathway.com/developers/showcases/lsh/lsh_chapter1. Using the LLM app, the company can gain deep insights from unstructured text data, understanding the sentiment and nuances of customer feedback. Streamlit: Interactive Dashboard for Visualization Streamlit is used to create an interactive web dashboard that visualizes the insights derived from customer feedback. This dashboard can show real-time metrics such as overall sentiment trends, and common topics in customer feedback, and even alert the team to emerging issues (See https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/drive_alert implementation of alerting to enhance this project). Here is a short demo of running the app in Azure: Overview of the Azure services the sample project uses Service Purpose https://azure.microsoft.com/en-us/products/ai-services?activetab=pivot:azureopenaiservicetab To use Azure OpenAI GPT model and embeddings. https://azure.microsoft.com/en-us/products/event-hubs To stream real-time events from various data sources. https://learn.microsoft.com/azure/container-apps/?WT.mc_id=javascript-0000-cxa Hosts our containerized applications (backend and frontend) with features like auto-scaling and load balancing. https://learn.microsoft.com/azure/container-registry/?WT.mc_id=javascript-0000-cxa Stores our Docker container images in a managed, private registry. https://learn.microsoft.com/azure/log-analytics/?WT.mc_id=javascript-0000-cxa Collects and analyzes telemetry and logs for insights into application performance and diagnostics. https://learn.microsoft.com/azure/azure-monitor/?WT.mc_id=javascript-0000-cxa Provides comprehensive monitoring of our applications, infrastructure, and network. Azure infrastructure with the main components As you can see in the below infrastructural diagram, we use two https://learn.microsoft.com/azure/container-apps/?WT.mc_id=javascript-0000-cxa to deploy the Pathway LLM App and Streamlit UI dashboard to a single Azure Resource Group. Simple architecture and reduced costs The current solution with the LLM App simplifies the AI pipeline infrastructure by consolidating capabilities into one platform. No need to integrate and maintain separate modules for your Gen AI app: Vector Databases (e.g. Pinecone/Weaviate/Qdrant) + LangChain + Cache (e.g. Redis) + API Framework (e.g. Fast API). It also reduced the cloud costs compared to other possible implementations using Vector Database/Azure Cognitive Search + Azure Functions (For hosting API and logic code) + Azure App Service. Let’s calculate it using the https://azure.microsoft.com/en-us/pricing/calculator/. For simplicity, we do not count costs for Azure OpenAI. 1. In the current solution with two Azure Container Apps (10 million requests per month) + one Azure Event Hubs (10 million events per month), you can see the total estimated cost per month here which is around 11 USD with the basic setup. See the report: https://azure.com/e/d3f1261757d14dc5a9ea1c414a00069f 2. In the second solution, we use Azure Event Hubs + Azure Function (to ingest real-time events and logic code) + Azure AI Search (vector search) + Container Apps (To host Streamlit UI Dashboard). You will end with an estimated monthly cost of 17 USD per month. We did not count costs for Storage accounts required by Functions. As you can see, the See the report: https://azure.com/e/0c934b47b2b745f596d59121f4020677 Tutorial - Creating the app The app development consists of two parts: backend API and frontend UI. Part 1: Design the Streamlit UI We will start with constructing Streamlit UI and create a simple web application with Streamlit. It interacts with the LLM App backend service over REST API and displays the insights derived from real-time customer feedback. See the full source code in the https://github.com/pathway-labs/azure-openai-real-time-data-app/blob/main/app/frontend/app.py file. st.title("Customer support and sentiment analysis dashboard") st.subheader("Example prompt") default_prompt = "Provide overall sentiment trends, and common topics and rating over time and sources with counts based on last feedback events and respond only in json without explanation and new line." st.text(default_prompt) placeholder = st.empty() for seconds in range(200): url = f"{api_host}" data = {"query": default_prompt, "user": "user"} response = requests.post(url, json=data) if response.status_code == 200: data_response = response.json() json_data = json.loads(data_response) with placeholder.container(): # Sentiment Trends sentiment_df = pd.DataFrame(list(json_data["sentiment_trends"].items()), columns=['Sentiment', 'Count']) color_map = {"positive": "green", "negative": "red", "neutral": "blue"} fig_sentiment = px.bar(sentiment_df, x='Sentiment', y='Count', title="Sentiment Trends", color='Sentiment', color_discrete_map=color_map) # Rating Over Time rating_data = json_data["rating_over_time"] rating_df = pd.DataFrame(rating_data) rating_df['Date'] = pd.to_datetime(rating_df['date']) fig_rating = px.line(rating_df, x='Date', y='rating', title="Average Rating Over Time", markers=True) # Streamlit layout st.plotly_chart(fig_sentiment, use_container_width=True) st.plotly_chart(fig_rating, use_container_width=True) # Convert the source counts to a DataFrame for visualization sources_df = pd.DataFrame(json_data["common_topics"], columns=['topic', 'count']) fig_sources = px.bar(sources_df, x='topic', y='count', title="Common Topics") st.plotly_chart(fig_sources, use_container_width=True) sources_df = pd.DataFrame(json_data["common_sources"], columns=['source', 'count']) fig_sources = px.bar(sources_df, x='source', y='count', title="Common Sources") st.plotly_chart(fig_sources, use_container_width=True) time.sleep(1) else: st.error( f"Failed to send data to API. Status code: {response.status_code}" ) In the above code, we define a default prompt to instruct the LLM App to respond with all necessary data such as sentiment trends, common topics, ratings over time, and common sources as structured in JSON format. We use https://docs.streamlit.io/library/api-reference/charts to visualize different dashboards. The dashboards are updated every second, likely fetching and displaying new data each time. Part 2: Build a backend API Next, we develop the backend logic where the app ingests streaming data from a Kafka topic provided by Azure Event Hubs and uses it to respond to user queries via an HTTP API. The function integrates with Azure OpenAI's Embeddings API for generating embeddings and a ChatGPT model for generating LLM responses. See the full source code in the https://github.com/pathway-labs/azure-openai-real-time-data-app/blob/main/app/frontend/app.py file. ... def run( *, host: str = "0.0.0.0", port: int = 8080 ): # Real-time data coming from the Kafka topic topic_data = pw.io.kafka.read( rdkafka_settings, topic="eventhubpathway", format="raw", autocommit_duration_ms=1000, ) # Tranform data to structured document transformed_topic_data = transform(topic_data) # Compute embeddings for each Kafka event using the OpenAI Embeddings API embedded_topic_data = embeddings(context=transformed_topic_data, data_to_embed=transformed_topic_data.doc) # Construct an index on the generated embeddings in real-time index = index_embeddings(embedded_topic_data) # Given a user question as a query from your API query, response_writer = pw.io.http.rest_connector( host=host, port=port, schema=QueryInputSchema, autocommit_duration_ms=50, ) # Generate embeddings for the query from the OpenAI Embeddings API embedded_query = embeddings(context=query, data_to_embed=pw.this.query) # Build prompt using indexed data responses = prompt(index, embedded_query, pw.this.query) # Feed the prompt to ChatGPT and obtain the generated answer. response_writer(responses) pw.run() ... Generated vector embeddings are indexed for efficient retrieval in real-time. The user's query is embedded using the same embeddings API endpoint, to enable a vector search against the indexed Kafka data. Finally, a new prompt is constructed using the indexed data and the embedded user query. This prompt is then used to generate a response by querying a model like ChatGPT. What is next As we have seen in the example of the customer feedback analysis app demo, used for businesses looking to harness real-time data for strategic decision-making and responsive customer service. This simplification in the architecture and implementation with Pathway’s LLM App means that your GenAI apps go to market within a short period (4-6 weeks), with lower costs and high security. Consider also visiting another showcase on https://pathway.com/developers/showcases/llm-alert-pathway/#use-llms-for-notifications-crafting-a-rag-app-with-real-time-alerting. You will see a few examples showcasing different possibilities with the LLM App in the GitHub Repo. Follow the instructions in https://github.com/pathwaycom/llm-app#get-started to try out different demos.
