Making Azure AI Foundry Agents Explainable — Knowledge Graphs + Source Attribution
90% of Azure AI demos work on stage — most never ship. The gap is architecture, not the model. I wrote up a field guide on taking Azure AI Foundry agents from POC to production, with knowledge graphs (GraphRAG) doing the grounding and source attribution. What the post covers Grounding with a knowledge graph — multi-hop retrieval + inline citations so every answer is traceable to a source (critical for regulated/medical use cases). Externalized state — keeping agent memory and session state outside the model. Identity at the boundary — Entra ID / RBAC instead of trusting the prompt. Observability & compliance — tracing, evaluation, and auditability. The stack — Azure AI Foundry + Model Context Protocol (MCP) + Azure Functions (Flex Consumption) + Azure OpenAI, from a real build (VeritasGraph medical MCP server). 📖 Full write-up: https://bibinprathap.com/blog/azure-ai-proof-of-concept-to-production ▶️ 6-min walkthrough: https://youtu.be/z-CPS5WUvyw?si=fmpo1RN28Bh9KbIb Questions for the community How are you grounding Foundry agents today — vector RAG, GraphRAG, or hybrid? Anyone combining knowledge graphs with MCP tools in Foundry? What worked / broke? For regulated domains, how are you handling source attribution and audit trails? Curious to compare notes — happy to share more detail on the GraphRAG retrieval design if useful.
How to Reliably Gauge LLM Confidence?
a { text-decoration: none; color: #464feb; } tr th, tr td { border: 1px solid #e6e6e6; } tr th { background-color: #f5f5f5; } I’m trying to estimate an LLM’s confidence in its answers in a way that correlates with correctness. Self-reported confidence is often misleading, and raw token probabilities mostly reflect fluency rather than truth. I don’t have grounding options like RAG, human feedback, or online search, so I’m looking for approaches that work in this constraint. What techniques have you found effective—entropy-based signals, calibration (temperature scaling), self-evaluation, or others? Any best practices for making confidence scores actionable?Azure OpenAI Model Upgrades: Prompt Safety Pitfalls with GPT-4o and Beyond
Upgrading to New Azure OpenAI Models? Beware Your Old Prompts Might Break. I recently worked on upgrading our Azure OpenAI integration from gpt-35-turbo to gpt-4o-mini, expecting it to be a straightforward configuration change. Just update the Azure Foundry resource endpoint, change the model name, deploy the code — and voilà, everything should work as before. Right? Not quite. The Unexpected Roadblock As soon as I deployed the updated code, I started seeing 400 status errors from the OpenAI endpoint. The message was cryptic: The response was filtered due to the prompt triggering Azure OpenAI's content management policy. At first, I assumed it was a bug in my SDK call or a malformed payload. But after digging deeper, I realized this wasn’t a technical failure — it was a content safety filter kicking in before the prompt even reached the model. The Prompt That Broke It Here’s the original system prompt that worked perfectly with gpt-35-turbo: YOU ARE A QNA EXTRACTOR IN TEXT FORMAT. YOU WILL GET A SET OF SURVEYJS QNA JSONS. YOU WILL CONVERT THAT INTO A TEXT DOCUMENT. FOR THE QUESTIONS WHERE NO ANSWER WAS GIVEN, MARK THOSE AS NO ANSWER. HERE IS THE QNA: BE CREATIVE AND PROFESSIONAL. I WANT TO GENERATE A DOCUMENT TO BE PUBLISHED. {{$style}} +++++ {{$input}} +++++ This prompt had been reliable for months. But with gpt-4o-mini, it triggered Azure’s new input safety layer, introduced in mid-2024. What Changed with GPT-4o-mini? Unlike gpt-35-turbo, the gpt-4o family: Applies stricter content filtering — not just on the output, but also on the input prompt. Treats system messages and user messages as role-based chat messages, passing them through moderation before the model sees them. Flags prompts that look like prompt injection attempts like aggressive instructions like “YOU ARE…”, “BE CREATIVE”, “GENERATE”, “PROFESSIONAL”. Flags unusual formatting (like `+++++`), artificial delimiters or token markers as it may look like encoded content. In short, the model didn’t even get a chance to process my prompt — it was blocked at the gate. Fixing It: Softening the Prompt The solution wasn’t to rewrite the entire logic, but to soften the system prompt and remove formatting that could be misinterpreted. Here’s what helped: - Replacing “YOU ARE…” with a gentler instruction like “Please help convert the following Q&A data…” - Removing creative directives like “BE CREATIVE” or “PROFESSIONAL” unless clearly contextualized. - Avoiding raw JSON markers and template syntax (`{{ }}`, `+++++`) in the prompt. Once I made these changes, the model responded smoothly — and the upgrade was finally complete. Evolving the Prompt — Not Abandoning It Interestingly, for some prompts I didn’t have to completely eliminate the “YOU ARE…” structure. Instead, I refined it to be more natural and less directive. Here’s a comparison: ❌ Old Prompt (Blocked) ✅ New Prompt (Accepted) YOU ARE A SOURCING AND PROCUREMENT MANAGER. YOU WILL GET BUYER'S REQUIREMENTS IN QNA FORMAT. HERE IS THE QNA: {{$input}} +++++ YOU WILL GENERATE TOP 10 {{$category}} RELATED QUESTIONS THAT CAN BE ASKED OF A SUPPLIER IN JSON FORMAT. THE JSON MUST HAVE QUESTION NUMBER AS THE KEY AND QUESTION TEXT AS THE QUESTION. DON'T ADD ANY DESCRIPTION TEXT OR FORMATTING IN THE OUTPUT. BE CREATIVE AND PROFESSIONAL. I WANT TO GENERATE AN RFX. You are an AI assistant that helps clarify sourcing requirements. You will receive buyer's requirements in QnA format. Here is the QnA: {$input} Your task is to generate the top 10 {$category} related questions that can be asked of a supplier, in JSON format. - The JSON must use the question number as the key and the question text as the value. - Do not include any description text or formatting in the output. - Focus on creating clear, professional, and relevant questions that will help prepare an RFX. Key Takeaways - Model upgrades aren’t just about configuration changes — they can introduce new moderation layers that affect prompt design. - Prompt safety filtering is now a first-class citizen in Azure OpenAI, especially for newer models. - System prompts need to be rewritten with moderation in mind, not just clarity or creativity. This experience reminded me that even small upgrades can surface big learning moments. If you're planning to move to gpt-4o-mini or any newer Azure OpenAI model, take a moment to review your prompts — they might need a little more finesse than before.
Azure AI Foundry/Azure AI Service - cannot access agents
I'm struggling with getting agents to work via API which were defined in AI Foundry (based on Azure AI Service). When defining agent in project in AI Foundry I can use it in playground via web browser. The issue appears when I'm trying to access them via API (call from Power Automate). When executing Run on agent I get info that agent cannot be found. The issue doesn't exist when using Azure OpenAI and defining assistants. I can use them both via API and web browser. I guess that another layer of management which is project might be an issue here. I saw usage of SDK in Python and first call is to connect to a project and then get an agent. Does anyone of you experienced the same? Is a way to select and run agent via API?
Principle Does not have Access to API/Operation
Hi all, I am trying to connect Azure OpenAI service to Azure AI Search service to Azure Gen 2 Data lake. In the Azure AI Foundry studio Chat Playground, I am able to add my data source, which is a .csv file in the data lake that has been indexed successfully. I use "System Assigned Managed Identity". The following RBAC has been applied: AI Search service has Cognitive Services OpenAI Contributor in Azure Open AI service Azure OpenAI service has Search Index Data Reader in AI Search Service Azure OpenAI service has Search Service Contributor in AI Search Service AI Search Service has Storage Blob Data Reader in Storage account (Data Lake) As mentioned when adding the data source it passes validation but when I try to ask a question, I get the error "We couldn't connect your data Principal does not have access to API/Operation"
Enhance Large Language Models (LLM)/Azure OpenAI Performance and Cost Efficiency
Basic techniques to optimize performance and cost efficiency of Large Language Models (LLM) in Azure OpenAI. Selection of Optimal Models: Choosing the right model is crucial for achieving the best performance. This involves understanding the capabilities of each model and selecting the one that best suits the task at hand. Factors to consider include the model's language capabilities, its ability to generate creative content, and its performance on tasks similar to the one you're working on. Enhancing Prompts: The quality of the prompts you provide to the model can significantly impact its performance. A well-crafted prompt can guide the model to generate more relevant and useful responses. This might involve specifying the format you want the answer in, or asking the model to think step-by-step or debate pros and cons before settling on an answer. Controlling Call Rates: To optimize costs and performance, it's important to manage the rate at which you make API calls. This involves understanding the rate limits of the API and designing your application to stay within these limits. You might need to implement a queuing system or use exponential backoff in case of rate limit errors. Provisioned Throughput Units (PTUs): PTUs allow you to reserve capacity for your application, ensuring consistent performance even during peak times. By correctly provisioning PTUs, you can balance cost and performance to meet your application's needs. Integration with Networks and Security: Azure OpenAI can be integrated with your existing network and security infrastructure. This might involve setting up Virtual Networks, using Private Link for secure communication, and managing access with Azure Active Directory. Integration with Data Sources and Cognitive Enterprise Search: Azure OpenAI can be used in conjunction with various data sources and Azure's Cognitive Search service. This allows you to build powerful applications that can search through large amounts of data, understand it, and generate human-like text based on it. Efficient Search using Vector Search: Vector search is a method of retrieving information that involves converting text into high-dimensional vectors and searching for similar vectors. This can be much more efficient than traditional search methods, especially for large datasets. Azure OpenAI includes support for vector search, allowing you to build efficient, AI-powered search applications. Utilizing Agent Pools for Different Tasks: Agent pools are a powerful feature that allows you to manage and distribute workloads across different resources. By creating separate agent pools for different tasks, you can ensure that each task has the resources it needs to run efficiently. This can help to optimize performance and reduce costs. Use Clear Meta Prompts and System Messages for Desired Outputs: Meta prompts and system messages play a crucial role in guiding the AI to produce the desired outputs. A meta prompt is a directive that instructs the AI on the format or type of response required. Use AI model combination for simple tasks and LLM for complex tasks: Different AI models have different strengths and capabilities. For simple tasks, such as answering straightforward questions or generating short pieces of text, a combination of smaller, specialized AI models might be sufficient. These models can be faster and more efficient than larger models, and can often produce high-quality results for simple tasks. It's my hope that these 10 methods will deepen your understanding of how to optimize large Language Models, thereby improving performance and cost efficiency.How to build a data streaming pipeline for real-time enterprise generative AI apps
How to build a data streaming pipeline for real-time enterprise generative AI apps using https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-kafka-overview + https://azure.microsoft.com/en-us/products/ai-services/openai-service + https://pathway.com/’s https://github.com/pathwaycom/llm-app+https://streamlit.io/ The source code is on GitHub: https://github.com/pathway-labs/azure-openai-real-time-data-app/tree/main Real-time AI app needs real-time data to respond with the most up-to-date information to user queries or perform quick actions autonomously. For example, a customer support team wants to improve its customer support by analyzing customer feedback and inquiries in real-time. They aim to understand common issues, track customer sentiment, and identify areas for improvement in their products and services. To achieve this, they need a system that can process large data streams, analyze text for insights, and present these insights in an accessible way. To help them we will build a https://github.com/pathway-labs/azure-openai-real-time-data-app with https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-kafka-overview, https://github.com/pathwaycom/llm-app, and https://azure.microsoft.com/en-us/products/ai-services/openai-service. This integrated system leverages the strengths of Pathway for robust data processing, LLMs like GPT for advanced text analytics, and Streamlit for user-friendly data visualization. This combination empowers businesses to build and deploy enterprise AI applications that provide the freshest contextual visual data. The new solution can help to multiple teams: Customer Support Team: They can use the dashboard to monitor customer satisfaction and common issues in real-time, allowing for quick responses. Product Development: Insights from customer feedback can inform product development, highlighting areas for improvement or innovation. Marketing and PR: Understanding customer sentiment trends helps in tailoring marketing campaigns and managing public relations more effectively. Implementation Let’s break down the main parts of the application architecture and understand the role of each in our solution. The project source code, deployment automation implementation, and setup guidelines can be found on https://github.com/pathway-labs/azure-openai-real-time-data-app. Azure Event Hubs & Kafka: Real-Time Data Streaming and Processing Azure Event Hubs collects real-time data from various sources, such as customer feedback forms, support chat logs, and social media mentions. This data is then streamed into a Kafka cluster for further processing. Large Language Models (LLMs) like GPT from Azure OpenAI: Text Analysis and Sentiment Detection The text data from Kafka is fed into an LLM for natural language processing using Pathway. This model performs sentiment analysis, key phrase extraction, and feedback categorization (e.g., identifying common issues or topics). Pathway to enable real-time data pipeline Pathway gains access to the https://pathway.com/developers/user-guide/concepts/welcome/ from Azure Event Hubs, it preprocesses, transforms, or joins them and the https://github.com/pathwaycom/llm-app helps to bring real-time context to the AI App with real-time vector indexing, semantic search, and retrieval capabilities. The text content of the events will be sent to Azure OpenAI embedding APIs via the LLM App to compute the embeddings and vector representations will be indexed using https://pathway.com/developers/showcases/lsh/lsh_chapter1. Using the LLM app, the company can gain deep insights from unstructured text data, understanding the sentiment and nuances of customer feedback. Streamlit: Interactive Dashboard for Visualization Streamlit is used to create an interactive web dashboard that visualizes the insights derived from customer feedback. This dashboard can show real-time metrics such as overall sentiment trends, and common topics in customer feedback, and even alert the team to emerging issues (See https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/drive_alert implementation of alerting to enhance this project). Here is a short demo of running the app in Azure: Overview of the Azure services the sample project uses Service Purpose https://azure.microsoft.com/en-us/products/ai-services?activetab=pivot:azureopenaiservicetab To use Azure OpenAI GPT model and embeddings. https://azure.microsoft.com/en-us/products/event-hubs To stream real-time events from various data sources. https://learn.microsoft.com/azure/container-apps/?WT.mc_id=javascript-0000-cxa Hosts our containerized applications (backend and frontend) with features like auto-scaling and load balancing. https://learn.microsoft.com/azure/container-registry/?WT.mc_id=javascript-0000-cxa Stores our Docker container images in a managed, private registry. https://learn.microsoft.com/azure/log-analytics/?WT.mc_id=javascript-0000-cxa Collects and analyzes telemetry and logs for insights into application performance and diagnostics. https://learn.microsoft.com/azure/azure-monitor/?WT.mc_id=javascript-0000-cxa Provides comprehensive monitoring of our applications, infrastructure, and network. Azure infrastructure with the main components As you can see in the below infrastructural diagram, we use two https://learn.microsoft.com/azure/container-apps/?WT.mc_id=javascript-0000-cxa to deploy the Pathway LLM App and Streamlit UI dashboard to a single Azure Resource Group. Simple architecture and reduced costs The current solution with the LLM App simplifies the AI pipeline infrastructure by consolidating capabilities into one platform. No need to integrate and maintain separate modules for your Gen AI app: Vector Databases (e.g. Pinecone/Weaviate/Qdrant) + LangChain + Cache (e.g. Redis) + API Framework (e.g. Fast API). It also reduced the cloud costs compared to other possible implementations using Vector Database/Azure Cognitive Search + Azure Functions (For hosting API and logic code) + Azure App Service. Let’s calculate it using the https://azure.microsoft.com/en-us/pricing/calculator/. For simplicity, we do not count costs for Azure OpenAI. 1. In the current solution with two Azure Container Apps (10 million requests per month) + one Azure Event Hubs (10 million events per month), you can see the total estimated cost per month here which is around 11 USD with the basic setup. See the report: https://azure.com/e/d3f1261757d14dc5a9ea1c414a00069f 2. In the second solution, we use Azure Event Hubs + Azure Function (to ingest real-time events and logic code) + Azure AI Search (vector search) + Container Apps (To host Streamlit UI Dashboard). You will end with an estimated monthly cost of 17 USD per month. We did not count costs for Storage accounts required by Functions. As you can see, the See the report: https://azure.com/e/0c934b47b2b745f596d59121f4020677 Tutorial - Creating the app The app development consists of two parts: backend API and frontend UI. Part 1: Design the Streamlit UI We will start with constructing Streamlit UI and create a simple web application with Streamlit. It interacts with the LLM App backend service over REST API and displays the insights derived from real-time customer feedback. See the full source code in the https://github.com/pathway-labs/azure-openai-real-time-data-app/blob/main/app/frontend/app.py file. st.title("Customer support and sentiment analysis dashboard") st.subheader("Example prompt") default_prompt = "Provide overall sentiment trends, and common topics and rating over time and sources with counts based on last feedback events and respond only in json without explanation and new line." st.text(default_prompt) placeholder = st.empty() for seconds in range(200): url = f"{api_host}" data = {"query": default_prompt, "user": "user"} response = requests.post(url, json=data) if response.status_code == 200: data_response = response.json() json_data = json.loads(data_response) with placeholder.container(): # Sentiment Trends sentiment_df = pd.DataFrame(list(json_data["sentiment_trends"].items()), columns=['Sentiment', 'Count']) color_map = {"positive": "green", "negative": "red", "neutral": "blue"} fig_sentiment = px.bar(sentiment_df, x='Sentiment', y='Count', title="Sentiment Trends", color='Sentiment', color_discrete_map=color_map) # Rating Over Time rating_data = json_data["rating_over_time"] rating_df = pd.DataFrame(rating_data) rating_df['Date'] = pd.to_datetime(rating_df['date']) fig_rating = px.line(rating_df, x='Date', y='rating', title="Average Rating Over Time", markers=True) # Streamlit layout st.plotly_chart(fig_sentiment, use_container_width=True) st.plotly_chart(fig_rating, use_container_width=True) # Convert the source counts to a DataFrame for visualization sources_df = pd.DataFrame(json_data["common_topics"], columns=['topic', 'count']) fig_sources = px.bar(sources_df, x='topic', y='count', title="Common Topics") st.plotly_chart(fig_sources, use_container_width=True) sources_df = pd.DataFrame(json_data["common_sources"], columns=['source', 'count']) fig_sources = px.bar(sources_df, x='source', y='count', title="Common Sources") st.plotly_chart(fig_sources, use_container_width=True) time.sleep(1) else: st.error( f"Failed to send data to API. Status code: {response.status_code}" ) In the above code, we define a default prompt to instruct the LLM App to respond with all necessary data such as sentiment trends, common topics, ratings over time, and common sources as structured in JSON format. We use https://docs.streamlit.io/library/api-reference/charts to visualize different dashboards. The dashboards are updated every second, likely fetching and displaying new data each time. Part 2: Build a backend API Next, we develop the backend logic where the app ingests streaming data from a Kafka topic provided by Azure Event Hubs and uses it to respond to user queries via an HTTP API. The function integrates with Azure OpenAI's Embeddings API for generating embeddings and a ChatGPT model for generating LLM responses. See the full source code in the https://github.com/pathway-labs/azure-openai-real-time-data-app/blob/main/app/frontend/app.py file. ... def run( *, host: str = "0.0.0.0", port: int = 8080 ): # Real-time data coming from the Kafka topic topic_data = pw.io.kafka.read( rdkafka_settings, topic="eventhubpathway", format="raw", autocommit_duration_ms=1000, ) # Tranform data to structured document transformed_topic_data = transform(topic_data) # Compute embeddings for each Kafka event using the OpenAI Embeddings API embedded_topic_data = embeddings(context=transformed_topic_data, data_to_embed=transformed_topic_data.doc) # Construct an index on the generated embeddings in real-time index = index_embeddings(embedded_topic_data) # Given a user question as a query from your API query, response_writer = pw.io.http.rest_connector( host=host, port=port, schema=QueryInputSchema, autocommit_duration_ms=50, ) # Generate embeddings for the query from the OpenAI Embeddings API embedded_query = embeddings(context=query, data_to_embed=pw.this.query) # Build prompt using indexed data responses = prompt(index, embedded_query, pw.this.query) # Feed the prompt to ChatGPT and obtain the generated answer. response_writer(responses) pw.run() ... Generated vector embeddings are indexed for efficient retrieval in real-time. The user's query is embedded using the same embeddings API endpoint, to enable a vector search against the indexed Kafka data. Finally, a new prompt is constructed using the indexed data and the embedded user query. This prompt is then used to generate a response by querying a model like ChatGPT. What is next As we have seen in the example of the customer feedback analysis app demo, used for businesses looking to harness real-time data for strategic decision-making and responsive customer service. This simplification in the architecture and implementation with Pathway’s LLM App means that your GenAI apps go to market within a short period (4-6 weeks), with lower costs and high security. Consider also visiting another showcase on https://pathway.com/developers/showcases/llm-alert-pathway/#use-llms-for-notifications-crafting-a-rag-app-with-real-time-alerting. You will see a few examples showcasing different possibilities with the LLM App in the GitHub Repo. Follow the instructions in https://github.com/pathwaycom/llm-app#get-started to try out different demos.
