Thursday Architecture Lesson #4: Designing Production-Ready AI Agent Architecture on Azure
AI agents are moving beyond simple chat experiences. The real enterprise question is no longer: “How do we build an AI chatbot?” It is: “How do we design an AI agent that can securely understand context, use enterprise data, call tools, take actions, and operate reliably in production?” That requires an architecture—not just a prompt. 🏗️ A Practical Azure AI Agent Architecture A production-oriented agent architecture can be thought of as six layers: 1️⃣ User & Experience Layer Users interact through: Microsoft Teams Web applications Copilot experiences APIs Custom enterprise applications The interface should remain separate from the agent's core reasoning and business logic. 2️⃣ Agent & Orchestration Layer This is where the agent reasons about the task, decides what to do next, and orchestrates tools and workflows. Microsoft Foundry provides an enterprise platform for building and operating AI agents, while Microsoft Agent Framework provides an open-source approach for building and orchestrating agents. 3️⃣ Knowledge & Grounding Layer An enterprise agent should not depend only on model knowledge. It may need access to: Enterprise documents Azure AI Search Databases Microsoft Graph APIs Business applications The goal is to provide the agent with the right context at the right time. 4️⃣ Tool & Action Layer This is where an agent becomes more than a chatbot. The agent may: → Query a database → Create a ticket → Call an API → Search documentation → Trigger an Azure workflow → Update a business system Microsoft Foundry supports connecting agents to business systems and tools, including integrations through APIs and Model Context Protocol (MCP). 5️⃣ Security & Governance Layer This layer is critical in enterprise AI. Consider: Microsoft Entra ID RBAC Managed identities Network isolation Data permissions Prompt-injection protection Content safety Auditability An agent should have only the permissions it needs to perform its task. 6️⃣ Observability & Evaluation Layer Production agents need more than application logs. Architects should monitor: Agent traces Tool calls Latency Token consumption Groundedness Accuracy Safety Failed workflows Business outcomes Microsoft Foundry is expanding its capabilities around tracing, evaluation, optimization, and runtime controls for production agents. 🔄 The Architecture Pattern A simplified enterprise flow looks like: User → Application → Agent → Knowledge + Tools → Enterprise Systems while: Security + Governance + Observability should span across the entire architecture. This is an important architectural shift. We are moving from: Application → API → Database towards: User → Agent → Reasoning → Tools → Enterprise Systems But the traditional architecture principles still matter: Identity. Security. Networking. Data. Reliability. Governance. Cost. AI does not replace architecture. It makes architecture more important. 💡 One Important Design Principle Don't give an AI agent unrestricted access to every system. Instead: Least privilege + bounded tools + controlled actions + observable execution should be the foundation. For example: A support agent might have permission to: ✅ Read customer information ✅ Search knowledge articles ✅ Create a support ticket But not: ❌ Delete customer records ❌ Change billing information ❌ Execute unrestricted database commands The agent's capabilities should be deliberately designed. 🎯 Architecture Takeaway When designing enterprise AI agents, don't start with: “Which model should we use?” Start with: What should the agent do? Then design: Context → Tools → Identity → Security → Data → Runtime → Observability → Business Outcome The model is only one component of the architecture. Architecture Question If you were designing an enterprise AI agent today, which layer would you focus on first? Data & Grounding Security & Identity Agent Orchestration Tools & Integrations Observability I'd be interested to hear how other architects approach this.Thursday Architecture Lesson #2
Azure Storage: Blob, Files, Disks or Azure SQL? Don't memorize the services. Understand the workload. Remember this: B → Blob Storage Need object storage for images, documents, backups, logs or large unstructured data. F → Azure Files Need shared file storage that multiple applications or servers can access using standard file shares. D → Managed Disks Need persistent block storage attached to Azure VMs. S → Azure SQL Need relational data with structured schemas, transactions, relationships and SQL capabilities. 🧠 The Architect's Rule: "Don't choose storage because it is available. Choose it because the workload needs it." The right storage architecture isn't the one with the most features. It's the one that gives the business the right balance of: Performance • Cost • Availability • Security • Scalability • Data access requirements • Operational complexity For example, storing application documents in Azure SQL just because the application already uses SQL may not always be the best architecture. Likewise, using Blob Storage for transactional relational data creates a completely different set of challenges. The question an architect should ask is not: "Which Azure storage service is the most powerful?" Instead ask: "What type of data do I have, and how does the application need to use it?" Architecture Lesson #2 takeaway: Start with the data characteristics and access pattern. Then choose the storage service. What is the first thing you consider when choosing an Azure storage service?Thursday Architecture Lesson #1
Azure Compute: VM, App Service, Container Apps or AKS? Don't memorize the services. Remember this: V → VM Need maximum control or have a legacy workload. A → App Service Want managed hosting for web apps or APIs. C → Container Apps Want containers without taking on Kubernetes complexity. K → AKS You genuinely need Kubernetes capabilities. 🧠 The Architect's Rule: “Don't choose AKS because you can. Choose AKS because you need it.” The right architecture isn't the one with the most powerful technology. It's the one that gives the business the right balance of: Capability • Cost • Security • Scalability • Operational complexity For example, if a simple web application can run effectively on App Service, introducing Kubernetes may add complexity without adding meaningful business value. The best architects don't ask: “What is the most advanced technology we can use?” They ask: “What is the simplest architecture that meets the requirements?” 💬 Your turn: If you were modernizing a traditional 3-tier application today, which would you choose first — VM, App Service, Container Apps or AKS? And most importantly, why?From AI PoC to Production: 7 Architecture Decisions Every Enterprise Must Get Right
Architecture Deep Dive Moving from an AI Proof of Concept to an enterprise-ready production workload requires much more than selecting the right model. AI is no longer just an experimentation topic. Across enterprises, teams are building copilots, RAG applications, AI agents, intelligent automation and domain-specific AI solutions. But there is a significant difference between making an AI Proof of Concept work and making an AI solution production-ready. A PoC asks: “Can we make AI do this?” Production asks much harder questions: “Can we make it secure, reliable, scalable, observable, governed and financially sustainable?” That is where architecture becomes critical. Microsoft’s Azure Well-Architected guidance for AI workloads highlights that AI systems introduce architectural considerations beyond traditional applications, including nondeterministic behavior, grounding data, model operations, testing, responsible AI and continuous evaluation. Here are seven architecture decisions I believe every enterprise should consider before moving an AI workload from PoC to production. 1. Start With the Business Outcome — Not the Model One of the most common mistakes is starting with: “Which AI model should we use?” The better question is: “What business problem are we solving?” Before selecting a model or Azure service, define: • The business outcome • The users • The expected experience • The measurable success criteria • Regulatory and compliance requirements • Data sensitivity • Expected scale For example: Instead of saying: “We want to build an enterprise chatbot.” Define the outcome: “We want employees to find accurate information from 500,000 internal documents in less than 5 seconds while respecting existing access permissions.” That single statement changes the architecture conversation completely. 2. Design the Data and Grounding Architecture First Enterprise AI is only as useful as the information it can access and trust. For many enterprise scenarios, the challenge isn't simply selecting a powerful model. The challenge is providing the model with the right context. This is where grounding and RAG architectures become important. A typical flow looks like: User → Application → Orchestration → Knowledge/Retrieval → Model → Response But an enterprise implementation also needs to consider: • Data ingestion • Chunking and enrichment • Metadata • Indexing • Access control • Data freshness • Source attribution • Retrieval quality • Auditability Microsoft's current AI architecture guidance explicitly treats the knowledge layer as a core architectural component and emphasizes enforcing data access policies and authorization within that layer. The key architectural question is therefore not: “Can the model answer the question?” It is: “Can the model answer the question using authorized, relevant and trustworthy enterprise data?” 3. Separate Intelligence, Inference, Knowledge and Tools AI applications are becoming more sophisticated. Modern architectures may involve models, agents, orchestration, enterprise data and external tools. Putting everything into one application layer quickly becomes difficult to secure, scale and operate. A better approach is to establish clear architectural boundaries. A useful conceptual model is: Client Layer ↓ Intelligence / Orchestration Layer ↓ Inference Layer ↓ Knowledge Layer ↓ Tools / Business APIs Each layer can have its own: • Identity • Security policies • Scaling strategy • Monitoring • Caching • Failure handling This separation becomes particularly important when moving from a simple chatbot to agentic AI applications. Microsoft's AI application design guidance recommends distinct client, intelligence, inference, knowledge and tools layers for intelligent applications. 4. Treat Security as an Architecture Principle — Not a Checklist AI introduces new security considerations. You need to think beyond traditional application security. Ask: • Who can access the AI application? • What data can the user retrieve? • Can the model access information the user cannot? • How are identities propagated across components? • How are prompts and responses protected? • How are AI tools authorized? • How are sensitive outputs detected? • How are activities audited? One particularly important principle is: The AI system should not become an alternative path around existing enterprise authorization. If an employee cannot access a document directly, the AI assistant should not expose that document through a generated response. Security therefore needs to exist across the entire AI architecture: Identity → Data → Retrieval → Model → Tools → Output 5. Design for Scale and Reliability Before You Need It A PoC might have: 10 users 100 documents 1 model 1 environment Production might have: 100,000 users Millions of documents Multiple models Multiple business applications Continuous availability requirements The architecture must therefore consider: • Horizontal scaling • Availability Zones • Regional resiliency • Load balancing • Model availability • Rate limiting • Failover • Caching • Capacity planning AI workloads also have unique infrastructure considerations. Inference capacity can become a bottleneck, and GPU-based workloads can introduce significant infrastructure costs. Microsoft's current Azure AI architecture guidance recommends designing for scalability and availability across the intelligence, orchestration, inference and knowledge layers. 6. Cost Must Be Designed Into the Architecture AI can create unexpected cost growth. A solution may work perfectly from a technical perspective and still fail the business case because of: • Token consumption • Model selection • GPU utilization • Storage • Data processing • Retrieval infrastructure • Logging • Network traffic • High-frequency inference Therefore, ask: “What is the expected cost per transaction?” Then model: Users × Requests × Tokens × Model Cost But don't stop there. Also evaluate: • Caching opportunities • Model routing • Smaller models for simpler tasks • Batch processing • GPU utilization • Resource scaling • Storage optimization Microsoft's Well-Architected guidance specifically highlights monitoring utilization and avoiding unnecessary AI infrastructure costs. The cheapest architecture isn't necessarily the best architecture. The goal is: Maximum business value per unit of AI spend. 7. Production Requires Continuous Evaluation and Observability Traditional applications usually monitor: CPU Memory Latency Errors Availability AI applications need more. You also need to understand: • Response quality • Grounding accuracy • Retrieval relevance • Hallucination rate • Model performance • Prompt effectiveness • Safety violations • User feedback • Token consumption • Cost per interaction AI is nondeterministic. The same input may not always produce exactly the same output. That means testing cannot simply end when the application goes live. Production evaluation becomes part of the architecture. Microsoft's guidance recommends extending observability to AI-specific quality metrics and supporting testing and evaluation with real production inputs. The Architecture Mindset Shift The biggest transition from PoC to production is not necessarily choosing a better model. It is changing the questions we ask. PoC thinking: “Can AI do it?” Production thinking: “Can the enterprise operate it safely and economically at scale?” That leads to a different architecture conversation: Business Outcome ↓ Data & Grounding ↓ Security & Identity ↓ AI Application Architecture ↓ Infrastructure & Scalability ↓ Observability & Governance ↓ Cost Optimization ↓ Continuous Evaluation My 7-Question Production Readiness Test Before approving an enterprise AI workload for production, I would ask: 1. What measurable business outcome are we delivering? 2. Can we trust and govern the data being used? 3. Can the AI respect existing identity and authorization boundaries? 4. Can every major component scale and recover from failure? 5. Can we measure AI quality—not just infrastructure health? 6. Do we understand the cost at production scale? 7. Can we continuously evaluate, improve and govern the solution? If the answer to several of these is “not yet”, the solution may still be a PoC. And that's perfectly fine. The objective isn't to rush an AI PoC into production. The objective is to build the architecture that makes production possible. Final Thought AI architecture is becoming less about: “Which model should we use?” And increasingly about: “How do we build an AI system that the enterprise can trust?” That is the real journey: PoC → Architecture → Production → Scale → Business Value The organizations that get this architecture right will be in a much stronger position to move from AI experimentation to sustainable enterprise AI adoption. What do you think is the biggest challenge when moving an enterprise AI solution from PoC to production — security, data, scalability, cost, or something else?