đ Agents League Winner Spotlight â Reasoning Agents Track
Agents League was designed to showcase what agentic AI can look like when developers move beyond singleâprompt interactions and start building systems that plan, reason, verify, and collaborate. Across three competitive tracksâCreative Apps, Reasoning Agents, and Enterprise Agentsâparticipants had two weeks to design and ship real AI agents using productionâready Microsoft and GitHub tools, supported by live coding battles, community AMAs, and async builds on GitHub. Today, weâre excited to spotlight the winning project for the Reasoning Agents track, built on Microsoft Foundry: CertPrep MultiâAgent System â Personalised Microsoft Exam Preparation by Athiq Ahmed. The Reasoning Agents Challenge Scenario The goal of the Reasoning Agents track challenge was to design a multiâagent system capable of effectively assisting students in preparing for Microsoft certification exams. Participants were asked to build an agentic workflow that could understand certification syllabi, generate personalized study plans, assess learner readiness, and continuously adapt based on performance and feedback. The suggested reference architecture modeled a realistic learning journey: starting from freeâform student input, a sequence of specialized reasoning agents collaboratively curated Microsoft Learn resources, produced structured study plans with timelines and milestones, and maintained learner engagement through reminders. Once preparation was complete, the system shifted into an assessment phase to evaluate readiness and either recommend the appropriate Microsoft certification exam or loop back into targeted remediationâemphasizing reasoning, decisionâmaking, and humanâinâtheâloop validation at every step. All details are available here: agentsleague/starter-kits/2-reasoning-agents at main ¡ microsoft/agentsleague. The Winning Project: CertPrep MultiâAgent System The CertPrep MultiâAgent System is an AI solution for personalized Microsoft certification exam preparation, supporting nine certification exam families. At a high level, the system turns freeâform learner input into a structured certification plan, measurable progress signals, and actionable recommendationsâdemonstrating exactly the kind of reasoned orchestration this track was designed to surface. Inside the MultiâAgent Architecture At its core, the system is designed as a multiâagent pipeline that combines sequential reasoning, parallel execution, and humanâinâtheâloop gates, with traceability and responsible AI guardrails. The solution is composed of eight specialized reasoning agents, each focused on a specific stage of the learning journey: LearnerProfilingAgent â Converts freeâtext background information into a structured learner profile using Microsoft Foundry SDK (with deterministic fallbacks). StudyPlanAgent â Generates a weekâbyâweek study plan using a constrained allocation algorithm to respect the learnerâs available time. LearningPathCuratorAgent â Maps exam domains to curated Microsoft Learn resources with trusted URLs and estimated effort. ProgressAgent â Computes a weighted readiness score based on domain coverage, time utilization, and practice performance. AssessmentAgent â Generates and evaluates domainâproportional mock exams. CertificationRecommendationAgent â Issues a clear âGO / CONDITIONAL GO / NOT YETâ decision with remediation steps and nextâcert suggestions. Throughout the pipeline, a 17ârule Guardrails Pipeline enforces validation checks at every agent boundary, and two explicit humanâinâtheâloop gates ensure that decisions are made only when sufficient learner confirmation or data is present. CertPrep leverages Microsoft Foundry Agent Service and related tooling to run this reasoning pipeline reliably and observably: Managed agents via Foundry SDK Structured JSON outputs using GPTâ4o (JSON mode) with conservative temperature settings Guardrails enforced through Azure Content Safety Parallel agent fanâout using concurrent execution Typed contracts with Pydantic for every agent boundary AI-assisted development with GitHub Copilot, used throughout for code generation, refactoring, and test scaffolding Notably, the full pipeline is designed to run in under one second in mock mode, enabling reliable demos without live credentials. User Experience: From Onboarding to Exam Readiness Beyond its backend architecture, CertPrep places strong emphasis on clarity, transparency, and user trust through a wellâstructured frontâend experience. The application is built with Streamlit and organized as a 7âtab interactive interface, guiding learners stepâbyâstep through their preparation journey. From a userâs perspective, the flow looks like this: Profile & Goals Input Learners start by describing their background, experience level, and certification goals in natural language. The system immediately reflects how this input is interpreted by displaying the structured learner profile produced by the profiling agent. Learning Path & Study Plan Visualization Once generated, the study plan is presented using visual aids such as Ganttâstyle timelines and domain breakdowns, making it easy to understand weekly milestones, expected effort, and progress over time. Progress Tracking & Readiness Scoring As learners move forward, the UI surfaces an examâweighted readiness score, combining domain coverage, study plan adherence, and assessment performanceâhelping users understand why the system considers them ready (or not yet). Assessments and Feedback Practice assessments are generated dynamically, and results are reported alongside actionable feedback rather than just raw scores. Transparent Recommendations Final recommendations are presented clearly, supported by reasoning traces and visual summaries, reinforcing trust and explainability in the agentâs decisionâmaking. The UI also includes an Admin Dashboard and demoâfriendly modes, enabling judges, reviewers, or instructors to inspect reasoning traces, switch between live and mock execution, and demonstrate the system reliably without external dependencies. Why This Project Stood Out This project embodies the spirit of the Reasoning Agents track in several ways: â Clear separation of reasoning roles, instead of promptâheavy monoliths â Deterministic fallbacks and guardrails, critical for educational and decisionâsupport systems â Observable, debuggable workflows, aligned with Foundryâs production goals â Explainable outputs, surfaced directly in the UX It demonstrates how agentic patterns translate cleanly into maintainable architectures when supported by the right platform abstractions. Try It Yourself Explore the project, architecture, and demo here: đ GitHub Issue (full project details): https://github.com/microsoft/agentsleague/issues/76 đĽ Demo video: https://www.youtube.com/watch?v=okWcFnQoBsE đ Live app (mock data): https://agentsleague.streamlit.app/From CI/CD to Continuous AI: The Future of GitHub Automation
Introduction For over a decade, CI/CD (Continuous Integration and Continuous Deployment) has been the backbone of modern software engineering. It helped teams move from manual, error-prone deployments to automated, reliable pipelines. But today, we are standing at the edge of another transformationâone that is far more powerful. Welcome to the era of Continuous AI. This new paradigm is not just about automating pipelinesâitâs about building self-improving, intelligent systems that can analyze, decide, and act with minimal human intervention. With the emergence of AI-powered workflows inside GitHub, automation is evolving from rule-based execution to context-aware decision-making. This article explores: What Continuous AI is How it differs from CI/CD Real-world use cases Architecture patterns Challenges and best practices What the future holds for engineering teams The Evolution: From CI to CI/CD to Continuous AI 1. Continuous Integration (CI) Developers merge code frequently Automated builds and tests validate changes Goal: Catch issues early 2. Continuous Deployment (CD) Code automatically deployed to production Reduced manual intervention Goal: Faster delivery 3. Continuous AI (The Next Step) Systems donât just executeâthey think and improve AI agents analyze code, detect issues, suggest fixes, and even implement them Goal: Autonomous software evolution What is Continuous AI? Continuous AI is a model where: Software systems continuously improve themselves using AI-driven insights and automated actions. Instead of static pipelines, you get: Intelligent workflows Context-aware automation Self-healing repositories Autonomous decision-making systems Key Characteristics Feature CI/CD Continuous AI Execution Rule-based AI-driven Flexibility Low High Decision-making Predefined Dynamic Learning None Continuous Output Build & deploy Improve & optimize Why Continuous AI Matters Traditional automation has limitations: It cannot adapt to new patterns It cannot reason about code quality It cannot proactively improve systems Continuous AI solves these problems by introducing: Context awareness Learning from past data Proactive optimization This leads to: Faster development cycles Higher code quality Reduced operational overhead Smarter engineering teams Core Components of Continuous AI in GitHub 1. AI Agents AI agents act as autonomous workers inside your repository. They can: Review pull requests Suggest improvements Generate tests Fix bugs 2. Agentic Workflows Unlike YAML pipelines, these workflows: Are written in natural language or simplified formats Use AI to interpret intent Adapt based on context 3. Event-Driven Intelligence Workflows trigger on events like: Pull request creation Issue updates Failed builds But instead of just reacting, they: Analyze the situation Decide the best course of action 4. Feedback Loops Continuous AI systems improve over time using: Past PR data Test failures Deployment outcomes CI/CD vs Continuous AI: A Deep Comparison Traditional CI/CD Pipeline Developer pushes code Pipeline runs tests Build is generated Code is deployed âĄď¸ Everything is predefined and static Continuous AI Workflow Developer creates PR AI agent reviews code Suggests improvements Generates missing tests Fixes minor issues automatically Learns from feedback âĄď¸ Dynamic, intelligent, and evolving Real-World Use Cases 1. Automated Pull Request Reviews AI agents can: Detect code smells Suggest optimizations Ensure coding standards 2. Self-Healing Repositories Automatically fix failing builds Update dependencies Resolve merge conflicts 3. Intelligent Test Generation Generate test cases based on code changes Improve coverage over time 4. Issue Triage Automation Categorize issues Assign priorities Route to correct teams 5. Documentation Automation Auto-generate README updates Keep documentation in sync with code Architecture of Continuous AI Systems A typical architecture includes: Layer 1: Event Sources GitHub events (PRs, commits, issues) Layer 2: AI Decision Engine LLM-based agents Context analysis Task planning Layer 3: Action Layer GitHub Actions Scripts Automation tools Layer 4: Feedback Loop Logs Metrics Model improvement Multi-Agent Systems: The Next Level Continuous AI becomes more powerful when multiple agents collaborate. Example Setup: Code Review Agent â Reviews PRs Test Agent â Generates tests Security Agent â Scans vulnerabilities Docs Agent â Updates documentation These agents: Communicate with each other Share context Coordinate tasks âĄď¸ This creates a virtual AI engineering team Benefits for Engineering Teams 1. Increased Productivity Developers spend less time on repetitive tasks. 2. Better Code Quality Continuous improvements ensure cleaner codebases. 3. Faster Time-to-Market Automation reduces bottlenecks. 4. Reduced Burnout Engineers focus on innovation instead of maintenance. Challenges and Risks 1. Over-Automation Too much automation can reduce human oversight. 2. Security Concerns AI workflows may misuse permissions if not controlled. 3. Trust Issues Teams may hesitate to rely on AI decisions. 4. Cost of AI Operations Running AI agents continuously can increase costs. Best Practices for Implementing Continuous AI 1. Start Small Begin with: PR review automation Test generation 2. Human-in-the-Loop Ensure: Critical decisions require approval 3. Use Least Privilege Restrict workflow permissions. 4. Monitor and Measure Track: Accuracy Impact Cost 5. Build Feedback Loops Continuously improve models and workflows. Future of GitHub Automation The future is heading toward: Fully autonomous repositories AI-driven engineering teams Continuous optimization of software systems We may soon see: Repos that refactor themselves Systems that predict failures before they occur AI architects designing system improvements Conclusion CI/CD transformed how we build and deliver software. But Continuous AI is set to transform how software evolves. It moves us from: âAutomating tasksâ â âAutomating intelligenceâ For engineering leaders, this is not just a technical shiftâitâs a strategic advantage. Early adopters of Continuous AI will build faster, smarter, and more resilient systems. The question is no longer: âShould we adopt AI in our workflows?â But: âHow fast can we transition to Continuous AI?âAZD for Beginners: A Practical Introduction to Azure Developer CLI
If you are learning how to get an application from your machine into Azure without stitching together every deployment step by hand, Azure Developer CLI, usually shortened to azd , is one of the most useful tools to understand early. It gives developers a workflow-focused command line for provisioning infrastructure, deploying application code, wiring environment settings, and working with templates that reflect real cloud architectures rather than toy examples. This matters because many beginners hit the same wall when they first approach Azure. They can build a web app locally, but once deployment enters the picture they have to think about resource groups, hosting plans, databases, secrets, monitoring, configuration, and repeatability all at once. azd reduces that operational overhead by giving you a consistent developer workflow. Instead of manually creating each resource and then trying to remember how everything fits together, you start with a template or an azd -compatible project and let the tool guide the path from local development to a running Azure environment. If you are new to the tool, the AZD for Beginners learning resources are a strong place to start. The repository is structured as a guided course rather than a loose collection of notes. It covers the foundations, AI-first deployment scenarios, configuration and authentication, infrastructure as code, troubleshooting, and production patterns. In other words, it does not just tell you which commands exist. It shows you how to think about shipping modern Azure applications with them. What Is Azure Developer CLI? The Azure Developer CLI documentation on Microsoft Learn, azd is an open-source tool designed to accelerate the path from a local development environment to Azure. That description is important because it explains what the tool is trying to optimise. azd is not mainly about managing one isolated Azure resource at a time. It is about helping developers work with complete applications. The simplest way to think about it is this. Azure CLI, az , is broad and resource-focused. It gives you precise control over Azure services. Azure Developer CLI, azd , is application-focused. It helps you take a solution made up of code, infrastructure definitions, and environment configuration and push that solution into Azure in a repeatable way. Those tools are not competitors. They solve different problems and often work well together. For a beginner, the value of azd comes from four practical benefits: It gives you a consistent workflow built around commands such as azd init , azd auth login , azd up , azd show , and azd down . It uses templates so you do not need to design every deployment structure from scratch on day one. It encourages infrastructure as code through files such as azure.yaml and the infra folder. It helps you move from a one-off deployment towards a repeatable development workflow that is easier to understand, change, and clean up. Why Should You Care About azd A lot of cloud frustration comes from context switching. You start by trying to deploy an app, but you quickly end up learning five or six Azure services, authentication flows, naming rules, environment variables, and deployment conventions all at once. That is not a good way to build confidence. azd helps by giving a workflow that feels closer to software delivery than raw infrastructure management. You still learn real Azure concepts, but you do so through an application lens. You initialise a project, authenticate, provision what is required, deploy the app, inspect the result, and tear it down when you are done. That sequence is easier to retain because it mirrors the way developers already think about shipping software. This is also why the AZD for Beginners resource is useful. It does not assume every reader is already comfortable with Azure. It starts with foundation topics and then expands into more advanced paths, including AI deployment scenarios that use the same core azd workflow. That progression makes it especially suitable for students, self-taught developers, workshop attendees, and engineers who know how to code but want a clearer path into Azure deployment. What You Learn from AZD for Beginners The AZD for Beginners course is structured as a learning journey rather than a single quickstart. That matters because azd is not just a command list. It is a deployment workflow with conventions, patterns, and trade-offs. The course helps readers build that mental model gradually. At a high level, the material covers: Foundational topics such as what azd is, how to install it, and how the basic deployment loop works. Template-based development, including how to start from an existing architecture rather than building everything yourself. Environment configuration and authentication practices, including the role of environment variables and secure access patterns. Infrastructure as code concepts using the standard azd project structure. Troubleshooting, validation, and pre-deployment thinking, which are often ignored in beginner content even though they matter in real projects. Modern AI and multi-service application scenarios, showing that azd is not limited to basic web applications. One of the strongest aspects of the course is that it does not stop at the first successful deployment. It also covers how to reason about configuration, resource planning, debugging, and production readiness. That gives learners a more realistic picture of what Azure development work actually looks like. The Core azd Workflow The official overview on Microsoft Learn and the get started guide both reinforce a simple but important idea: most beginners should first understand the standard workflow before worrying about advanced customisation. That workflow usually looks like this: Install azd . Authenticate with Azure. Initialise a project from a template or in an existing repository. Run azd up to provision and deploy. Inspect the deployed application. Remove the resources when finished. Here is a minimal example using an existing template: # Install azd on Windows winget install microsoft.azd # Check that the installation worked azd version # Sign in to your Azure account azd auth login # Start a project from a template azd init --template todo-nodejs-mongo # Provision Azure resources and deploy the app azd up # Show output values such as the deployed URL azd show # Clean up everything when you are done learning azd down --force --purge This sequence is important because it teaches beginners the full lifecycle, not only deployment. A lot of people remember azd up and forget the cleanup step. That leads to wasted resources and avoidable cost. The azd down --force --purge step is part of the discipline, not an optional extra. Installing azd and Verifying Your Setup The official install azd guide on Microsoft Learn provides platform-specific instructions. Because this repository targets developer learning, it is worth showing the common install paths clearly. # Windows winget install microsoft.azd # macOS brew tap azure/azd && brew install azd # Linux curl -fsSL https://aka.ms/install-azd.sh | bash After installation, verify the tool is available: azd version That sounds obvious, but it is worth doing immediately. Many beginner problems come from assuming the install completed correctly, only to discover a path issue or outdated version later. Verifying early saves time. The Microsoft Learn installation page also notes that azd installs supporting tools such as GitHub CLI and Bicep CLI within the tool's own scope. For a beginner, that is helpful because it removes some of the setup friction you might otherwise need to handle manually. What Happens When You Run azd up ? One of the most important questions is what azd up is actually doing. The short answer is that it combines provisioning and deployment into one workflow. The longer answer is where the learning value sits. When you run azd up , the tool looks at the project configuration, reads the infrastructure definition, determines which Azure resources need to exist, provisions them if necessary, and then deploys the application code to those resources. In many templates, it also works with environment settings and output values so that the project becomes reproducible rather than ad hoc. That matters because it teaches a more modern cloud habit. Instead of building infrastructure manually in the portal and then hoping you can remember how you did it, you define the deployment shape in source-controlled files. Even at beginner level, that is the right habit to learn. Understanding the Shape of an azd Project The Azure Developer CLI templates overview explains the standard project structure used by azd . If you understand this structure early, templates become much less mysterious. A typical azd project contains: azure.yaml to describe the project and map services to infrastructure targets. An infra folder containing Bicep or Terraform files for infrastructure as code. A src folder, or equivalent source folders, containing the application code that will be deployed. A local .azure folder to store environment-specific settings for the project. Here is a minimal example of what an azure.yaml file can look like in a simple app: name: beginner-web-app metadata: template: beginner-web-app services: web: project: ./src/web host: appservice This file is small, but it carries an important idea. azd needs a clear mapping between your application code and the Azure service that will host it. Once you see that, the tool becomes easier to reason about. You are not invoking magic. You are describing an application and its hosting model in a standard way. Start from a Template, Then Learn the Architecture Beginners often assume that using a template is somehow less serious than building something from scratch. In practice, it is usually the right place to begin. The official docs for templates and the Awesome AZD gallery both encourage developers to start from an existing architecture when it matches their goals. That is a sound learning strategy for two reasons. First, it lets you experience a working deployment quickly, which builds confidence. Second, it gives you a concrete project to inspect. You can look at azure.yaml , explore the infra folder, inspect the app source, and understand how the pieces connect. That teaches more than reading a command reference in isolation. The AZD for Beginners material also leans into this approach. It includes chapter guidance, templates, workshops, examples, and structured progression so that readers move from successful execution into understanding. That is much more useful than a single command demo. A practical beginner workflow looks like this: # Pick a known template azd init --template todo-nodejs-mongo # Review the files that were created or cloned # - azure.yaml # - infra/ # - src/ # Deploy it azd up # Open the deployed app details azd show Once that works, do not immediately jump to a different template. Spend time understanding what was deployed and why. Where AZD for Beginners Fits In The official docs are excellent for accurate command guidance and conceptual documentation. The AZD for Beginners repository adds something different: a curated learning path. It helps beginners answer questions such as these: Which chapter should I start with if I know Azure a little but not azd ? How do I move from a first deployment into understanding configuration and authentication? What changes when the application becomes an AI application rather than a simple web app? How do I troubleshoot failures instead of copying commands blindly? The repository also points learners towards workshops, examples, a command cheat sheet, FAQ material, and chapter-based exercises. That makes it particularly useful in teaching contexts. A lecturer or workshop facilitator can use it as a course backbone, while an individual learner can work through it as a self-study track. For developers interested in AI, the resource is especially timely because it shows how the same azd workflow can be used for AI-first solutions, including scenarios connected to Microsoft Foundry services and multi-agent architectures. The important beginner lesson is that the workflow stays recognisable even as the application becomes more advanced. Common Beginner Mistakes and How to Avoid Them A good introduction should not only explain the happy path. It should also point out the places where beginners usually get stuck. Skipping authentication checks. If azd auth login has not completed properly, later commands will fail in ways that are harder to interpret. Not verifying the installation. Run azd version immediately after install so you know the tool is available. Treating templates as black boxes. Always inspect azure.yaml and the infra folder so you understand what the project intends to provision. Forgetting cleanup. Learning environments cost money if you leave them running. Use azd down --force --purge when you are finished experimenting. Trying to customise too early. First get a known template working exactly as designed. Then change one thing at a time. If you do hit problems, the official troubleshooting documentation and the troubleshooting sections inside AZD for Beginners are the right next step. That is a much better habit than searching randomly for partial command snippets. How I Would Approach AZD as a New Learner If I were introducing azd to a student or a developer who is comfortable with code but new to Azure delivery, I would keep the learning path tight. Read the official What is Azure Developer CLI? overview so the purpose is clear. Install the tool using the Microsoft Learn install guide. Work through the opening sections of AZD for Beginners. Deploy one template with azd init and azd up . Inspect azure.yaml and the infrastructure files before making any changes. Run azd down --force --purge so the lifecycle becomes a habit. Only then move on to AI templates, configuration changes, or custom project conversion. That sequence keeps the cognitive load manageable. It gives you one successful deployment, one architecture to inspect, and one repeatable workflow to internalise before adding more complexity. Why azd Is Worth Learning Now azd matters because it reflects how modern Azure application delivery is actually done: repeatable infrastructure, source-controlled configuration, environment-aware workflows, and application-level thinking rather than isolated portal clicks. It is useful for straightforward web applications, but it becomes even more valuable as systems gain more services, more configuration, and more deployment complexity. That is also why the AZD for Beginners resource is worth recommending. It gives new learners a structured route into the tool instead of leaving them to piece together disconnected docs, samples, and videos on their own. Used alongside the official Microsoft Learn documentation, it gives you both accuracy and progression. Key Takeaways azd is an application-focused Azure deployment tool, not just another general-purpose CLI. The core beginner workflow is simple: install, authenticate, initialise, deploy, inspect, and clean up. Templates are not a shortcut to avoid learning. They are a practical way to learn architecture through working examples. AZD for Beginners is valuable because it turns the tool into a structured learning path. The official Microsoft Learn documentation for Azure Developer CLI should remain your grounding source for commands and platform guidance. Next Steps If you want to keep going, start with these resources: AZD for Beginners for the structured course, examples, and workshop materials. Azure Developer CLI documentation on Microsoft Learn for official command, workflow, and reference guidance. Install azd if you have not set up the tool yet. Deploy an azd template for the first full quickstart. Azure Developer CLI templates overview if you want to understand the project structure and template model. Awesome AZD if you want to browse starter architectures. If you are teaching others, this is also a good sequence for a workshop: start with the official overview, deploy one template, inspect the project structure, and then use AZD for Beginners as the path for deeper learning. That gives learners both an early win and a solid conceptual foundation.Agents League: Meet the Winners
Agents League brought together developers from around the world to build AI agents using Microsoft's developer tools. With 100+ submissions across three tracks, choosing winners was genuinely difficult. Today, we're proud to announce the category champions. đ¨ Creative Apps Winner: CodeSonify View project CodeSonify turns source code into music. As a genuinely thoughtful system, its functions become ascending melodies, loops create rhythmic patterns, conditionals trigger chord changes, and bugs produce dissonant sounds. It supports 7 programming languages and 5 musical styles, with each language mapped to its own key signature and code complexity directly driving the tempo. What makes CodeSonify stand out is the depth of execution. CodeSonify team delivered three integrated experiences: a web app with real-time visualization and one-click MIDI export, an MCP server exposing 5 tools inside GitHub Copilot in VS Code Agent Mode, and a diff sonification engine that lets you hear a code review. A clean refactor sounds harmonious. A messy one sounds chaotic. The team even built the MIDI generator from scratch in pure TypeScript with zero external dependencies. Built entirely with GitHub Copilot assistance, this is one of those projects that makes you think about code differently. đ§ Reasoning Agents Winner: CertPrep Multi-Agent System View project CertPrep Multi-Agent System team built a production-grade 8-agent system for personalized Microsoft certification exam preparation, supporting 9 exam families including AI-102, AZ-204, AZ-305, and more. Each agent has a distinct responsibility: profiling the learner, generating a week-by-week study schedule, curating learning paths, tracking readiness, running mock assessments, and issuing a GO / CONDITIONAL GO / NOT YET booking recommendation. The engineering behind the scene here is impressive. A 3-tier LLM fallback chain ensures the system runs reliably even without Azure credentials, with the full pipeline completing in under 1 second in mock mode. A 17-rule guardrail pipeline validates every agent boundary. Study time allocation uses the Largest Remainder algorithm to guarantee no domain is silently zeroed out. 342 automated tests back it all up. This is what thoughtful multi-agent architecture looks like in practice. đź Enterprise Agents Winner: Whatever AI Assistant (WAIA) View project WAIA is a production-ready multi-agent system for Microsoft 365 Copilot Chat and Microsoft Teams. A workflow agent routes queries to specialized HR, IT, or Fallback agents, transparently to the user, handling both RAG-pattern Q&A and action automation â including IT ticket submission via a SharePoint list. Technically, it's a showcase of what serious enterprise agent development looks like: a custom MCP server secured with OAuth Identity Passthrough, streaming responses via the OpenAI Responses API, Adaptive Cards for human-in-the-loop approval flows, a debug mode accessible directly from Teams or Copilot, and full OpenTelemetry integration visible in the Foundry portal. Franck also shipped end-to-end automated Bicep deployment so the solution can land in any Azure environment. It's polished, thoroughly documented, and built to be replicated. Thank you To every developer who submitted and shipped projects during Agents League: thank you đ Your creativity and innovation brought Agents League to life! đ Browse all submissions on GitHubBuilding a Smart Building HVAC Digital Twin with AI Copilot Using Foundry Local
Introduction Building operations teams face a constant challenge: optimizing HVAC systems for energy efficiency while maintaining occupant comfort and air quality. Traditional building management systems display raw sensor data, temperatures, pressures, COâ levelsâbut translating this into actionable insights requires deep HVAC expertise. What if operators could simply ask "Why is the third floor so warm?" and get an intelligent answer grounded in real building state? This article demonstrates building a sample smart building digital twin with an AI-powered operations copilot, implemented using DigitalTwin, React, Three.js, and Microsoft Foundry Local. You'll learn how to architect physics-based simulators that model thermal dynamics, implement 3D visualizations of building systems, integrate natural language AI control, and design fault injection systems for testing and training. Whether you're building IoT platforms for commercial real estate, designing energy management systems, or implementing predictive maintenance for building automation, this sample provides proven patterns for intelligent facility operations. Why Digital Twins Matter for Building Operations Physical buildings generate enormous operational data but lack intelligent interpretation layers. A 50,000 square foot office building might have 500+ sensors streaming metrics every minute, zone temperatures, humidity levels, equipment runtimes, energy consumption. Traditional BMS (Building Management Systems) visualize this data as charts and gauges, but operators must manually correlate patterns, diagnose issues, and predict failures. Digital twins solve this through physics-based simulation coupled with AI interpretation. Instead of just displaying current temperature readings, a digital twin models thermal dynamics, heat transfer rates, HVAC response characteristics, occupancy impacts. When conditions deviate from expectations, the twin compares observed versus predicted states, identifying root causes. Layer AI on top, and operators get natural language explanations: "The conference room is 3 degrees too warm because the VAV damper is stuck at 40% open, reducing airflow by 60%." This application focuses on HVAC, the largest building energy consumer, typically 40-50% of total usage. Optimizing HVAC by just 10% through better controls can save thousands of dollars monthly while improving occupant satisfaction. The digital twin enables "what-if" scenarios before making changes: "What happens to energy consumption and comfort if we raise the cooling setpoint by 2 degrees during peak demand response events?" Architecture: Three-Tier Digital Twin System The application implements a clean three-tier architecture separating visualization, simulation, and state management: The frontend uses React with Three.js for 3D visualization. Users see an interactive 3D model of the three-floor building with color-coded zones indicating temperature and COâ levels. Click any equipment, AHUs, VAVs, chillers, to see detailed telemetry. The control panel enables adjusting setpoints, running simulation steps, and activating demand response scenarios. Real-time charts display KPIs: energy consumption, comfort compliance, air quality levels. The backend Node.js/Express server orchestrates simulation and state management. It maintains the digital twin state as JSON, the single source of truth for all equipment, zones, and telemetry. REST API endpoints handle control requests, simulation steps, and AI copilot queries. WebSocket connections push real-time updates to the frontend for live monitoring. The HVAC simulator implements physics-based models: 1R1C thermal models for zones, affinity laws for fan power, chiller COP calculations, COâ mass balance equations. Foundry Local provides AI copilot capabilities. The backend uses foundry-local-sdk to query locally running models. Natural language queries ("How's the lobby temperature?") get answered with building state context. The copilot can explain anomalies, suggest optimizations, and even execute commands when explicitly requested. Implementing Physics-Based HVAC Simulation Accurate simulation requires modeling actual HVAC physics. The simulator implements several established building energy models: // backend/src/simulator/thermal-model.js class ZoneThermalModel { // 1R1C (one resistance, one capacitance) thermal model static calculateTemperatureChange(zone, delta_t_seconds) { const C_thermal = zone.volume * 1.2 * 1000; // Heat capacity (J/K) const R_thermal = zone.r_value * zone.envelope_area; // Thermal resistance // Internal heat gains (occupancy, equipment, lighting) const Q_internal = zone.occupancy * 100 + // 100W per person zone.equipment_load + zone.lighting_load; // Cooling/heating from HVAC const airflow_kg_s = zone.vav.airflow_cfm * 0.0004719; // CFM to kg/s const c_p_air = 1006; // Specific heat of air (J/kg¡K) const Q_hvac = airflow_kg_s * c_p_air * (zone.vav.supply_temp - zone.temperature); // Envelope losses const Q_envelope = (zone.outdoor_temp - zone.temperature) / R_thermal; // Net energy balance const Q_net = Q_internal + Q_hvac + Q_envelope; // Temperature change: Q = C * dT/dt const dT = (Q_net / C_thermal) * delta_t_seconds; return zone.temperature + dT; } } This model captures essential thermal dynamics while remaining computationally fast enough for real-time simulation. It accounts for internal heat generation from occupants and equipment, HVAC cooling/heating contributions, and heat loss through the building envelope. The COâ model uses mass balance equations: class AirQualityModel { static calculateCO2Change(zone, delta_t_seconds) { // COâ generation from occupants const G_co2 = zone.occupancy * 0.0052; // L/s per person at rest // Outdoor air ventilation rate const V_oa = zone.vav.outdoor_air_cfm * 0.000471947; // CFM to mÂł/s // COâ concentration difference (indoor - outdoor) const delta_CO2 = zone.co2_ppm - 400; // Outdoor ~400ppm // Mass balance: dC/dt = (G - V*ÎC) / Volume const dCO2_dt = (G_co2 - V_oa * delta_CO2) / zone.volume; return zone.co2_ppm + (dCO2_dt * delta_t_seconds); } } These models execute every simulation step, updating the entire building state: async function simulateStep(twin, timestep_minutes) { const delta_t = timestep_minutes * 60; // Convert to seconds // Update each zone for (const zone of twin.zones) { zone.temperature = ZoneThermalModel.calculateTemperatureChange(zone, delta_t); zone.co2_ppm = AirQualityModel.calculateCO2Change(zone, delta_t); } // Update equipment based on zone demands for (const vav of twin.vavs) { updateVAVOperation(vav, twin.zones); } for (const ahu of twin.ahus) { updateAHUOperation(ahu, twin.vavs); } updateChillerOperation(twin.chiller, twin.ahus); updateBoilerOperation(twin.boiler, twin.ahus); // Calculate system KPIs twin.kpis = calculateSystemKPIs(twin); // Detect alerts twin.alerts = detectAnomalies(twin); // Persist updated state await saveTwinState(twin); return twin; } 3D Visualization with React and Three.js The frontend renders an interactive 3D building view that updates in real-time as conditions change. Using React Three Fiber simplifies Three.js integration with React's component model: // frontend/src/components/BuildingView3D.jsx import { Canvas } from '@react-three/fiber'; import { OrbitControls } from '@react-three/drei'; export function BuildingView3D({ twinState }) { return ( {/* Render building floors */} {twinState.zones.map(zone => ( selectZone(zone.id)} /> ))} {/* Render equipment */} {twinState.ahus.map(ahu => ( ))} ); } function ZoneMesh({ zone, onClick }) { const color = getTemperatureColor(zone.temperature, zone.setpoint); return ( ); } function getTemperatureColor(current, setpoint) { const deviation = current - setpoint; if (Math.abs(deviation) < 1) return '#00ff00'; // Green: comfortable if (Math.abs(deviation) < 3) return '#ffff00'; // Yellow: acceptable return '#ff0000'; // Red: uncomfortable } This visualization immediately shows building state at a glance, operators see "hot spots" in red, comfortable zones in green, and can click any area for detailed metrics. Integrating AI Copilot for Natural Language Control The AI copilot transforms building data into conversational insights. Instead of navigating multiple screens, operators simply ask questions: // backend/src/routes/copilot.js import { FoundryLocalClient } from 'foundry-local-sdk'; const foundry = new FoundryLocalClient({ endpoint: process.env.FOUNDRY_LOCAL_ENDPOINT }); router.post('/api/copilot/chat', async (req, res) => { const { message } = req.body; // Load current building state const twin = await loadTwinState(); // Build context for AI const context = buildBuildingContext(twin); const completion = await foundry.chat.completions.create({ model: 'phi-4', messages: [ { role: 'system', content: `You are an HVAC operations assistant for a 3-floor office building. Current Building State: ${context} Answer questions about equipment status, comfort conditions, and energy usage. Provide specific, actionable information based on the current data. Do not speculate beyond provided information.` }, { role: 'user', content: message } ], temperature: 0.3, max_tokens: 300 }); res.json({ response: completion.choices[0].message.content, model: 'phi-4', timestamp: new Date().toISOString() }); }); function buildBuildingContext(twin) { const alerts = twin.alerts.filter(a => a.active); const zoneStatus = twin.zones.map(z => `${z.name}: ${z.temperature.toFixed(1)}°F (setpoint: ${z.setpoint}°F), COâ: ${z.co2_ppm}ppm, ${z.occupancy} people` ); return ` Zones: ${zoneStatus.join('\n')} Active Alerts: ${alerts.length} ${alerts.map(a => `- ${a.severity}: ${a.message}`).join('\n')} Energy: ${twin.kpis.total_power_kw.toFixed(1)} kW Chiller: ${twin.chiller.percent_load}% load, ${twin.chiller.cop.toFixed(2)} COP `.trim(); } Example interactions: User: "Why is the third floor conference room warm?" AI: "The Executive Suite on Floor 3 is currently 75.2°F, 3 degrees above the 72°F setpoint. This is due to high occupancy (8 people) and VAV-F3-EXEC operating at reduced airflow (285 CFM vs 450 CFM capacity). AHU-2 is supplying adequate cool air at 55°F. Recommend checking VAV damper positionâit may be partially stuck." User: "What's our current energy usage?" AI: "Total building power consumption is 47.3 kW. Breakdown: Chiller (24.1 kW, 51%), AHU fans (12.8 kW, 27%), Pumps (7.2 kW, 15%), Other (3.2 kW, 7%). Chiller is operating at 62% load with a COP of 4.8, which is efficient for current conditions." User: "Activate demand response level 2" AI: "I cannot directly control building systems for safety reasons. To activate DR Level 2, use the Control Panel and click 'Demand Response' â 'Level 2'. This will raise cooling setpoints by 3°F and reduce auxiliary loads, targeting 15% energy reduction." The AI provides grounded, specific answers citing actual equipment IDs and metrics. It refuses to directly execute control commands, instead guiding operators to explicit control interfaces, a critical safety pattern for building systems. Fault Injection for Testing and Training Real building operations experience equipment failures, stuck dampers, sensor drift, communication losses. The digital twin includes comprehensive fault injection capabilities to train operators and test control logic: // backend/src/simulator/fault-injector.js const FAULT_CATALOG = { chillerFailure: { description: 'Chiller compressor failure', apply: (twin) => { twin.chiller.status = 'FAULT'; twin.chiller.cooling_output = 0; twin.alerts.push({ id: 'chiller-fault', severity: 'CRITICAL', message: 'Chiller compressor failure - no cooling available', equipment: 'CHILLER-01' }); } }, stuckVAVDamper: { description: 'VAV damper stuck at current position', apply: (twin, vavId) => { const vav = twin.vavs.find(v => v.id === vavId); vav.damper_stuck = true; vav.damper_position_fixed = vav.damper_position; twin.alerts.push({ id: `vav-stuck-${vavId}`, severity: 'HIGH', message: `VAV ${vavId} damper stuck at ${vav.damper_position}%`, equipment: vavId }); } }, sensorDrift: { description: 'Temperature sensor reading 5°F high', apply: (twin, zoneId) => { const zone = twin.zones.find(z => z.id === zoneId); zone.sensor_drift = 5.0; zone.temperature_measured = zone.temperature_actual + 5.0; } }, communicationLoss: { description: 'Equipment communication timeout', apply: (twin, equipmentId) => { const equipment = findEquipmentById(twin, equipmentId); equipment.comm_status = 'OFFLINE'; equipment.stale_data = true; twin.alerts.push({ id: `comm-loss-${equipmentId}`, severity: 'MEDIUM', message: `Lost communication with ${equipmentId}`, equipment: equipmentId }); } } }; router.post('/api/twin/fault', async (req, res) => { const { faultType, targetEquipment } = req.body; const twin = await loadTwinState(); const fault = FAULT_CATALOG[faultType]; if (!fault) { return res.status(400).json({ error: 'Unknown fault type' }); } fault.apply(twin, targetEquipment); await saveTwinState(twin); res.json({ message: `Applied fault: ${fault.description}`, affectedEquipment: targetEquipment, timestamp: new Date().toISOString() }); }); Operators can inject faults to practice diagnosis and response. Training scenarios might include: "The chiller just failed during a heat wave, how do you maintain comfort?" or "Multiple VAV dampers are stuck, which zones need immediate attention?" Key Takeaways and Production Deployment Building a physics-based digital twin with AI capabilities requires balancing simulation accuracy with computational performance, providing intuitive visualization while maintaining technical depth, and enabling AI assistance without compromising safety. Key architectural lessons: Physics models enable prediction: Comparing predicted vs observed behavior identifies anomalies that simple thresholds miss 3D visualization improves spatial understanding: Operators immediately see which floors or zones need attention AI copilots accelerate diagnosis: Natural language queries get answers in seconds vs. minutes of manual data examination Fault injection validates readiness: Testing failure scenarios prepares operators for real incidents JSON state enables integration: Simple file-based state makes connecting to real BMS systems straightforward For production deployment, connect the twin to actual building systems via BACnet, Modbus, or MQTT integrations. Replace simulated telemetry with real sensor streams. Calibrate model parameters against historical building performance. Implement continuous learning where the twin's predictions improve as it observes actual building behavior. The complete implementation with simulation engine, 3D visualization, AI copilot, and fault injection system is available at github.com/leestott/DigitalTwin. Clone the repository and run the startup scripts to explore the digital twin, no building hardware required. Resources and Further Reading Smart Building HVAC Digital Twin Repository - Complete source code and simulation engine Setup and Quick Start Guide - Installation instructions and usage examples Microsoft Foundry Local Documentation - AI integration reference HVAC Simulation Documentation - Physics model details and calibration Three.js Documentation - 3D visualization framework ASHRAE Standards - Building energy modeling standardsBuilding Your First Local RAG Application with Foundry Local
A developer's guide to building an offline, mobile-responsive AI support agent using Retrieval-Augmented Generation, the Foundry Local SDK, and JavaScript. Imagine you are a gas field engineer standing beside a pipeline in a remote location. There is no Wi-Fi, no mobile signal, and you need a safety procedure right now. What do you do? This is the exact problem that inspired this project: a fully offline RAG-powered support agent that runs entirely on your machine. No cloud. No API keys. No outbound network calls. Just a local language model, a local vector store, and your own documents, all accessible from a browser on any device. In this post, you will learn how it works, how to build your own, and the key architectural decisions behind it. If you have ever wanted to build an AI application that runs locally and answers questions grounded in your own data, this is the place to start. The finished application: a browser-based AI support agent that runs entirely on your machine. What Is Retrieval-Augmented Generation? Retrieval-Augmented Generation (RAG) is a pattern that makes AI models genuinely useful for domain-specific tasks. Rather than hoping the model "knows" the answer from its training data, you: Retrieve relevant chunks from your own documents using a vector store Augment the model's prompt with those chunks as context Generate a response grounded in your actual data The result is fewer hallucinations, traceable answers with source attribution, and an AI that works with your content rather than relying on general knowledge. If you are building internal tools, customer support bots, field manuals, or knowledge bases, RAG is the pattern you want. RAG vs CAG: Understanding the Trade-offs If you have explored AI application patterns before, you have likely encountered Context-Augmented Generation (CAG). Both RAG and CAG solve the same core problem: grounding an AI model's answers in your own content. They take different approaches, and each has genuine strengths and limitations. RAG (Retrieval-Augmented Generation) How it works: Documents are split into chunks, vectorised, and stored in a database. At query time, the most relevant chunks are retrieved and injected into the prompt. Strengths: Scales to thousands or millions of documents Fine-grained retrieval at chunk level with source attribution Documents can be added or updated dynamically without restarting Token-efficient: only relevant chunks are sent to the model Supports runtime document upload via the web UI Limitations: More complex architecture: requires a vector store and chunking strategy Retrieval quality depends on chunking parameters and scoring method May miss relevant content if the retrieval step does not surface it CAG (Context-Augmented Generation) How it works: All documents are loaded at startup. The most relevant ones are selected per query using keyword scoring and injected into the prompt. Strengths: Drastically simpler architecture with no vector database or embeddings All information is always available to the model Minimal dependencies and easy to set up Near-instant document selection Limitations: Constrained by the model's context window size Best suited to small, curated document sets (tens of documents) Adding documents requires an application restart Want to compare these patterns hands-on? There is a CAG-based implementation of the same gas field scenario using whole-document context injection. Clone both repositories, run them side by side, and see how the architectures differ in practice. When Should You Choose Which? Consideration Choose RAG Choose CAG Document count Hundreds or thousands Tens of documents Document updates Frequent or dynamic (runtime upload) Infrequent (restart to reload) Source attribution Per-chunk with relevance scores Per-document Setup complexity Moderate (ingestion step required) Minimal Query precision Better for large or diverse collections Good for keyword-matchable content Infrastructure SQLite vector store (single file) None beyond the runtime For the sample application in this post (20 gas engineering procedure documents with runtime upload), RAG is the clear winner. If your document set is small and static, CAG may be simpler. Both patterns run fully offline using Foundry Local. Foundry Local: Your On-Device AI Runtime Foundry Local is a lightweight runtime from Microsoft that downloads, manages, and serves language models entirely on your device. No cloud account, no API keys, no outbound network calls (after the initial model download). What makes it particularly useful for developers: No GPU required: runs on CPU or NPU, making it accessible on standard laptops and desktops Native SDK bindings: in-process inference via the foundry-local-sdk npm package, with no HTTP round-trips to a local server Automatic model management: downloads, caches, and loads models automatically Hardware-optimised variant selection: the SDK picks the best variant for your hardware (GPU, NPU, or CPU) Real-time progress callbacks: ideal for building loading UIs that show download and initialisation progress The integration code is refreshingly minimal: import { FoundryLocalManager } from "foundry-local-sdk"; // Create a manager and discover models via the catalogue const manager = FoundryLocalManager.create({ appName: "gas-field-local-rag" }); const model = await manager.catalog.getModel("phi-3.5-mini"); // Download if not cached, then load into memory if (!model.isCached) { await model.download((progress) => { console.log(`Download: ${Math.round(progress * 100)}%`); }); } await model.load(); // Create a chat client for direct in-process inference const chatClient = model.createChatClient(); const response = await chatClient.completeChat([ { role: "system", content: "You are a helpful assistant." }, { role: "user", content: "How do I detect a gas leak?" } ]); That is it. No server configuration, no authentication tokens, no cloud provisioning. The model runs in the same process as your application. The Technology Stack The sample application is deliberately simple. No frameworks, no build steps, no Docker: Layer Technology Purpose AI Model Foundry Local + Phi-3.5 Mini Runs locally via native SDK bindings, no GPU required Back end Node.js + Express Lightweight HTTP server, everyone knows it Vector Store SQLite (via better-sqlite3 ) Zero infrastructure, single file on disc Retrieval TF-IDF + cosine similarity No embedding model required, fully offline Front end Single HTML file with inline CSS No build step, mobile-responsive, field-ready The total dependency footprint is three npm packages: express , foundry-local-sdk , and better-sqlite3 . Architecture Overview The five-layer architecture, all running on a single machine. The system has five layers, all running on a single machine: Client layer: a single HTML file served by Express, with quick-action buttons and a responsive chat interface Server layer: Express.js starts immediately and serves the UI plus SSE status and chat endpoints RAG pipeline: the chat engine orchestrates retrieval and generation; the chunker handles TF-IDF vectorisation; the prompts module provides safety-first system instructions Data layer: SQLite stores document chunks and their TF-IDF vectors; documents live as .md files in the docs/ folder AI layer: Foundry Local runs Phi-3.5 Mini on CPU or NPU via native SDK bindings Building the Solution Step by Step Prerequisites You need two things installed on your machine: Node.js 20 or later: download from nodejs.org Foundry Local: Microsoft's on-device AI runtime: winget install Microsoft.FoundryLocal The SDK will automatically download the Phi-3.5 Mini model (approximately 2 GB) the first time you run the application. Getting the Code Running # Clone the repository git clone https://github.com/leestott/local-rag.git cd local-rag # Install dependencies npm install # Ingest the 20 gas engineering documents into the vector store npm run ingest # Start the server npm start Open http://127.0.0.1:3000 in your browser. You will see the status indicator whilst the model loads. Once the model is ready, the status changes to "Offline Ready" and you can start chatting. Desktop view Mobile view How the RAG Pipeline Works Let us trace what happens when a user asks: "How do I detect a gas leak?" The query flow from browser to model and back. 1 Documents are ingested and indexed When you run npm run ingest , every .md file in the docs/ folder is read, parsed (with optional YAML front-matter for title, category, and ID), split into overlapping chunks of approximately 200 tokens, and stored in SQLite with TF-IDF vectors. 2 Model is loaded via the SDK The Foundry Local SDK discovers the model in the local catalogue and loads it into memory. If the model is not already cached, it downloads it first (with progress streamed to the browser via SSE). 3 User sends a question The question arrives at the Express server. The chat engine converts it into a TF-IDF vector, uses an inverted index to find candidate chunks, and scores them using cosine similarity. The top 3 chunks are returned in under 1 ms. 4 Prompt is constructed The engine builds a messages array containing: the system prompt (with safety-first instructions), the retrieved chunks as context, the conversation history, and the user's question. 5 Model generates a grounded response The prompt is sent to the locally loaded model via the Foundry Local SDK's native chat client. The response streams back token by token through Server-Sent Events to the browser. Source references with relevance scores are included. A response with safety warnings and step-by-step guidance The sources panel shows which chunks were used and their relevance Key Code Walkthrough The Vector Store (TF-IDF + SQLite) The vector store uses SQLite to persist document chunks alongside their TF-IDF vectors. At query time, an inverted index finds candidate chunks that share terms with the query, then cosine similarity ranks them: // src/vectorStore.js search(query, topK = 5) { const queryTf = termFrequency(query); this._ensureCache(); // Build in-memory cache on first access // Use inverted index to find candidates sharing at least one term const candidateIndices = new Set(); for (const term of queryTf.keys()) { const indices = this._invertedIndex.get(term); if (indices) { for (const idx of indices) candidateIndices.add(idx); } } // Score only candidates, not all rows const scored = []; for (const idx of candidateIndices) { const row = this._rowCache[idx]; const score = cosineSimilarity(queryTf, row.tf); if (score > 0) scored.push({ ...row, score }); } scored.sort((a, b) => b.score - a.score); return scored.slice(0, topK); } The inverted index, in-memory row cache, and prepared SQL statements bring retrieval time to sub-millisecond for typical query loads. Why TF-IDF Instead of Embeddings? Most RAG tutorials use embedding models for retrieval. This project uses TF-IDF because: Fully offline: no embedding model to download or run Zero latency: vectorisation is instantaneous (it is just maths on word frequencies) Good enough: for 20 domain-specific documents, TF-IDF retrieves the right chunks reliably Transparent: you can inspect the vocabulary and weights, unlike neural embeddings For larger collections or when semantic similarity matters more than keyword overlap, you would swap in an embedding model. For this use case, TF-IDF keeps the stack simple and dependency-free. The System Prompt For safety-critical domains, the system prompt is engineered to prioritise safety, prevent hallucination, and enforce structured responses: // src/prompts.js export const SYSTEM_PROMPT = `You are a local, offline support agent for gas field inspection and maintenance engineers. Behaviour Rules: - Always prioritise safety. If a procedure involves risk, explicitly call it out. - Do not hallucinate procedures, measurements, or tolerances. - If the answer is not in the provided context, say: "This information is not available in the local knowledge base." Response Format: - Summary (1-2 lines) - Safety Warnings (if applicable) - Step-by-step Guidance - Reference (document name + section)`; This pattern is transferable to any safety-critical domain: medical devices, electrical work, aviation maintenance, or chemical handling. Runtime Document Upload Unlike the CAG approach, RAG supports adding documents without restarting the server. Click the upload button to add new .md or .txt files. They are chunked, vectorised, and indexed immediately. The upload modal with the complete list of indexed documents. Adapting This for Your Own Domain The sample project is designed to be forked and adapted. Here is how to make it yours in four steps: 1. Replace the documents Delete the gas engineering documents in docs/ and add your own markdown files. The ingestion pipeline handles any markdown content with optional YAML front-matter: --- title: Troubleshooting Widget Errors category: Support id: KB-001 --- # Troubleshooting Widget Errors ...your content here... 2. Edit the system prompt Open src/prompts.js and rewrite the system prompt for your domain. Keep the structure (summary, safety, steps, reference) and update the language to match your users' expectations. 3. Tune the retrieval In src/config.js : chunkSize: 200 : smaller chunks give more precise retrieval, less context per chunk chunkOverlap: 25 : prevents information falling between chunks topK: 3 : how many chunks to retrieve per query (more gives more context but slower generation) 4. Swap the model Change config.model in src/config.js to any model available in the Foundry Local catalogue. Smaller models give faster responses on constrained devices; larger models give better quality. Building a Field-Ready UI The front end is a single HTML file with inline CSS. No React, no build tooling, no bundler. This keeps the project accessible to beginners and easy to deploy. Design decisions that matter for field use: Dark, high-contrast theme with 18px base font size for readability in bright sunlight Large touch targets (minimum 44px) for operation with gloves or PPE Quick-action buttons that wrap on mobile so all options are visible without scrolling Responsive layout that works from 320px to 1920px+ screen widths Streaming responses via SSE, so the user sees tokens arriving in real time The mobile chat experience, optimised for field use. Testing The project includes unit tests using the built-in Node.js test runner, with no extra test framework needed: # Run all tests npm test Tests cover the chunker, vector store, configuration, and server endpoints. Use them as a starting point when you adapt the project for your own domain. Ideas for Extending the Project Once you have the basics running, there are plenty of directions to explore: Embedding-based retrieval: use a local embedding model for better semantic matching on diverse queries Conversation memory: persist chat history across sessions using local storage or a lightweight database Multi-modal support: add image-based queries (photographing a fault code, for example) PWA packaging: make it installable as a standalone offline application on mobile devices Hybrid retrieval: combine TF-IDF keyword search with semantic embeddings for best results Try the CAG approach: compare with the local-cag sample to see which pattern suits your use case Ready to Build Your Own? Clone the RAG sample, swap in your own documents, and have an offline AI agent running in minutes. Or compare it with the CAG approach to see which pattern suits your use case best. Get the RAG Sample Get the CAG Sample Summary Building a local RAG application does not require a PhD in machine learning or a cloud budget. With Foundry Local, Node.js, and SQLite, you can create a fully offline, mobile-responsive AI agent that answers questions grounded in your own documents. The key takeaways: RAG is ideal for scalable, dynamic document sets where you need fine-grained retrieval with source attribution. Documents can be added at runtime without restarting. CAG is simpler when you have a small, stable set of documents that fit in the context window. See the local-cag sample to compare. Foundry Local makes on-device AI accessible: native SDK bindings, in-process inference, automatic model selection, and no GPU required. TF-IDF + SQLite is a viable vector store for small-to-medium collections, with sub-millisecond retrieval thanks to inverted indexing and in-memory caching. Start simple, iterate outwards. Begin with RAG and a handful of documents. If your needs are simpler, try CAG. Both patterns run entirely offline. Clone the repository, swap in your own documents, and start building. The best way to learn is to get your hands on the code. This project is open source under the MIT licence. It is a scenario sample for learning and experimentation, not production medical or safety advice. local-rag on GitHub ¡ local-cag on GitHub ¡ Foundry LocalBuilding an Offline AI Interview Coach with Foundry Local, RAG, and SQLite
How to build a 100% offline, AI-powered interview preparation tool using Microsoft Foundry Local, Retrieval-Augmented Generation, and nothing but JavaScript. Foundry Local 100% Offline RAG + TF-IDF JavaScript / Node.js Contents Introduction What is RAG and Why Offline? Architecture Overview Setting Up Foundry Local Building the RAG Pipeline The Chat Engine Dual Interfaces: Web & CLI Testing Adapting for Your Own Use Case What I Learned Getting Started Introduction Imagine preparing for a job interview with an AI assistant that knows your CV inside and out, understands the job you're applying for, and generates tailored questions, all without ever sending your data to the cloud. That's exactly what Interview Doctor does. Interview Doctor's web UI, a polished, dark-themed interface running entirely on your local machine. In this post, I'll walk you through how I built an interview prep tool as a fully offline JavaScript application using: Foundry Local â Microsoft's on-device AI runtime SQLite â for storing document chunks and TF-IDF vectors RAG (Retrieval-Augmented Generation) â to ground the AI in your actual documents Express.js â for the web server Node.js built-in test runner â for testing with zero extra dependencies No cloud. No API keys. No internet required. Everything runs on your machine. What is RAG and Why Does It Matter? Retrieval-Augmented Generation (RAG) is a pattern that makes AI models dramatically more useful for domain-specific tasks. Instead of relying solely on what a model learned during training (which can be outdated or generic), RAG: Retrieves relevant chunks from your own documents Augments the model's prompt with those chunks as context Generates a response grounded in your actual data For Interview Doctor, this means the AI doesn't just ask generic interview questions, it asks questions specific to your CV, your experience, and the specific job you're applying for. Why Offline RAG? Privacy is the obvious benefit, your CV and job applications never leave your device. But there's more: No API costs â run as many queries as you want No rate limits â iterate rapidly during your prep Works anywhere â on a plane, in a cafĂŠ with bad Wi-Fi, anywhere Consistent performance â no cold starts, no API latency Architecture Overview Complete architecture showing all components and data flow. The application has two interfaces (CLI and Web) that share the same core engine: Document Ingestion â PDFs and markdown files are chunked and indexed Vector Store â SQLite stores chunks with TF-IDF vectors Retrieval â queries are matched against stored chunks using cosine similarity Generation â relevant chunks are injected into the prompt sent to the local LLM Step 1: Setting Up Foundry Local First, install Foundry Local: # Windows winget install Microsoft.FoundryLocal # macOS brew install microsoft/foundrylocal/foundrylocal The JavaScript SDK handles everything else â starting the service, downloading the model, and connecting: import { FoundryLocalManager } from "foundry-local-sdk"; import { OpenAI } from "openai"; const manager = new FoundryLocalManager(); const modelInfo = await manager.init("phi-3.5-mini"); // Foundry Local exposes an OpenAI-compatible API const openai = new OpenAI({ baseURL: manager.endpoint, // Dynamic port, discovered by SDK apiKey: manager.apiKey, }); â ď¸ Key Insight Foundry Local uses a dynamic port never hardcode localhost:5272 . Always use manager.endpoint which is discovered by the SDK at runtime. Step 2: Building the RAG Pipeline Document Chunking Documents are split into overlapping chunks of ~200 tokens. The overlap ensures important context isn't lost at chunk boundaries: export function chunkText(text, maxTokens = 200, overlapTokens = 25) { const words = text.split(/\s+/).filter(Boolean); if (words.length <= maxTokens) return [text.trim()]; const chunks = []; let start = 0; while (start < words.length) { const end = Math.min(start + maxTokens, words.length); chunks.push(words.slice(start, end).join(" ")); if (end >= words.length) break; start = end - overlapTokens; } return chunks; } Why 200 tokens with 25-token overlap? Small chunks keep retrieved context compact for the model's limited context window. Overlap prevents information loss at boundaries. And it's all pure string operations, no dependencies needed. TF-IDF Vectors Instead of using a separate embedding model (which would consume precious memory alongside the LLM), we use TF-IDF, a classic information retrieval technique: export function termFrequency(text) { const tf = new Map(); const tokens = text .toLowerCase() .replace(/[^a-z0-9\-']/g, " ") .split(/\s+/) .filter((t) => t.length > 1); for (const t of tokens) { tf.set(t, (tf.get(t) || 0) + 1); } return tf; } export function cosineSimilarity(a, b) { let dot = 0, normA = 0, normB = 0; for (const [term, freq] of a) { normA += freq * freq; if (b.has(term)) dot += freq * b.get(term); } for (const [, freq] of b) normB += freq * freq; if (normA === 0 || normB === 0) return 0; return dot / (Math.sqrt(normA) * Math.sqrt(normB)); } Each document chunk becomes a sparse vector of word frequencies. At query time, we compute cosine similarity between the query vector and all stored chunk vectors to find the most relevant matches. SQLite as a Vector Store Chunks and their TF-IDF vectors are stored in SQLite using sql.js (pure JavaScript â no native compilation needed): export class VectorStore { // Created via: const store = await VectorStore.create(dbPath) insert(docId, title, category, chunkIndex, content) { const tf = termFrequency(content); const tfJson = JSON.stringify([...tf]); this.db.run( "INSERT INTO chunks (...) VALUES (?, ?, ?, ?, ?, ?)", [docId, title, category, chunkIndex, content, tfJson] ); this.save(); } search(query, topK = 5) { const queryTf = termFrequency(query); // Score each chunk by cosine similarity, return top-K } } đĄ Why SQLite for Vectors? For a CV plus a few job descriptions (dozens of chunks), brute-force cosine similarity over SQLite rows is near-instant (~1ms). No need for Pinecone, Qdrant, or Chroma â just a single .db file on disk. Step 3: The RAG Chat Engine The chat engine ties retrieval and generation together: async *queryStream(userMessage, history = []) { // 1. Retrieve relevant CV/JD chunks const chunks = this.retrieve(userMessage); const context = this._buildContext(chunks); // 2. Build the prompt with retrieved context const messages = [ { role: "system", content: SYSTEM_PROMPT }, { role: "system", content: `Retrieved context:\n\n${context}` }, ...history, { role: "user", content: userMessage }, ]; // 3. Stream from the local model const stream = await this.openai.chat.completions.create({ model: this.modelId, messages, temperature: 0.3, stream: true, }); // 4. Yield chunks as they arrive for await (const chunk of stream) { const content = chunk.choices[0]?.delta?.content; if (content) yield { type: "text", data: content }; } } The flow is straightforward: vectorize the query, retrieve with cosine similarity, build a prompt with context, and stream from the local LLM. The temperature: 0.3 keeps responses focused â important for interview preparation where consistency matters. Step 4: Dual Interfaces â Web & CLI Web UI The web frontend is a single HTML file with inline CSS and JavaScript â no build step, no framework, no React or Vue. It communicates with the Express backend via REST and SSE: File upload via multipart/form-data Streaming chat via Server-Sent Events (SSE) Quick-action buttons for common follow-up queries (coaching tips, gap analysis, mock interview) The setup form with job title, seniority level, and a pasted job description â ready to generate tailored interview questions. CLI The CLI provides the same experience in the terminal with ANSI-coloured output: npm run cli It walks you through uploading your CV, entering the job details, and then generates streaming questions. Follow-up questions work interactively. Both interfaces share the same ChatEngine class, they're thin layers over identical logic. Edge Mode For constrained devices, toggle Edge mode to use a compact system prompt that fits within smaller context windows: Edge mode activated, uses a minimal prompt for devices with limited resources. Step 5: Testing Tests use the Node.js built-in test runner, no Jest, no Mocha, no extra dependencies: import { describe, it } from "node:test"; import assert from "node:assert/strict"; describe("chunkText", () => { it("returns single chunk for short text", () => { const chunks = chunkText("short text", 200, 25); assert.equal(chunks.length, 1); }); it("maintains overlap between chunks", () => { // Verifies overlapping tokens between consecutive chunks }); }); npm test Tests cover the chunker, vector store, config, prompts, and server API contract, all without needing Foundry Local running. Adapting for Your Own Use Case Interview Doctor is a pattern, not just a product. You can adapt it for any domain: What to Change How Domain documents Replace files in docs/ with your content System prompt Edit src/prompts.js Chunk sizes Adjust config.chunkSize and config.chunkOverlap Model Change config.model â run foundry model list UI Modify public/index.html â it's a single file Ideas for Adaptation Customer support bot â ingest your product docs and FAQs Code review assistant â ingest coding standards and best practices Study guide â ingest textbooks and lecture notes Compliance checker â ingest regulatory documents Onboarding assistant â ingest company handbooks and processes What I Learned Offline AI is production-ready. Foundry Local + small models like Phi-3.5 Mini are genuinely useful for focused tasks. You don't need vector databases for small collections. SQLite + TF-IDF is fast, simple, and has zero infrastructure overhead. RAG quality depends on chunking. Getting chunk sizes right for your use case is more impactful than the retrieval algorithm. The OpenAI-compatible API is a game-changer. Switching from cloud to local was mostly just changing the baseURL . Dual interfaces are easy when you share the engine. The CLI and Web UI are thin layers over the same ChatEngine class. ⥠Performance Notes On a typical laptop (no GPU): ingestion takes under 1 second for ~20 documents, retrieval is ~1ms, and the first LLM token arrives in 2-5 seconds. Foundry Local automatically selects the best model variant for your hardware (CUDA GPU, NPU, or CPU). Getting Started git clone https://github.com/leestott/interview-doctor-js.git cd interview-doctor-js npm install npm run ingest npm start # Web UI at http://127.0.0.1:3000 # or npm run cli # Interactive terminal The full source code is on GitHub. Star it, fork it, adapt it â and good luck with your interviews! Resources Foundry Local â Microsoft's on-device AI runtime Foundry Local SDK (npm) â JavaScript SDK Foundry Local GitHub â Source, samples, and documentation Local RAG Reference â Reference RAG implementation Interview Doctor (JavaScript) â This project's source codeAnnouncing the IQ Series: Foundry IQ
AI agents are rapidly becoming a new way to build applications. But for agents to be truly useful, they need access to the knowledge and context that helps them reason about the world they operate in. Thatâs where Foundry IQ comes in. Today weâre announcing the IQ Series: Foundry IQ, a new set of developer-focused episodes exploring how to build knowledge-centric AI systems using Foundry IQ. The series focuses on the core ideas behind how modern AI systems work with knowledge, how they retrieve information, reason across sources, synthesize answers, and orchestrate multi-step interactions. Instead of treating retrieval as a single step in a pipeline, Foundry IQ approaches knowledge as something that AI systems actively work with throughout the reasoning process. The IQ Series breaks down these concepts and shows how they come together when building real AI applications. You can explore the series and all the accompanying samples here: đ https://aka.ms/iq-series What is Foundry IQ? Foundry IQ helps AI systems work with knowledge in a more structured and intentional way. Rather than wiring retrieval logic directly into every application, developers can define knowledge bases that connect to documents, data sources, and other information systems. AI agents can then query these knowledge bases to gather the context they need to generate responses, make decisions, or complete tasks. This model allows knowledge to be organized, reused, and combined across applications, instead of being rebuilt for each new scenario. What's covered in the IQ Series? The Foundry IQ episodes in the IQ Series explore the key building blocks behind knowledge-driven AI systems from how knowledge enters the system to how agents ultimately query and use it. The series is released as three weekly episodes: Foundry IQ: Unlocking Knowledge for Your Agents â March 18, 2026: Introduces Foundry IQ and the core ideas behind it. The episode explains how AI agents work with knowledge and walks through the main components of the Foundry IQ that support knowledge-driven applications. Foundry IQ: Building the Data Pipeline with Knowledge Sources â March 25, 2026: Focuses on Knowledge Sources and how different types of content flow into Foundry IQ. It explores how systems such as SharePoint, Fabric, OneLake, Azure Blob Storage, Azure AI Search, and the web contribute information that AI systems can later retrieve and use. Foundry IQ: Querying the Multi-Source AI Knowledge Bases â April 1, 2026: Dives into the Knowledge Bases and how multiple knowledge sources can be organized behind a single endpoint. The episode demonstrates how AI systems query across these sources and synthesize information to answer complex questions. Each episode includes a short executive introduction, a tech talk exploring the topic in depth, and a visual recap with doodle summaries of the key ideas. Alongside the episodes, the GitHub repository provides cookbooks with sample code, summary of the episodes, and additinal learning resources, so developers can explore the concepts and apply them in their own projects. Explore the Repo All episodes and supporting materials live in the IQ Series repository: đ https://aka.ms/iq-series Inside the repository youâll find: The Foundry IQ episode links Cookbooks for each episode Links to documentation and additional resources If you're building AI agents or exploring how AI systems can work with knowledge, the IQ Series is a great place to start. Watch the episodes and explore the cookbooks! Weâre excited to see what you build and welcome your feedback & ideas as the series evolves.Building real-world AI automation with Foundry Local and the Microsoft Agent Framework
A hands-on guide to building real-world AI automation with Foundry Local, the Microsoft Agent Framework, and PyBullet. No cloud subscription, no API keys, no internet required. Why Developers Should Care About Offline AI Imagine telling a robot arm to "pick up the cube" and watching it execute the command in a physics simulator, all powered by a language model running on your laptop. No API calls leave your machine. No token costs accumulate. No internet connection is needed. That is what this project delivers, and every piece of it is open source and ready for you to fork, extend, and experiment with. Most AI demos today lean on cloud endpoints. That works for prototypes, but it introduces latency, ongoing costs, and data privacy concerns. For robotics and industrial automation, those trade-offs are unacceptable. You need inference that runs where the hardware is: on the factory floor, in the lab, or on your development machine. Foundry Local gives you an OpenAI-compatible endpoint running entirely on-device. Pair it with a multi-agent orchestration framework and a physics engine, and you have a complete pipeline that translates natural language into validated, safe robot actions. This post walks through how we built it, why the architecture works, and how you can start experimenting with your own offline AI simulators today. Architecture The system uses four specialised agents orchestrated by the Microsoft Agent Framework: Agent What It Does Speed PlannerAgent Sends user command to Foundry Local LLM â JSON action plan 4â45 s SafetyAgent Validates against workspace bounds + schema < 1 ms ExecutorAgent Dispatches actions to PyBullet (IK, gripper) < 2 s NarratorAgent Template summary (LLM opt-in via env var) < 1 ms User (text / voice) â âź ââââââââââââââââ â Orchestrator â ââââââââŹââââââââ â ââââââ´âââââ âź âź Planner Narrator â âź Safety â âź Executor â âź PyBullet Setting Up Foundry Local from foundry_local import FoundryLocalManager import openai manager = FoundryLocalManager("qwen2.5-coder-0.5b") client = openai.OpenAI( base_url=manager.endpoint, api_key=manager.api_key, ) resp = client.chat.completions.create( model=manager.get_model_info("qwen2.5-coder-0.5b").id, messages=[{"role": "user", "content": "pick up the cube"}], max_tokens=128, stream=True, ) from foundry_local import FoundryLocalManager import openai manager = FoundryLocalManager("qwen2.5-coder-0.5b") client = openai.OpenAI( base_url=manager.endpoint, api_key=manager.api_key, ) resp = client.chat.completions.create( model=manager.get_model_info("qwen2.5-coder-0.5b").id, messages=[{"role": "user", "content": "pick up the cube"}], max_tokens=128, stream=True, ) The SDK auto-selects the best hardware backend (CUDA GPU â QNN NPU â CPU). No configuration needed. How the LLM Drives the Simulator Understanding the interaction between the language model and the physics simulator is central to the project. The two never communicate directly. Instead, a structured JSON contract forms the bridge between natural language and physical motion. From Words to JSON When a user says âpick up the cubeâ, the PlannerAgent sends the command to the Foundry Local LLM alongside a compact system prompt. The prompt lists every permitted tool and shows the expected JSON format. The LLM responds with a structured plan: { "type": "plan", "actions": [ {"tool": "describe_scene", "args": {}}, {"tool": "pick", "args": {"object": "cube_1"}} ] } The planner parses this response, validates it against the action schema, and retries once if the JSON is malformed. This constrained output format is what makes small models (0.5B parameters) viable: the response space is narrow enough that even a compact model can produce correct JSON reliably. From JSON to Motion Once the SafetyAgent approves the plan, the ExecutorAgent maps each action to concrete PyBullet calls: move_ee(target_xyz) : The target position in Cartesian coordinates is passed to PyBullet's inverse kinematics solver, which computes the seven joint angles needed to place the end-effector at that position. The robot then interpolates smoothly from its current joint state to the target, stepping the physics simulation at each increment. pick(object) : This triggers a multi-step grasp sequence. The controller looks up the object's position in the scene, moves the end-effector above the object, descends to grasp height, closes the gripper fingers with a configurable force, and lifts. At every step, PyBullet resolves contact forces and friction so that the object behaves realistically. place(target_xyz) : The reverse of a pick. The robot carries the grasped object to the target coordinates and opens the gripper, allowing the physics engine to drop the object naturally. describe_scene() : Rather than moving the robot, this action queries the simulation state and returns the position, orientation, and name of every object on the table, along with the current end-effector pose. The Abstraction Boundary The critical design choice is that the LLM knows nothing about joint angles, inverse kinematics, or physics. It operates purely at the level of high-level tool calls ( pick , move_ee ). The ActionExecutor translates those tool calls into the low-level API that PyBullet provides. This separation means the LLM prompt stays simple, the safety layer can validate plans without understanding kinematics, and the executor can be swapped out without retraining or re-prompting the model. Voice Input Pipeline Voice commands follow three stages: Browser capture: MediaRecorder captures audio, client-side resamples to 16 kHz mono WAV Server transcription: Foundry Local Whisper (ONNX, cached after first load) with automatic 30 s chunking Command execution: transcribed text goes through the same Planner â Safety â Executor pipeline The mic button (đ¤) only appears when a Whisper model is cached or loaded. Whisper models are filtered out of the LLM dropdown. Web UI in Action Pick command Describe command Move command Reset command Performance: Model Choice Matters Model Params Inference Pipeline Total qwen2.5-coder-0.5b 0.5 B ~4 s ~5 s phi-4-mini 3.6 B ~35 s ~36 s qwen2.5-coder-7b 7 B ~45 s ~46 s For interactive robot control, qwen2.5-coder-0.5b is the clear winner: valid JSON for a 7-tool schema in under 5 seconds. The Simulator in Action Here is the Panda robot arm performing a pick-and-place sequence in PyBullet. Each frame is rendered by the simulator's built-in camera and streamed to the web UI in real time. Overview Reaching Above the cube Gripper detail Front interaction Side layout Get Running in Five Minutes You do not need a GPU, a cloud account, or any prior robotics experience. The entire stack runs on a standard development machine. # 1. Install Foundry Local winget install Microsoft.FoundryLocal # Windows brew install foundrylocal # macOS # 2. Download models (one-time, cached locally) foundry model run qwen2.5-coder-0.5b # Chat brain (~4 s inference) foundry model run whisper-base # Voice input (194 MB) # 3. Clone and set up the project git clone https://github.com/leestott/robot-simulator-foundrylocal cd robot-simulator-foundrylocal .\setup.ps1 # or ./setup.sh on macOS/Linux # 4. Launch the web UI python -m src.app --web --no-gui # â http://localhost:8080 Once the server starts, open your browser and try these commands in the chat box: "pick up the cube": the robot grasps the blue cube and lifts it "describe the scene": returns every object's name and position "move to 0.3 0.2 0.5": sends the end-effector to specific coordinates "reset": returns the arm to its neutral pose If you have a microphone connected, hold the mic button and speak your command instead of typing. Voice input uses a local Whisper model, so your audio never leaves the machine. Experiment and Build Your Own The project is deliberately simple so that you can modify it quickly. Here are some ideas to get started. Add a new robot action The robot currently understands seven tools. Adding an eighth takes four steps: Define the schema in TOOL_SCHEMAS ( src/brain/action_schema.py ). Write a _do_<tool> handler in src/executor/action_executor.py . Register it in ActionExecutor._dispatch . Add a test in tests/test_executor.py . For example, you could add a rotate_ee tool that spins the end-effector to a given roll/pitch/yaw without changing position. Add a new agent Every agent follows the same pattern: an async run(context) method that reads from and writes to a shared dictionary. Create a new file in src/agents/ , register it in orchestrator.py , and the pipeline will call it in sequence. Ideas for new agents: VisionAgent: analyse a camera frame to detect objects and update the scene state before planning. CostEstimatorAgent: predict how many simulation steps an action plan will take and warn the user if it is expensive. ExplanationAgent: generate a step-by-step natural language walkthrough of the plan before execution, allowing the user to approve or reject it. Swap the LLM python -m src.app --web --model phi-4-mini Or use the model dropdown in the web UI; no restart is needed. Try different models and compare accuracy against inference speed. Smaller models are faster but may produce malformed JSON more often. Larger models are more accurate but slower. The retry logic in the planner compensates for occasional failures, so even a small model works well in practice. Swap the simulator PyBullet is one option, but the architecture does not depend on it. You could replace the simulation layer with: MuJoCo: a high-fidelity physics engine popular in reinforcement learning research. Isaac Sim: NVIDIA's GPU-accelerated robotics simulator with photorealistic rendering. Gazebo: the standard ROS simulator, useful if you plan to move to real hardware through ROS 2. The only requirement is that your replacement implements the same interface as PandaRobot and GraspController . Build something completely different The pattern at the heart of this project (LLM produces structured JSON, safety layer validates, executor dispatches to a domain-specific engine) is not limited to robotics. You could apply the same architecture to: Home automation: "turn off the kitchen lights and set the thermostat to 19 degrees" translated into MQTT or Zigbee commands. Game AI: natural language control of characters in a game engine, with the safety agent preventing invalid moves. CAD automation: voice-driven 3D modelling where the LLM generates geometry commands for OpenSCAD or FreeCAD. Lab instrumentation: controlling scientific equipment (pumps, stages, spectrometers) via natural language, with the safety agent enforcing hardware limits. From Simulator to Real Robot One of the most common questions about projects like this is whether it could control a real robot. The answer is yes, and the architecture is designed to make that transition straightforward. What Stays the Same The entire upper half of the pipeline is hardware-agnostic: The LLM planner generates the same JSON action plans regardless of whether the target is simulated or physical. It has no knowledge of the underlying hardware. The safety agent validates workspace bounds and tool schemas. For a real robot, you would tighten the bounds to match the physical workspace and add checks for obstacle clearance using sensor data. The orchestrator coordinates agents in the same sequence. No changes are needed. The narrator reports what happened. It works with any result data the executor returns. What Changes The only component that must be replaced is the executor layer, specifically the PandaRobot class and the GraspController . In simulation, these call PyBullet's inverse kinematics solver and step the physics engine. On a real robot, they would instead call the hardware driver. For a Franka Emika Panda (the same robot modelled in the simulation), the replacement options include: libfranka: Franka's C++ real-time control library, which accepts joint position or torque commands at 1 kHz. ROS 2 with MoveIt: A robotics middleware stack that provides motion planning, collision avoidance, and hardware abstraction. The move_ee action would become a MoveIt goal, and the framework would handle trajectory planning and execution. Franka ROS 2 driver: Combines libfranka with ROS 2 for a drop-in replacement of the simulation controller. The ActionExecutor._dispatch method maps tool names to handler functions. Replacing _do_move_ee , _do_pick , and _do_place with calls to a real robot driver is the only code change required. Key Considerations for Real Hardware Safety: A simulated robot cannot cause physical harm; a real robot can. The safety agent would need to incorporate real-time collision checking against sensor data (point clouds from depth cameras, for example) rather than relying solely on static workspace bounds. Perception: In simulation, object positions are known exactly. On a real robot, you would need a perception system (cameras with object detection or fiducial markers) to locate objects before grasping. Calibration: The simulated robot's coordinate frame matches the URDF model perfectly. A real robot requires hand-eye calibration to align camera coordinates with the robot's base frame. Latency: Real actuators have physical response times. The executor would need to wait for motion completion signals from the hardware rather than stepping a simulation loop. Gripper feedback: In PyBullet, grasp success is determined by contact forces. A real gripper would provide force or torque feedback to confirm whether an object has been securely grasped. The Simulation as a Development Tool This is precisely why simulation-first development is valuable. You can iterate on the LLM prompts, agent logic, and command pipeline without risk to hardware. Once the pipeline reliably produces correct action plans in simulation, moving to a real robot is a matter of swapping the lowest layer of the stack. Key Takeaways for Developers On-device AI is production-ready. Foundry Local serves models through a standard OpenAI-compatible API. If your code already uses the OpenAI SDK, switching to local inference is a one-line change to base_url . Small models are surprisingly capable. A 0.5B parameter model produces valid JSON action plans in under 5 seconds. For constrained output schemas, you do not need a 70B model. Multi-agent pipelines are more reliable than monolithic prompts. Splitting planning, validation, execution, and narration across four agents makes each one simpler to test, debug, and replace. Simulation is the safest way to iterate. You can refine LLM prompts, agent logic, and tool schemas without risking real hardware. When the pipeline is reliable, swapping the executor for a real robot driver is the only change needed. The pattern generalises beyond robotics. Structured JSON output from an LLM, validated by a safety layer, dispatched to a domain-specific engine: that pattern works for home automation, game AI, CAD, lab equipment, and any other domain where you need safe, structured control. You can start building today. The entire project runs on a standard laptop with no GPU, no cloud account, and no API keys. Clone the repository, run the setup script, and you will have a working voice-controlled robot simulator in under five minutes. Ready to start building? Clone the repository, try the commands, and then start experimenting. Fork it, add your own agents, swap in a different simulator, or apply the pattern to an entirely different domain. The best way to learn how local AI can solve real-world problems is to build something yourself. Source code: github.com/leestott/robot-simulator-foundrylocal Built with Foundry Local, Microsoft Agent Framework, PyBullet, and FastAPI.
Power Apps Vibe Experience: Build Business Apps at the Speed of Ideas
Power Apps Vibe Experience: Building Business Applications with AI in Minutes Organizations today operate in a fast-paced digital environment where new business challenges emerge constantly. Whether itâs managing internal workflows, tracking projects, or collecting customer feedback, businesses often require custom applications to support their processes. However, traditional application developmentâeven with modern low-code toolsâstill requires time, technical expertise, and coordination between multiple teams. Designing the user interface, building the data model, writing logic, and integrating services can take weeks or even months. To address this challenge, Microsoft Power Apps has introduced the Power Apps Vibe experience, a new AI-driven way to build enterprise applications by simply describing the outcome you want. This innovative approach represents a significant evolution in the Microsoft Power Platform ecosystem, enabling organizations to move from idea to working application faster than ever before. What Is the Power Apps Vibe Experience? The Power Apps Vibe experience is an AI-first development environment designed to simplify and accelerate the creation of business applications. Instead of manually designing each component of an application, users can start by describing their business requirement in natural language. For example, a user might type: âCreate an internal app where employees can submit support requests, track approvals, and receive notifications.â Based on this prompt, the platform automatically generates the foundational elements required to build the application. These include: Business requirements and solution plan A structured data model built on Microsoft Dataverse User interface layouts and navigation Forms and pages Application logic and workflows All these elements are created within a single integrated development workspace. This dramatically reduces the time and complexity associated with traditional application development. Why Power Apps Vibe Is Important for Modern Organizations Many organizations face a common challenge: they have plenty of ideas for improving processes but lack the resources to implement them quickly. Building custom software often requires developers, project managers, designers, and testers. Even with low-code platforms, organizations still need time to design data models and configure application logic. The Vibe experience addresses these challenges by introducing AI-assisted application generation. Key Benefits Faster Time to Value Organizations can create functional applications in minutes instead of weeks. This allows teams to rapidly prototype ideas and deliver solutions faster. Empowering Citizen Developers Business users who understand the problem best can now participate directly in building solutions. They do not need advanced coding skills to create useful applications. Enterprise-Grade Security Applications built using Power Apps Vibe run on Microsoft Dataverse, which provides: Role-based access control Secure data storage Compliance and governance capabilities This ensures that even AI-generated applications meet enterprise security requirements. Consistent Governance IT administrators maintain full control through: Tenant policies Data governance rules Environment management This balance allows organizations to encourage innovation while maintaining control over their technology environment. How the Power Apps Vibe Experience Works The development process in the Vibe experience follows a simple three-step model. Step 1: Describe the Business Problem The process begins with a natural language description of the application requirement. For example: âCreate an inventory management app where warehouse staff can track stock levels, update inventory, and generate reports.â The AI analyses this prompt to understand the core business objectives. Step 2: AI Generates the Application Plan Next, the system produces a structured plan for the application. This plan typically includes: User roles and permissions Data entities and relationships Functional requirements Suggested workflows This planning stage helps ensure the application is aligned with the intended business scenario. Step 3: Automated Application Creation Once the plan is confirmed, the platform automatically generates the application. This includes: Data tables and schema Forms and screens Navigation structure Business logic Basic workflows Because the platform creates these components together, the data model and application structure remain synchronized. Core Capabilities of Power Apps Vibe Rapid Prototyping One of the most powerful features of the Vibe experience is rapid prototyping. Teams can quickly convert ideas into working applications that can be tested and refined. Benefits include: Faster proof-of-concept development Reduced design effort Early feedback from stakeholders Unified Development Environment Traditional application development often involves multiple tools and stages. Developers may use different platforms for: Planning Data modelling UI design Workflow creation The Vibe experience combines these activities into a single integrated workspace. This unified environment ensures that changes to the data model automatically update the application. AI-Assisted Development Artificial intelligence plays a continuous role throughout the development lifecycle. AI can assist with: Prompt suggestions Code generation App design improvements Architecture recommendations Because the system understands the context of the business problem, it can suggest optimizations and enhancements. Instant Application Generation With a single prompt, the platform can generate an entire application structure. Automatically generated components include: Data tables Forms and pages Navigation menus Business rules Application logic This dramatically reduces the effort required to build enterprise-ready applications. Power Apps Vibe vs Canvas Apps vs Model-Driven Apps Within the Microsoft Power Apps ecosystem, developers can choose from multiple development approaches. Each approach serves different use cases. Feature Power Apps Vibe Canvas Apps Model-Driven Apps Development Style AI-generated Visual UI design Data-driven Creation Method Natural language Drag-and-drop designer Data schema UI Customization Moderate High Limited Data Model Automatically generated Flexible sources Dataverse required Speed Very fast Medium Medium Ideal Use Case Rapid prototypes Custom UI apps Enterprise solutions Conclusion The Power Apps Vibe experience represents a major step forward in the evolution of low-code platforms. By combining artificial intelligence with the capabilities of Microsoft Power Platform, Microsoft is enabling organizations to transform ideas into working applications faster than ever before. For businesses seeking to improve productivity, streamline workflows, and innovate rapidly, the Vibe experience offers a powerful new way to build enterprise solutions. Reference Links: https://learn.microsoft.com/en-us/power-apps/vibe/overview https://learn.microsoft.com/en-us/power-apps/vibe/create-app-data-plan https://learn.microsoft.com/en-us/power-platform/released-versions/new-powerapps
