Agentic Mentor: A Specification-Driven, Multi-Agent Learning Tool
Project Overview Rather than relying on the model to know when the student understands, we built a pipeline where progress itself is gated: a student cannot move forward until they can demonstrate they understood what was just built. And rather than depending on commercial, token-billed APIs to make this teaching possible, the system is designed to run on locally hosted or free-tier models, keeping it accessible to students and institutions alike. As our contribution to Microsoft, we've handed over the full public repository behind Agentic Mentor. Given how central token-based services have become to software development, we see real potential for this approach in initiatives like GitHub Education, where affordable, agentic tools for students matter. In this post, we'll walk through what motivated the project, how Agentic Mentor works under the hood, and what we learned putting it to the test on a real piece of university coursework. The Project Journey This project was completed over three months. The first few weeks were spent on background research and requirements elicitation. We reviewed the literature on AI-assisted learning, specification-driven development and prompt ambiguity, elicited functional and non-functional requirements from our client supervisor at Microsoft, Lee Stott, and our academic supervisors, and broke the work into components we could build and test independently. This told us what was realistically achievable in the time we had, and which features were core to the tool rather than desirable extras. Implementation was coordinated through GitHub, with the team meeting daily to keep parallel work in sync and weekly meetings with both our supervisors and our client to report progress and check we were still building the right thing. We worked this way because the shape of the tool was still settling once implementation began, and that frequency of contact meant a wrong assumption surfaced in days rather than weeks. The most consequential decision of the project came out of those conversations. Agentic Mentor had been proposed as an assessment tool, but the AI does not write the assessment brief, so it cannot be relied on to read it as the academic intended, and grading on top of that reading would carry the model's misunderstanding into a student's mark. We pivoted to a learning tool, and the design of the pipeline followed from there. The final month went on completing both interfaces and evaluating the system, first against SpecBench as a correctness check, then against a real master's-level coursework. Two problems surfaced. Our GPU infrastructure went offline, so we moved to free-tier cloud models and scheduled runs around quota resets. And one of the questions the system generated turned out to be subtly wrong, and we had read it and accepted it without question, which is the automation bias we had spent the project writing about. Technical Details Agentic Mentor is built as a multi-agent, specification-driven pipeline. It has four agents; Research, Ingestion, Mentoring, and Viva. Each agent was implemented using the Microsoft Agent Framework. We chose to give each phase its own dedicated agent so that every stage could be equipped with the specific tools its task required, rather than relying on one general-purpose agent to handle the whole assignment. The agents run linearly, with each one writing its output to disk and the orchestrator passing the resulting file paths to the next stage. This design lets a session pause and resume. An overview of Agentic Mentor's architecture is seen in the figure below. Research Agent. This stage grounds the pipeline in external context. It uses GitHub and arXiv MCP servers to gather relevant literature and existing implementations. MCP gave the agent a uniform interface to both sources, meaning further ones could be added later simply by connecting another server, without reworking the agent itself. Ingestion Agent. Built around GitHub SpecKit, an open-source toolkit for spec-driven development, this agent converts the assignment brief and research context into structured specification files. Rather than letting the model resolve ambiguities in the brief on its own, SpecKit's clarification step surfaces unclear points directly to the student, who must answer before the pipeline continues. This choice keeps the student engaged in design decisions rather than letting the model make them silently. Mentoring Agent. This is where the student and the model actually build the project together, phase by phase, with the agent writing code and explaining its reasoning as it goes. Progress is gated: the student cannot move to the next phase until every task is complete, and a short multiple-choice checkpoint has been passed. The checkpoint tests understanding of what was just built rather than simply advancing on request. Viva Agent. Once implementation is complete, this agent interviews the student. Answers are checked against key points prepared in advance, with feedback given after each response and a full transcript saved for later review. How Microsoft's tools shaped the project. The Microsoft Agent Framework was the backbone that made the multi-agent design practical: it let us build four functionally distinct agents that could each carry their own tools and responsibilities while still communicating cleanly through a shared orchestration layer. Together with GitHub SpecKit's structured approach to specification-driven development, these tools gave Agentic Mentor a technical foundation that would have been considerably harder to assemble from scratch, and were central to making the pipeline's phase-gated, specification-first design actually work in practice rather than remaining a concept on paper. Demo This demo video shows an end-to-end demonstration of the VS Code extension of Agentic Mentor. Results and outcomes We evaluated Agentic Mentor in two phases: a benchmark check for basic correctness, followed by a real master's-level coursework that better matched the system's intended use. Phase 1: Code Correctness We used SpecBench to evaluate Agentic Mentor’s code correctness. SpecBench is a benchmark of 30 systems-level programming tasks with pre-existing test suites. We ran six of them using a locally served Qwen3.6-27B model. Our results show that tasks with common structural logic scored well, such as json_parser achieving a 97.7% pass rate. However, more complex tasks performed worse. On crypto_primitives, hallucinations prevented a testable solution being produced at all. This result reflects the limits of a small local model such as Qwen3.6-27B. However, the success of some tasks show that smaller local models have promise. Phase 2: Student Understanding The second phase evaluated the system against a real academic assignment on Test-Driven Development, which the team had previously completed without Agentic Mentor. Claude Sonnet 5 was used to reduce hallucinations and improve reasoning. We combined our own assessment as students with an interview with Jens Krinke, the module leader who set and assessed the coursework. From our perspective, we found that Agentic Mentor made a large, loosely specified project easier to approach, and the questions at each stage required active recall which helped us understand the assignment better. It improved on our original attempt by constructing a synthetic repository with known test–production pairs as concrete acceptance criteria, linking test and production files by co-occurrence in commit history rather than naming conventions, and producing unit test coverage within a properly separated project structure. The assessment of its understanding was less favourable. Jens rated its comprehension as comparable to a typical student's, but no better: it treated commit history as capable of confirming the presence of TDD, when it can only reveal the degree of its absence, and it missed an implicit hint regarding commit size. Lessons Learned Building Agentic Mentor taught our team a great deal about coding agents, local models, and the practical challenges of applying AI in education. These lessons shaped both the tool itself and how we think about deploying AI in learning contexts. Knowing where the tool fits. One of our clearest takeaways was that Agentic Mentor works best on well-specified, undergraduate-level assignments rather than open-ended, research-style coursework. Our evaluation on SpecBench's json_parser task, where the tool achieved a 97.7% pass rate, showed just how effective it can be on concrete, well-bounded problems. Our Test-Driven Development case study showed the same tool struggling once the task demanded interpreting ambiguous or partially hidden objectives. Even so, our own experience using the tool showed genuine improvements in our learning outcomes and even surfaced implementation ideas we hadn't considered ourselves, which reinforced our belief in the promise of agentic learning tools when applied to the right kind of problem. Navigating the local model trade-off. Our extensive experimentation with locally hosted models taught us that there is a trade-off between accessibility and quality. Local models performed well on smaller, simpler tasks, but as task complexity grew, code correctness declined and hallucinations became more frequent. This was a valuable lesson in engineering trade-offs: a fully local, cost-free deployment is achievable in principle, but a genuinely effective one currently depends on access to larger, cloud-hosted models. Rethinking what the tool should be for. Perhaps our biggest shift in thinking came from an idea we abandoned. We originally envisioned Agentic Mentor as an assessment tool, capable of evaluating a student's understanding for official grading. Working through the implementation made clear why that vision doesn't hold up: an AI system that didn't generate the assessment brief itself cannot be guaranteed to interpret an educator's intent correctly. This is a limitation of natural language interpretation that is well documented in the literature we reviewed. Recognizing this early enough to change course was itself a valuable exercise in engineering judgment, and it led us to reposition Agentic Mentor as a learning tool focused on helping students engage with agentic AI, rather than an assessment tool that asks AI to make judgments it isn't equipped to make reliably. Implications for Educators Our experience building and evaluating Agentic Mentor highlighted three critical takeaways for educators looking to integrate AI into their classrooms. Watch out for automation bias. Our academic evaluation revealed that the AI agents occasionally held technical misconceptions, which raised up the risk of automation bias; one of the biggest risks in AI-assisted education. When an AI speaks with absolute confidence, it’s incredibly easy to believe it. If students trust the model's outputs without sufficient questioning, they bypass the critical thinking the coursework was meant to provoke. Educational AI needs built-in friction to force the AI to highlight its own uncertainties and encourage students to question the output. Be careful using AI as an Assessment tool. The discovery of the AI’s technical misconceptions also suggests caution using AI for grading or formal assessment. Because the model can misunderstand core concepts the exact same way a student might, it cannot reliably evaluate student comprehension. This also led us to pivot Agentic Mentor strictly to a learning tool. AI is an excellent tool for supporting student learning through phased tasks, but it is not currently reliable enough to act as an autonomous judge of a student's underlying understanding. Agentic Development is reshaping software engineering. As agentic development reshapes software engineering, educators must shift their focus from teaching students how to write code manually, to teaching them how to prompt coding agents. This includes using practices such as specification-driven development, which reduces AI errors in code by preventing ambiguities in natural language prompts. Future Development The limitations we encountered while building Agentic Mentor point to two clear directions for further development. Both would make the system a more reliable learning tool for students. Validation agent Agentic Mentor currently interprets a project with the same gaps and misconceptions a typical student might bring to it. Because the system presents its output with confidence, those gaps can be passed on unchallenged. We propose extending the pipeline with a validation agent. This component would be dedicated to interrogating the questions and conclusions the system produces. Its purpose would be to introduce a deliberate layer of hesitation, surfacing assumptions explicitly rather than allowing them to reach the student as established fact. The intended outcome is a reduction in automation bias, encouraging students to engage critically with the system's output rather than accepting it uncritically. Integration with academic platforms As long as the system's understanding of an assignment is constrained in the same ways a student's is, it cannot be relied upon to identify a task's critical elements. A future version could accept explicit input from educators, specifying those critical elements in advance along with guidance on how students should be directed through them. This would require Agentic Mentor to move beyond a standalone local tool and integrate with the institution's existing education platform. Such integration would give educators direct control over the system's behaviour, making it both more trustworthy and more productive as a learning aid. Conclusion Agentic Mentor was built to help students learn from their coding coursework rather than simply complete it, embedding the learning process into the structure of the system instead of relying on vibe coding. We found that a model cannot be trusted to be consistently correct. Running on smaller local models trades capability for accessibility. And measuring student understanding is difficult without a large user study. Agentic Mentor nonetheless represents a step towards integrating AI-assisted development into education while preserving student understanding. With adequate guardrails against automation bias, it can serve as an accessible tool that adapts the learning process to the changes AI has brought to software development. Call to Action The challenge of "one-click" AI code generation in education is here to stay, but with Agentic Mentor, we aim to keep students actively engaged in problem-solving rather than bypassing it. We invite you to explore our work and help us build the future of AI in education. Explore the Code: Visit the Agentic Mentor GitHub Repository Run It Your Way: Execute the orchestrator via the CLI (see the main README), or use our custom GUI by running the VS Code Extension (located in the agentic-mentor-extension folder). Tools: Our pipeline is built on the Microsoft Agent Framework for multi-agent routing and GitHub SpecKit to enforce strict Specification-Driven Development. Team Our team involved in developing this project included 6 members. All of us are Masters students at UCL, studying either Software Systems Engineering or Artificial Intelligence and Data Engineering. Mark Connor – Team Leader – Software Engineer GitHub URL: https://github.com/markjconnor LinkedIn URL: http://www.linkedin.com/in/mark-connor2003 Alexander Filippov – Software Engineer GitHub URL: https://github.com/ucabavf LinkedIn URL: https://www.linkedin.com/in/alexander-f-003a5721b Weeraya Hew – Software Engineer GitHub URL: https://github.com/tingwry LinkedIn URL: https://www.linkedin.com/in/weeraya-hew-924a19261 Tanishka Jaikrishnia – Software Engineer GitHub URL: https://github.com/tanishkajaikrishnia LinkedIn URL: https://www.linkedin.com/in/tanishka-jaikrishnia-96b652274/ Pranav Kannan – Software Engineer GitHub URL: https://github.com/pranavk295 LinkedIn URL: https://www.linkedin.com/in/pranav-kannan-0b2a11221 Gabriel Mardakhaev – Software Engineer GitHub URL: https://github.com/gabmardakhaev LinkedIn URL: https://www.linkedin.com/in/gabriel-mardakhaev Special Thanks to Contributors We want to express our deepest gratitude to the following contributors, whose ongoing support and dedication led to the success of this project. Lee Stott, Principal Cloud Advocate Microsoft He Ye, Academic Supervisor, UCL Jens Krinke, Senior Lecturer and Academic Supervisor, UCL
Use GitHub Copilot Agent Mode to create a Copilot Chat application in 5 minutes
GitHub Copilot just got an upgrade—Agent Mode is here. 🚀 With GitHub Copilot Agent Mode, you can build applications faster than ever. In just 5 minutes, you can create a Facebook Messenger-style Copilot Chat app using Flask and GPT-4o. Simply upload a screenshot, prompt Copilot, and watch it generate your code. Fine-tune your UI, integrate GitHub Models, and deploy effortlessly. The future of AI-powered development is here—everyone can program. Are you ready to try it?Festival Web
Microsoft anuncia una nueva iniciativa para ayudarte a impulsar tu carrera en desarrollo web llamada el Festival Web, es una serie de charlas en vivo que comienza desde el 31 de octubre y finaliza el 14 de noviembre. En estas charlas podrás aprender de la mano de expertos y conocer diferentes herramientas como VSCode y GitHub. ¡Regístrate en las charlas en vivo del Festival Web y comienza tu viaje en el mundo de la programación web con expertos de la industria! Además, hay una oportunidad gratuita para quienes desean iniciarse en este campo. ¡Descubre más información sobre esta iniciativa de Microsoft en este blog!Markdown Language: An Essential Tool for Programmers
Discover the simplicity and power of Markdown, the essential tool every programmer needs for documentation, readability, and seamless integration with version control systems like GitHub. Learn the basics and elevate your projects with clear, professional documentation.Visual Studio Code AI Toolkit: Run LLMs locally
AI Toolkit is here Getting the LLMs/ SLMs on our local machines. This toolkit lets us easily download the models on our local machine. Evaluation of the model. Whenever we need to evaluate a model to check for the feasibility to any particular application, then this tool lets us do it in a playground environment, which is what we will seeing in this blog. Fine-tuning, this majorly delas with training the model further to do the tasks that we specifically want the model to do. Usually, it does a generic task and has generic data, with fine-tuning we can give it a particular flavor to perform particular task..NET MAUI on Linux with Visual Studio Code
Explore Cross-Platform Development with .NET MAUI on Linux! Dive into the latest release of the .NET MAUI extension for Visual Studio Code, enabling Linux users to develop apps for Android, Windows, iOS, and macOS. This guide offers a step-by-step tutorial on setting up your Linux system for .NET MAUI development, including installation of essential tools and leveraging the C# Dev Kit extension. Whether you're working on Ubuntu or another Linux distribution, this article, enriched with a video walkthrough by Gerald Versluis, simplifies the journey to creating powerful, versatile applications with .NET MAUI.
