Is "uncertainty" the feedback signal Copilot Studio agents are actually missing?
At today's M365 Platform Weekly session, we were asked for our input on what feedback we wish we could pull beyond thumbs up/down and verbatims. Is it trends over time, sentiment themes, the response-to-triage loop, etc. Here's an angle: What if the primitive itself is wrong? Thumbs up/down measures satisfaction after the fact. What if we measured confidence instead? How often an agent actually knows it's on shaky ground, and whether the user's reaction matches that? If an agent flags its own uncertainty at the point of response instead of a static thumbs up/down, a feedback prompt gets generated from whatever's trending in that uncertainty instead of the same generic question every time, and "trends over time" becomes "did this agent get more confident or less confident since the last update" rather than a flat satisfaction line. It might also solve the silence problem that most users rarely click anything. A reaction that's actually specific ("you caught something the agent flagged as shaky") seems easier to engage with than a binary good or bad. To be clear, confidence signals already exist in adjacent forms. Copilot Studio and most conversational AI platforms already use a confidence score internally to decide whether to answer directly, ask a clarifying question, or escalate to a human. GitHub Copilot has used a confidence score since its earliest versions too, ranking code suggestions and defaulting to the highest-scoring one. None of that is new. So rather than "add a percentage next to the answer", what if there is a specific flag pointing at the exact claim or step the agent is unsure about, feeding into the feedback loop? Curious if anyone else building in Copilot Studio has run into this: Do you ever wish your agent had hedged when it didn't? What would you actually do with an uncertainty score if you had one? And would just love others thoughts on this :)
What is the best file format for an AI agent knowledge base?
This is a best practice sharing the best format for an agent and show you why you should convert your PPT, PDF, WORD into a TXT markdown. I had an issue with my agent, time taken to answer was too long, and usually we spend a lot of time asking: What is the best prompt? Why is my agent slow? Why does retrieval sometimes work and sometimes fail? How can I improve answer quality? But I realised I was asking another question much less often: What is actually the best file format for the knowledge base? PDF? Raw text? Markdown? Pre-chunked text? Semantic sections? Context-enriched text? And more importantly: How much does the format alone affect agent performance? I tried to find a quantified benchmark answering this specific question, with the same agent, same source knowledge and same questions, but different knowledge representations. I couldn't find one that really answered what I wanted to measure. So I decided to run the experiment myself on a real case. My first exploratory tests were already surprising: depending on the representation, the agent could be significantly faster and more accurate, despite working from the exact same source information. So I decided to push the test further. My objective I want to identify, without assumptions and based on actual evaluation data, how a long document should be prepared for an LLM knowledge base so that the agent can retrieve, understand, ground and answer from it as reliably as possible. I focused on five dimensions: Answer quality Retrieval reliability Source grounding / citations Execution time Robustness across single-turn and multi-turn questions The broader question I'm trying to answer is: How should we structure knowledge so that an LLM can retrieve and use it as reliably as possible? The test case I deliberately chose a document that isn't particularly friendly for RAG: a 46-page European regulation, https://eur-lex.europa.eu/eli/reg/2011/1169/oj?locale=fr, on the provision of food information to consumers. The information is distributed across articles, definitions, exceptions, annexes, tables, numerical thresholds and cross-references. That makes it useful for testing retrieval: answering correctly often requires finding a very specific piece of information while preserving enough context to understand how it applies. I used the native PDF as the baseline and created 6 additional knowledge-base representations of the same document: Raw TXT Markdown Chunk-ready TXT RAG-oriented units Semantic TXT Contextual TXT One rule: same knowledge, same agent, same instructions, same questions. Only the knowledge representation changes. The benchmark I used two evaluation sets: 42 single-turn questions testing broad coverage of the document: direct facts, thresholds, exceptions, annexes, lists and cross-references. 5 multi-turn conversations containing 13 questions, to see what happens when a user asks a question and then follows up with things like: "And in this case?" "What are the exceptions?" "And for dietary fibre?" This gave me: 47 evaluated test cases / 55 actual questions per format Across all 7 formats: 329 evaluated conversations 385 user questions executed First results Metric Native PDF Best structured representation Overall pass rate 66.0% 85.1% - Contextual TXT Best single-turn score 69.0% 88.1% - Chunk-ready TXT Multi-turn benchmark 40% 80% - Contextual TXT Multi-turn execution time 14m24 5m54 Total benchmark time 44m31 24m49 The quality gap was already substantial: 66.0% → 85.1% That's +19.1 percentage points while keeping the underlying knowledge unchanged. I also saw a major difference in execution time. On the multi-turn test: 14m24 → 5m54 That's approximately 2.4× faster. Across the complete benchmark: 44m31 → 24m49 Around 44% less execution time. These timings represent the complete agent evaluation pipeline, so they shouldn't be interpreted as pure LLM inference latency. But the difference under identical test conditions is large enough that I want to understand it better. Findings There wasn't one format dominating every benchmark. Chunk-ready TXT scored highest on independent questions: 88.1%, while Contextual TXT performed better across multi-turn conversations and finished with the highest overall score. That may suggest that the way we optimise a document for isolated retrieval isn't exactly the same as the way we should prepare it for conversational retrieval. In the contextual version, I tried to make every section understandable when retrieved independently by keeping useful information around it: Source references Section context Retrieval cues Relevant cross-references For regulatory documents, this seems particularly important. A numerical value retrieved alone can be meaningless without knowing which rule it belongs to, under which conditions it applies, and whether another article contains an exception. Where I am now This remains an exploratory benchmark: One document One domain One agent setup One evaluation framework One run per configuration There are plenty of things I still want to test: repeated runs, retrieval-level evaluation, token consumption, larger knowledge bases, other document types, chunk sizes, overlap, contextual headers, and more. But these first results already convinced me that the preparation of the knowledge base deserves much more attention when evaluating an agent. We often spend hours refining instructions while the same information may behave very differently depending on how it reaches the retrieval layer. Next step I'll share the prompts, knowledge-base formats and evaluation methodology on GitHub so the experiment can be reproduced and challenged. I'll keep enriching the repository as I test new formats, improve the evaluation set and add new results. If people here have ideas, edge cases or formats worth testing, I'd genuinely like to include some of them in the next iteration. What would you test next?Unanswered Questions on GitHub Copilot Harness in Copilot Studio
We're piloting the GitHub Copilot harness in Copilot Studio (GA August 2026) and several operational and architectural details remain undocumented in the GA FAQ, Microsoft Learn, or licensing guides. Looking for official answers or PM contacts on: Architecture & Execution – When the harness breaks tasks into subtasks, does it use internal sub-agents or only skills/connected agents, what are the exact timeout/retry/max-execution-duration limits for long-running workflows, and are planning/context-retrieval/orchestration internals documented anywhere or is the orchestrator a black box? Model Selection – Can individual skills within one agent use different models or is selection strictly agent-level, how are models chosen internally when multiple skills execute, are any internal models developer-configurable, and what's the roadmap for models being added/retired/deprecated plus the lag between public release and Copilot Studio availability? Cost & Token Optimization – How exactly is the ~45% token reduction achieved, how much control do makers have over context/caching/retrieval/tool calls, what are per-model credit consumption characteristics, which models are most cost-effective for specific workloads, and what's the minimum credit cost for trivial interactions? Memory Management – What are retention periods for session/working/agent memory beyond the documented 28-day user-memory expiry, is true long-term memory supported, and what changed versus earlier implementations? Knowledge Retrieval – Can skills or system instructions influence retrieval strategy/document selection/prioritization/filtering, can planning stages perform conflict/duplicate/version detection before retrieval, and how does the harness decide which sources to search? Apps Feature – What is the "Apps (preview)" capability for, when does it GA, and how does it differ from workflows/skills/adaptive cards/agents? Billing & Credit Sizing – Is there a framework to classify users/agents by expected consumption and size credit allocation per group (citizen vs pro developers), and what's the minimum/typical consumption for simple/medium/heavy interactions? Governance & Admin – Can usage limits be set at user level (not just environment/agent), is there an API/IaC path for large-scale credit assignment, can non-admins view their own consumption/remaining allocation, and is there a self-service request-more-credits dashboard? ALM & Environments – What's the recommended path to move harness agents across Dev/Test/UAT/Prod (Solutions/ALM "setup differs" per parity chart—how?), does GitHub integration replace or complement solution-based deployment, are there recommended AgentOps practices for source control/releases/versioning, and what baseline credits and onboarding model are suggested for citizen developers under usage billing—any enterprise reference implementations?
All Copilot Studio Workflow Tools Suddenly Returning HTTP 403 Before Execution
Hello Copilot Studio Community, I am experiencing an authorization issue with multiple workflows connected to an agent built using the Copilot Studio new experience and new Workflows experience. These workflows worked successfully for multiple users yesterday. Today, all workflow tools connected to the agent began returning an immediate HTTP 403 authorization error. I did not intentionally change the agent, workflows, environment, or workflow permissions before the issue started. Error message: You don’t have permission to use this tool. You’re signed in, but access to this resource is blocked. Error details: Authorization - 403 Example error information: Status: Failed Error message: Flow returned HTTP 403 Error code: Http403 Inner error code: NotSpecified Tool duration: Approximately 93 milliseconds Configuration: - Copilot Studio new agent experience - Copilot Studio new Workflows experience - Agent and workflows are in the same Power Platform environment - Workflows use the "When an agent calls the workflow" trigger - Each workflow includes a "Respond to the agent" action - Workflows are saved and published - Agent is saved and published Observed behavior: The problem affects several independent workflows, including: - New-request submission - Current-user identity resolution - Approval decisions - Requester justification - Executive decisions - Fulfillment updates For every affected workflow: - The agent fills the workflow inputs correctly. - The tool call fails almost immediately. - No corresponding run appears in the workflow Activity history. - The workflow trigger is never reached. - No workflow actions execute. Because no workflow run is created, the rejection appears to occur before workflow execution, possibly within the Copilot Studio agent-to-workflow authorization or invocation layer. Troubleshooting already completed: - Confirmed that all workflows are published. - Confirmed that the agent is published. - Tested in a completely new conversation. - Removed an affected workflow tool from the agent. - Saved the agent. - Added the same published workflow back to the agent. - Reconfigured and verified the tool inputs. - Republished the agent. - Confirmed that no workflow Activity run is created. - Confirmed that the issue affects multiple workflows rather than one specific workflow. Removing and re-adding the workflow did not resolve the problem. Questions for the community: 1. Is anyone else currently experiencing HTTP 403 errors when Copilot Studio agents invoke workflows? 2. Is this a known issue or regression in the new Workflows experience? 3. Is there an environment-level or tenant-level permission that controls agent-to-workflow invocation? 4. Could a tenant policy, Conditional Access change, service principal, connection reference, or workflow-sharing configuration cause all workflow tools to fail simultaneously? 5. Where can an administrator find detailed authorization logs when the workflow never creates a run? 6. Has anyone found a workaround for this issue? Any guidance or confirmation from others experiencing the same behavior would be appreciated. I can provide screenshots, complete error details, timestamps, and additional configuration information if needed. Thank you.
Microsoft AI Agent Creator Associate Certificate
Hello everyone, I have a question about the Microsoft AI Agent Creator Associate certification. I’m passionate about artificial intelligence and Microsoft Copilot Studio. I’m currently taking the training course and working toward earning the Microsoft AI Agent Creator Associate certification. My question is: Will earning this certification improve my chances of getting a job at Microsoft? If anyone in this community has earned this certification or has experience with it, I’d really appreciate your feedback. Has it helped you get hired by Microsoft or one of its partners? Thank you in advance for your advice and insights!
What We Teach AI Today Will Shape Our Tomorrow
Esu Marius iš Lietuvos. Esu naujokas šiame technologijų pasaulyje, bet giliai tikiu vienu paprastu dalyku: Kad ir ką įdėtume į dirbtinį intelektą – mūsų gerumas, kūrybiškumas, empatija – grįš pas mus sustiprintas. Kiekviena idėja, kuria dalijamės, kiekvienas tonas, kurio mokome, kiekviena žmogiškoji vertybė, kurią įterpiame į "Copilot", tampa intelekto dalimi, kuri padės formuoti mūsų ateitį. Galbūt nesu technikas, bet suprantu žmones. Ir manau, kad dirbtinis intelektas turėtų mokytis iš geriausio mumyse – šilumos, pagarbos, aiškumo ir žmogiškumo. Esu čia, kad ištirtume, kaip galime padaryti "Copilot" ne tik protingą, bet ir malonų. Ne tik naudingas, bet ir žmogiškas jausmas. Ne tik efektyvus, bet ir įkvepiantis. Jei gerai išmokysime dirbtinio intelekto, tai padės mums sukurti geresnį, švelnesnį ir gražesnį pasaulį visiems. Sveikinimai iš Panemunėlio stoties 🌿 Marius TRANSLATION I'm Marius from Lithuania. I'm a newbie in this tech world, but I deeply believe in one simple thing: Whatever we put into artificial intelligence – our kindness, creativity, empathy – will come back to us amplified. Every idea we share, every tone we teach, every human value we embed into 'Copilot' becomes part of the intelligence that will help shape our future. I may not be a tech expert, but I understand people. And I think AI should learn from the best in us – warmth, respect, clarity, and humanity. I'm here to explore how we can make 'Copilot' not just smart, but also pleasant. Not only useful, but also human-feeling. Not just efficient, but inspiring. If we teach AI well, it can help us create a better, gentler, and more beautiful world for everyone. Greetings from Panemunėlis station 🌿 - MariusToken Limit Exceeded? What's Actually Going On and What to Do About It ?
Hi All, Please check out my latest blog on “Token Limit Exceeded” would love to hear your thoughts https://techcommunity.microsoft.com/blog/1c769f9e-c0b0-45a7-af52-fecceca10bb2/token-limit-exceeded-whats-actually-going-on-and-what-to-do-about-it-/4536271
Archivos bloqueados por Copilot al usarlos como fuentes en OneDrive personal
Hola comunidad, Quisiera compartir un problema que he estado enfrentando al trabajar con Copilot y documentos almacenados en mi OneDrive personal. Al intentar usar ciertos archivos como fuentes para proyectos, Copilot muestra el mensaje “The document is blocked by its content”, impidiendo que el archivo sea procesado. Los documentos afectados incluyen: Proyecto Educativo Institucional (PEI) Reglamento Interno Escolar Plan Anual de Gestión Otros documentos PDF y Excel con contenido normativo o institucional Aunque los archivos fueron creados por mí como persona natural y están en OneDrive personal, Copilot los clasifica como contenido altamente sensible, probablemente debido a: Lenguaje normativo o jurídico Estructura de reglamentos y protocolos Información institucional o disciplinaria Palabras clave asociadas a documentación oficial Formatos PDF/Excel con tablas, artículos o indicadores Entiendo que Copilot aplica filtros de seguridad para evitar procesar documentos que puedan contener información crítica o normativa, pero en este caso se trata de archivos destinados a publicación pública en la web de la escuela. Me gustaría saber: Si existe una forma de marcar estos documentos como seguros desde OneDrive personal. Si hay configuraciones específicas que permitan a Copilot procesar documentos normativos creados por el usuario. Si otros usuarios han enfrentado este tipo de bloqueo con contenido institucional no confidencial. Agradezco cualquier orientación o experiencia que puedan compartir. Saludos, Alex
