Transforming Enterprise Documents into AI Knowledge with Agentic RAG
TECHNICAL RESEARCH PAPER Research Disclaimer: This article is a technical research paper intended solely for educational, research, and illustrative purposes. It reflects the author’s personal technical perspectives. All concepts, architectural designs, diagrams, and implementation approaches are provided “AS IS” for technical exploration and review, not as production-ready development guidance. Organizations considering implementing these approaches should consult their internal software development standards, enterprise architecture teams, corporate security policies, and governance requirements. Any production implementation must undergo appropriate end-to-end architectural, security, privacy, compliance, and responsible AI or ethics reviews, along with necessary testing and organizational approvals before deployment. Abstract. This research paper explores three independent approaches to transforming enterprise documents and knowledge repositories into AI-powered search experiences using Agentic Retrieval-Augmented Generation (RAG), Microsoft Foundry, and ASP.NET Core. Organizations can evaluate and adopt any one approach independently according to their business requirements, organizational size, document volume, technical complexity, available resources, and security considerations. Approach 1 — Microsoft Foundry File Search: A streamlined option with managed ingestion, vector storage, retrieval, and an optional ASP.NET Core/Blazor conversational web interface. It may be a sufficient standalone solution for focused document-search needs. Approach 2 — Microsoft Foundry IQ: A knowledge-centric option built around Azure AI Search and reusable knowledge bases, suitable for applications that need more configurable retrieval or shared knowledge across authorized agents. It does not require Approach 1 to be implemented first. Approach 3 — Unified Enterprise Agentic Search: A broader design combining federated and indexed retrieval across approved enterprise repositories, for organizations with extensive integration, authorization, governance, and cross-source search requirements. These are alternatives, not mandatory sequential stages. Approaches 1 and 2 may each be complete solutions. Organizations considering Approach 3 may optionally pilot Approach 1 or 2 to assess retrieval quality, citations, user experience, and operational readiness, but neither pilot is a prerequisite. The examples are research illustrations, not production-ready implementation instructions. 1. Three independent architectural approaches Select an approach based on organizational needs rather than following a required progression: Optional evaluation path: An enterprise exploring Approach 3 can run a focused File Search or Foundry IQ POC first. A small organization may stop at Approach 1 or 2 if its requirements are met. 2. Three independent implementations 3. How documents become AI-searchable For indexed sources, the broad workflow is similar across managed File Search and Foundry IQ-backed Azure AI Search, although ingestion details, formats, and service capabilities differ. Federated retrieval may query existing repositories without creating new embeddings. Indexed document processing and agent retrieval workflow The original document is not replaced by vectors alone: searchable text and source references are retained. Scanned PDFs and images may need OCR or multimodal processing. 3.1 Example: what happens to one PDF? Source: IIS-AppPool-Troubleshooting.pdf. Example passage: “If an IIS application pool repeatedly terminates, examine WAS events and investigate rapid-fail protection.” Illustrative document chunk, vector, and metadata A query such as “Why does my IIS application pool keep crashing?” may retrieve the relevant passage through semantic similarity. The agent uses the readable retrieved content to explain the issue and cite its source. Architecture note: This paper targets the new Microsoft Foundry Agent Service (Responses API-based), not classic Foundry agents or legacy Assistants API implementations. 4. Approach 1 — Microsoft Foundry File Search 4.1 Objective Explore a potentially standalone AI-powered search solution over 20 approved technical documents with limited ingestion configuration. Evaluate one Foundry File Search-enabled agent through the built-in Playground and/or a custom conversational web application. 4.2 Architecture flow Approach 1 — Microsoft Foundry File Search architecture 4.3 What is a managed vector store? A managed vector store exposes searchable passages, embeddings, and metadata while abstracting ingestion and retrieval. In standard Foundry Agent Service setup, files can remain in customer-managed Azure Storage and Azure AI Search can back the vector store; basic setup uses Microsoft-managed resources. Where supported, multiple agents in the same project can reference the same vector store ID through File Search tool configuration, subject to API and permission boundaries. 4.4 User experience options Option 1: Use the Foundry Agent Playground to ask questions, inspect citations, and evaluate multi-document synthesis. Option 2: Build an ASP.NET Core/Blazor portal using supported APIs/SDKs with multi-turn chat, streaming where supported, clickable citations, error handling, and Microsoft Entra ID sign-in. Keep privileged credentials on the server. 4.5 Illustrative implementation and deliverables • Prepare 20 approved documents and a fixed question set. • Configure a compatible Foundry project, model, agent, and File Search tool. • Upload files; verify managed processing and retrieval quality. • Test in the Playground; optionally implement and test the Blazor portal. • Evaluate answer grounding, citations, latency, ingestion behavior, and costs. Possible standalone deliverable: File Search agent, managed vector store, working selected interface(s), and an evaluation report. The custom portal is optional; Approach 2 is not required. 5. Approach 2 — Microsoft Foundry IQ 5.1 Objective Evaluate a reusable knowledge-base architecture with Azure AI Search-backed retrieval. This is an independent option: it can be built directly without first implementing Foundry File Search. For a comparative research exercise, the same 20 documents and questions may be reused. 5.2 Architecture flow Approach 2 — Microsoft Foundry IQ architecture 5.3 What is an Azure AI Search index? An Azure AI Search index is a structured, searchable collection of content extracted from enterprise data sources, including documents, websites, and other supported repositories. It stores searchable text, vector embeddings, metadata, and source references, enabling keyword, vector, hybrid, and semantic search to retrieve relevant information efficiently. During ingestion, documents can be processed, divided into smaller chunks, and converted into vector embeddings that represent their semantic meaning. When a user submits a natural-language question, Azure AI Search can combine keyword matching with vector similarity and semantic ranking to identify relevant passages, which an AI agent can use to generate grounded responses with citations to the original sources. 5.4 What Foundry IQ adds? Microsoft Foundry IQ builds on Azure AI Search by providing reusable knowledge bases and agentic retrieval capabilities across supported enterprise data sources. Multiple authorized AI agents can access the same knowledge base without duplicating document ingestion, while agentic retrieval can intelligently plan and execute searches to identify relevant information. Supported remote SharePoint knowledge sources can also be queried through the Copilot Retrieval API without creating a separate local vector index. This capability may require eligible Microsoft 365 Copilot licensing for end users, with access governed by source-specific permissions. For indexed content, document-level security trimming depends on supported knowledge-source capabilities and properly configured access control lists (ACLs) or security filters. 5.5 User experience and optional comparison A Foundry IQ solution may use the Agent Playground and/or its own ASP.NET Core/Blazor application. If an organization is comparing Approaches 1 and 2, it can optionally reuse a common web UI and provide an agent selector. Reusing Approach 1 is not a dependency. 5.6 Illustrative implementation and deliverables • Provision compatible Azure AI Search and Foundry resources. • Create the knowledge base and supported knowledge source; validate ingestion. • Connect a Foundry agent; evaluate retrieval and citations in the Playground. • Optionally build or reuse a Blazor interface for the same agent. • Measure relevance, knowledge reuse, latency, costs, and operational complexity. Direct-file ingestion may depend on preview, region, or API availability; Azure Blob ingestion may be needed. Possible standalone deliverable: reusable knowledge base, index, connected agent, selected interface(s), and evaluation report. 6. Foundry File Search versus Foundry IQ Section 6 — Foundry File Search versus Foundry IQ Key takeaway: Both File Search and Foundry IQ can support keyword, vector, and semantic relevance, with hybrid retrieval where supported. File Search emphasizes managed processing and search behavior; Foundry IQ provides reusable knowledge bases and more configurable Azure AI Search-backed sources. The distinction is control, source integration, and operations—not whether embeddings or semantic relevance exist. Both may be standalone solutions. 7. Approach 3 — Unified Enterprise Agentic Search 7.1 Objective Explore a comprehensive architecture for organizations requiring search across multiple repositories, with both federated and indexed retrieval, grounded answers, citations, and permission-aware access. This approach can be evaluated directly; smaller POCs are optional validation exercises, not prerequisites. 7.2 Unified enterprise architecture flow Section 7 — Unified Enterprise Agentic Search architecture 7.3 Option A — Federated enterprise search Query approved repositories through source APIs/connectors while retaining documents in place. Delegated access and source-native ACLs must be enforced for each connector. Remote SharePoint retrieval through Copilot Retrieval can require Microsoft 365 Copilot licenses for end users; source-specific permission behavior must be validated. Other repositories may require custom connectors or indexing. 7.4 Option B — Indexed enterprise AI search Index approved extracted text, vectors, metadata, and source references in Azure AI Search. Foundry IQ can provide reusable knowledge retrieval. OCR and complex layout extraction are not guaranteed by default File Search or every Foundry IQ knowledge source: test each format and add Azure AI Search OCR/Document Intelligence layout skills or other processing where needed. Derived indexed content requires explicit authorization and security trimming. 7.5 Combined example For “Find documentation explaining HTTP 503 errors in IIS and compare it with internal troubleshooting procedures,” the agent could retrieve current SharePoint guidance through Option A and indexed engineering articles through Option B, synthesize findings, and cite original sources. 8. A Unified UI for All Three Approaches: ASP.NET Core / Blazor Conversational Portal Section 8 — Unified ASP.NET Core/Blazor conversational portal for all three approaches The shared Blazor UI and ASP.NET Core backend can route requests to any of the three independent approaches. Approach 3 additionally coordinates federated and indexed retrieval; all paths return grounded answers and source citations. The retrieval router shown in the diagram is custom ASP.NET Core application/BFF logic, not a standalone Microsoft product. It selects the configured File Search, Foundry IQ, or federated/indexed enterprise path and normalizes citations. The architecture targets the new Microsoft Foundry Agent Service (Responses API-based). 9. Security and governance Section 9 — Security and governance Use approved noncustomer documents for initial research. Validate source permissions and prevent restricted content disclosure. Azure AI Search does not automatically preserve original repository ACLs when content is copied into an index. Indexed security trimming requires appropriate ACL fields and identity-aware filters or supported permission integration; not every knowledge source provides user-level trimming out of the box. Customer data requires organizational approvals. 10. Independent POC evaluation and success criteria Evaluate only the approach or approaches an organization is considering. A side-by-side comparison is optional, not a required sequence. If comparing File Search and Foundry IQ, use identical documents, questions, models, and prompts where possible. Section 10 — Independent POC evaluation and success criteria Illustrative acceptance target: at least 90% correct retrieval on a curated test set, valid citations for grounded responses, and reliable operation through the selected interface(s). These are proposed research criteria, not platform guarantees. Approach 3 additionally requires source-permission, cross-repository, and governance validation. 11. Independent estimates and potential deliverables Section 11 — Independent estimates and potential deliverables These are independent, illustrative POC estimates, not a cumulative rollout schedule. The Approach 2 estimate assumes compatible resources and an existing interface or Playground-only evaluation; building a new custom application can add effort. Approach 3 needs a separately scoped plan for connectors, access control, compliance reviews, testing, deployment, and operational readiness. 12. Research findings and needs-based architectural direction Approach 1: Consider Foundry File Search where a manageable document collection and simpler managed retrieval meet the organization’s requirements. Approach 2: Consider Foundry IQ where reusable knowledge, multiple agents, or greater search configurability are important. Approach 3: Consider unified enterprise retrieval where many systems, security boundaries, and content types must be integrated. No approach is universally preferred, and organization size alone is not a definitive selection criterion. A smaller organization may need advanced governance, while a large enterprise team may only need a focused File Search solution. Optional POCs using Approach 1 or 2 can inform an Approach 3 decision without becoming mandatory prerequisites. Envisioned outcome: A needs-appropriate, citation-grounded enterprise knowledge search experience built with Microsoft technologies, subject to technical validation, organizational security and privacy review, and responsible AI governance. The architectures remain independent research options rather than an endorsed implementation sequence. 13. Implementation notes and service lifecycle Current agent runtime: Target the current Microsoft Foundry Agent Service, not classic agents. Classic agents are deprecated and scheduled for retirement on March 31, 2027. Agentic retrieval: Foundry IQ query planning uses a supported model at low/medium reasoning effort. Minimal effort skips LLM query planning; model and API support must be checked. Availability and previews: Verify regional support, RBAC, API version, and GA-versus-preview status for File Search, Foundry IQ knowledge sources, and portal features before implementation. POC schedule: The illustrative estimates assume ready resources and a working Playground or reusable UI; region, permissions, and unsupported document formats can extend delivery. Technical validation: Microsoft Learn — Foundry File Search; Foundry IQ FAQ; Agentic Retrieval; Foundry Agent Service (classic) migration; Azure AI Search OCR and Document Layout skills.
AllowAnonymous attribute not working
Hi, I have a .NET Core 10 MVC web app and I want to create views that can be viewed without requiring login. I have tried adding [AllowAnonymous] both to my controller and to individual views, but the user always gets redirected to the login screen. I am thinking that it must be due to the setup in my Program.cs. Here is my Program.cs : var builder = WebApplication.CreateBuilder(args); // Add services to the container. var connectionString = builder.Configuration.GetConnectionString("DefaultConnection") ?? throw new InvalidOperationException("Connection string 'DefaultConnection' not found."); builder.Services.AddDbContext<ApplicationDbContext>( options => options.UseSqlServer( builder.Configuration.GetConnectionString("DefaultConnection"), x => x.MigrationsAssembly("PBP"))); builder.Services.AddDatabaseDeveloperPageExceptionFilter(); builder.Services.AddIdentity<ApplicationUser, IdentityRole>(options => options.SignIn.RequireConfirmedAccount = true) .AddEntityFrameworkStores<ApplicationDbContext>() .AddDefaultUI() .AddDefaultTokenProviders(); //require this for editing syncfusion datagrid builder.Services.AddMvc().AddNewtonsoftJson(options => { options.SerializerSettings.ContractResolver = new DefaultContractResolver(); }); builder.Services.AddTransient<IEmailSender, EmailSender>(); builder.Services.Configure<AuthMessageSenderOptions>(builder.Configuration); builder.Services.AddTransient<SessionHelper>(); builder.Services.AddApplicationInsightsTelemetry(new Microsoft.ApplicationInsights.AspNetCore.Extensions.ApplicationInsightsServiceOptions { ConnectionString = builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"] }); //session //https://learn.microsoft.com/en-us/aspnet/core/fundamentals/app-state?view=aspnetcore-7.0 builder.Services.AddDistributedMemoryCache(); builder.Services.AddSession(options => { options.IdleTimeout = TimeSpan.FromSeconds(1200); options.Cookie.Name = ".PBP.Session"; options.Cookie.IsEssential = true; }); builder.Services.ConfigureApplicationCookie(options => { //options.ExpireTimeSpan = TimeSpan.FromSeconds(1800); //default is 14 days options.LoginPath = "/Identity/Account/Login"; options.SlidingExpiration = true; }); var app = builder.Build(); // Configure the HTTP request pipeline. if (app.Environment.IsDevelopment()) { app.UseMigrationsEndPoint(); } else { app.UseExceptionHandler("/Home/Error"); // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts. app.UseHsts(); } app.UseHttpsRedirection(); app.MapStaticAssets(); app.UseRouting(); app.UseAuthorization(); app.UseSession(); app.UseForwardedHeaders(new ForwardedHeadersOptions { ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto }); app.MapControllerRoute( name: "default", pattern: "{controller=Home}/{action=Index}/{id?}").WithStaticAssets(); app.MapRazorPages().WithStaticAssets(); app.Run(); Can anybody advise me on if this is the issue and if so what changes I need to make so that I can successfully use AllowAnonymous? Many thanks, Mike
It appears and error that does not exist
I am using VS 2022 Community and when i develope asp.net apps it happens sometimes that it appears an error that does not exist adn only after many hours of searching for the error or rewriting the same code the app begins to run correctly. Somehitng simmilar happened to any of you ? how can i avoid these strange behaviour of the compiler ? Thank you in advance Luis MartinConnect .Net (4.6.2) to Dataverse using the Dataverse plugin
Hi Microsoft Tech Community, We have a Dataverse environment in our tenant, and I have its URL. I'm building a .NET application and need to connect it to Dataverse for data access. From the Learn docs, ChatGPT, and Copilot, it seems I need to use an App ID (Client ID) and Client Secret in the connection string (e.g., via Microsoft.PowerPlatform.Dataverse.Client). However, the Dataverse environment itself doesn't expose any App ID or Client Secret—where would I even find those if they exist? Here's the code snippet we found online that's using OAuth with AppId and RedirectUri, but we're unsure about the app registration part: using Microsoft.Crm.Sdk.Messages; using Microsoft.PowerPlatform.Dataverse.Client; using Microsoft.Xrm.Sdk; class Program { // TODO Enter your Dataverse environment's URL and logon info. static string url = "https://yourorg.crm.dynamics.com"; static string userName = "email address removed for privacy reasons"; static string password = "yourPassword"; // This service connection string uses the info provided above. // The AppId and RedirectUri are provided for sample code testing. static string connectionString = $@" AuthType = OAuth; Url = {url}; UserName = {userName}; Password = {password}; AppId = 51f81489-12ee-4a9e-aaae-a2591f45987d; RedirectUri = app://58145B91-0C36-4500-8554-080854F2AC97; LoginPrompt=Auto; RequireNewInstance = True"; static void Main() { //ServiceClient implements IOrganizationService interface IOrganizationService service = new ServiceClient(connectionString); var response = (WhoAmIResponse)service.Execute(new WhoAmIRequest()); Console.WriteLine($"User ID is {response.UserId}."); // Pause the console so it does not close. Console.WriteLine("Press the <Enter> key to exit."); Console.ReadLine(); } } Do I need to create a separate app registration in Microsoft Entra ID (Azure AD), link it to the Dataverse environment (perhaps as an application user?), and then use that app's Client ID/Secret in my .NET code? If so, could someone outline the exact steps, including permissions and Power Platform admin center setup? Any guidance or links to official walkthroughs would be greatly appreciated—thanks!SharePoint List View Threshold Error while Fetching 5000+ Records in list.
I am fetching data from a SharePoint list that contains more than 5000 records inside my function. While retrieving the data, I am getting the following error: “Flow Reminders→ The attempted operation is prohibited because it exceeds the list view threshold.” Scenario: The SharePoint list has 5000+ items. Data is being fetched programmatically. The error occurs during query execution. What I need help with What is the recommended and best-practice approach to retrieve large datasets from a SharePoint list without hitting the list view threshold? Questions: What is the recommended approach to fetch large data from a SharePoint list without hitting the list view threshold? Any suggestions, examples, or documentation references would be greatly appreciated. Thanks.
Debugging Java script in old asp core app
Hy, I need to modify an asp.net.core app witch I made years ago. in version 2.2. I do succeed to alter C# code in visual studio (2022), but I need to alter also some Java script. The problem is that if I try to do so and I want to debug it it seems that the script in my browser (MS Edge) is not changing. I have enabled Java script debugging in VS. I emptied the browsers cache but nothing seems to help. When I run the application in debugging mode and hit F12 I still see the old script. Please help!Getting a ConnectionString empty error in a webapi project with .net 10
I'm running in .NET 10 with the most recent update for everything. I'm in EF. I'm ASP.NET Core. Below is my program.cs file. Inside of the program.cs file, when I call to get my connection string, I get back what I expect. When I hit the LoginController, I see the settings in the IConfiuguration that is handed in. However, when I set a breakpoint in the FSMUserStore constructor, I check the context that is handed in, and it's connection string is not set correctly (at all). I've run through Visual Studio debugger so many times, and tried so many things, I'm just lost on this. I think the problem is in how I am setting up for dependency injection in the program.cs file, but I've tried so many things that I don't know what else to try. Anyone have a suggestion? using ElmahCore.Mvc; using Microsoft.AspNetCore.Authentication.JwtBearer; using Microsoft.AspNetCore.Identity; using Microsoft.EntityFrameworkCore; using Microsoft.IdentityModel.Tokens; using Portal.Libraries; using PortalDataModels.Models; using System.Text; using Twilio.TwiML.Voice; var builder = WebApplication.CreateBuilder(args); IConfiguration configuration = new ConfigurationBuilder() .AddJsonFile("appsettings.json") .AddEnvironmentVariables() .Build(); // connString get's the correct value var connString = configuration.GetConnectionString("DefaultConnection"); // Add services to the container. // For Entity Framework builder.Services.AddDbContext<PortalDataModels.Models.POA_CSMContext>(options => options.UseSqlServer(configuration.GetConnectionString(connString))); builder.Services.AddScoped<IUserStore<FSMUser>, FSMUserStore>(); builder.Services.AddIdentity<FSMUser, IdentityRole>() .AddEntityFrameworkStores<POA_CSMContext>() .AddUserManager<UserManager<FSMUser>>() .AddSignInManager<SignInManager<FSMUser>>() .AddDefaultTokenProviders(); builder.Services.AddElmah<ElmahCore.Sql.SqlErrorLog>(options => { options.ConnectionString = connString; options.SqlServerDatabaseTableName = "Elmah_Error"; //Defaults to ELMAH_Error if not set options.OnPermissionCheck = context => false; }); // Adding Authentication builder.Services.AddAuthentication(options => { options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme; options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme; options.DefaultScheme = JwtBearerDefaults.AuthenticationScheme; }) // Adding Jwt Bearer .AddJwtBearer(options => { options.SaveToken = true; options.RequireHttpsMetadata = false; options.MapInboundClaims = false; options.TokenValidationParameters = new TokenValidationParameters() { ValidateIssuer = true, ValidateAudience = true, ValidAudience = configuration["Jwt:ValidAudience"], ValidIssuer = configuration["Jwt:ValidIssuer"], IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(configuration["Jwt:Key"])) }; }); builder.Services.AddScoped<UserManager<FSMUser>>(); builder.Services.AddControllers(); // Learn more about configuring OpenAPI at https://aka.ms/aspnet/openapi builder.Services.AddOpenApi(); var app = builder.Build(); // Configure the HTTP request pipeline. if (app.Environment.IsDevelopment()) { app.MapOpenApi(); } app.UseHttpsRedirection(); app.UseAuthentication(); app.UseAuthorization(); app.MapControllers(); app.Run();When should I use SQL Server, MySQL, or PostgreSQL?
I'm working on a web application (using C#/.NET backend), and I need to choose a relational database. I'm considering three options: Microsoft SQL Server MySQL PostgreSQL I want to understand: What are the strengths and weaknesses of each? Which database is better suited for: Transactional performance Advanced querying / analytics Ease of use with .NET Cross-platform support Are there any licensing or hosting concerns I should consider? My use case involves: Moderate traffic Complex queries with joins APIs and background jobs Deployment to cloud (Azure) I’d appreciate any recommendations or real-world experience comparing these databases.Is Native RDLC Report Support Planned for Future .NET Versions?
Good day Team, Given the demand for cross-platform reporting solutions, is there any plan to provide native RDLC (Report Definition Language Client-side) support for .NET Core or upcoming .NET releases (e.g., .NET 9 and beyond)? Many enterprise applications rely on RDLC, and the current lack of official support limits modernization efforts and forces continued reliance on the legacy .NET Framework or Windows-only workarounds. Is RDLC on the roadmap for future .NET versions, or should we consider alternative technologies for long-term reporting requirements? Any official guidance or updates would be greatly appreciated.
