Accelerate Agent Development: Hacks for Building with Microsoft Sentinel data lake

As a Senior Product Manager | Developer Architect on the App Assure team working to bring Microsoft Sentinel and Security Copilot solutions to market, I interact with many ISVs building agents on Microsoft Sentinel data lake for the first time. I’ve written this article to walk you through one possible approach for agent development – the process I use when building sample agents internally at Microsoft. If you have questions about this, or other methods for building your agent, App Assure offers guidance through our Sentinel Advisory Service. 

Throughout this post, I include screenshots and examples from Gigamon’s Security Posture Insight Agent. 

 This article assumes you have: 

• An existing SaaS or security product with accessible telemetry.
• A small ISV team (2–3 engineers + 1 PM).
• Focus on a single high value scenario for the first agent. 

The Composite Application Model (What You Are Building)

When I begin designing an agent, I think end-to-end, from data ingestion requirements through agentic logic, following the Composite application model.

The Composite Application Model consists of five layers: 

1. Data Sources – Your product’s raw security, audit, or operational data. 
2. Ingestion – Getting that data into Microsoft Sentinel.
3. Sentinel data lake & Microsoft Graph – Normalization, storage, and correlation.
4. Agent – Reasoning logic that queries data and produces outcomes.
5. End User – Security Copilot or SaaS experiences that invoke the agent.

This separation allows for evolving data ingestion and agent logic simultaneously. It also helps avoid downstream surprises that require going back and rearchitecting the entire solution. 

Optional Prerequisite

You are enrolled in the ISV Success Program, so you can earn Azure Credits to provision Security Compute Units (SCUs) for Security Copilot Agents. 

Phase 1: Data Ingestion Design & Implementation 

Choose Your Ingestion Strategy 

The first choice I face when designing an agent is how the data is going to flow into my Sentinel workspace. Below I document two primary methods for ingestion.  

• Option A: Codeless Connector Framework (CCF) 
  • This is the best option for ISVs with REST APIs. To build a CCF solution, reference our documentation for getting started. 
• Option B: CCF Push (Public Preview) 
  • In this instance, an ISV pushes events directly to Sentinel via a CCF Push connector. Our MS Learn documentation is a great place to get started using this method.

Additional Note:

• In the event you find that CCF does not support your needs, reach out to App Assure so we can capture your requirements for future consideration. Azure Functions remains an option if you’ve documented your CCF feature needs.

Phase 2: Onboard to Microsoft Sentinel data lake 

Once my data is flowing into Sentinel, I onboard a single Sentinel workspace to data lake. This is a one-time action and cannot be repeated for additional workspaces. 

Onboarding Steps 

1. Go to the Defender portal. 
2. Follow the Sentinel Data lake onboarding instructions. 
3. Validate that tables are visible in the lake.
4. See Running KQL Queries in data lake for additional information.  

Phase 3: Build and Test the Agent in Microsoft Foundry

Once my data is successfully ingested into data lake, I begin the agent development process. There are multiple ways to build agents depending on your needs and tooling preferences. For this example, I chose Microsoft Foundry because it fit my needs for real-time logging, cost efficiency, and greater control. 

1. Create a Microsoft Foundry Instance 

Foundry is used as a tool for your development environment. Reference our QuickStart guide for setting up your Foundry instance.

Required Permissions: 

• Security Reader (Entra or Subscription)
• Azure AI Developer at the resource group

After setup, click Create Agent.

 

2. Design the Agent 

     A strong first agent: 

• Solves one narrow security problem.
• Has deterministic outputs.
• Uses explicit instructions, not vague prompts.

     Example agent responsibilities: 

• To query Sentinel data lake (Sentinel data exploration tool).
• To summarize recent incidents.
• To correlate ISVs specific signals with Sentinel alerts and other ISV tables (Sentinel data exploration tool).

3. Implement Agent Instructions

     Well-designed agent instructions should include: 

• Role definition ("You are a security investigation agent…").
• Data sources it can access.
• Step by step reasoning rules.
• Output format expectations.

Sample Instructions can be found here: Agent Instructions

 

4. Configure the Microsoft Model Context Protocol (MCP) tooling for your agent

For your agent to query, summarize and correlate all the data your connector has sent to data lake, take the following steps: 

• Select Tools, and under Catalog, type Sentinel, and then select Microsoft Sentinel Data Exploration. For more information about the data exploration tool collection in MCP server, see our documentation.

• I always test repeatedly with real data until outputs are consistent. For more information on testing and validating the agent, please reference our documentation. 

Phase 4: Migrate the Agent to Security Copilot

Once the agent works in Foundry, I migrate it to Security Copilot. To do this: 

1. Copy the full instruction set from Foundry 
2. Provision a SCU for your Security Copilot workspace. For instructions, please reference this documentation. 
   • Make note of this process as you will be charged per hour per SCU 
   • Once you are done testing you will need to deprovision the capacity to prevent additional charges 
3. Open Security Copilot and use Create From Scratch Agent Builder as outlined here. 
4. Add Sentinel data exploration MCP tools (these are the same instructions from the Foundry agent in the previous step). 
   • For more information on linking the Sentinel MCP tools, please refer to this article.
5. Paste and adapt instructions.

At this stage, I always validate the following: 

• Agent Permissions – I have confirmed the agent has the necessary permissions to interact with the MCP tool and read data from your data lake instance.  
• Agent Performance – I have confirmed a successful interaction with measured latency and benchmark results. 

This step intentionally avoids reimplementation. I am reusing proven logic. 

Phase 5: Execute, Validate, and Publish 

After setting up my agent, I navigate to the Agents tab to manually trigger the agent. For more information on testing an agent you can refer to this article.

Now that the agent has been executed successfully, I download the agent Manifest file from the environment so that it can be packaged. 

• Click View code on the Agent under the Build tab as outlined in this documentation. 

Publishing to the Microsoft Security Store

If I were publishing my agent to the Microsoft Security Store, these are the steps I would follow:  

1. Finalize ingestion reliability. 
2. Document required permissions. 
3. Define supported scenarios clearly. 
4. Package agent instructions and guidance (by following these instructions). 

Summary 

Based on my experience developing Security Copilot agents on Microsoft Sentinel data lake, this playbook provides a practical, repeatable framework for ISVs to accelerate their agent development and delivery while maintaining high standards of quality. This foundation enables rapid iteration—future agents can often be built in days, not weeks, by reusing the same ingestion and data lake setup. 

When starting on your own agent development journey, keep the following in mind: 

• To limit initial scope. 
• To reuse Microsoft managed infrastructure. 
• To separate ingestion from intelligence. 

What Success Looks Like 

At the end of this development process, you will have the following: 

• A Microsoft Sentinel data connector live in Content Hub (or in process) that provides a data ingestion path.  
• Data visible in data lake.  
• A tested agent running in Security Copilot. 
• Clear documentation for customers. 

A key success factor I look for is clarity over completeness. A focused agent is far more likely to be adopted. 

Need help?  

If you have any issues as you work to develop your agent, please reach out to the App Assure team for support via our Sentinel Advisory Service . Or if you have any other tips, please comment below, I’d love to hear your feedback.