Securing Enterprise AI Agents with Microsoft Sentinel
1. Introduction Enterprise adoption of Generative AI is accelerating rapidly through Microsoft 365 Copilot, Copilot Studio, Azure AI Foundry Agents, Security Copilot, and custom AI agents integrated with business applications. Unlike traditional SaaS applications, AI agents can: Access enterprise data Query internal knowledge repositories Invoke APIs and MCP tools Execute workflows Interact with business applications Make decisions on behalf of users While these capabilities improve productivity, they introduce a new attack surface that security teams must monitor and secure. Common AI threats include: Prompt Injection Cross Prompt Injection Attacks (XPIA) Jailbreak Attempts Unauthorized Tool Invocation Data Exfiltration through AI Agents Agent Identity Abuse Excessive Data Access Malicious MCP Tool Execution Traditional SOC monitoring platforms were designed for users, devices, applications and infrastructure—not autonomous AI systems. To address this challenge, Microsoft provides a comprehensive AI security monitoring framework built around: Agent 365 Observability Microsoft Agent Identities Microsoft Copilot Logs Defender XDR Defender for AI Microsoft Sentinel Together these components provide end-to-end observability of: User prompts Agent execution paths Tool invocations Safety signal detections Agent identities Security alerts 2. Reference Architecture AI Security Monitoring Architecture 3. Integration Architecture Microsoft provides multiple telemetry sources that complement one another. 3.1 Agent Runtime Telemetry Sentinel Data Connector Agent 365 Data Connector Table UnifiedAgentObservability Captures runtime behavior of AI agents including: User prompts Session IDs Conversation IDs Agent identities MCP tool invocations Connector invocations Tool arguments Tool responses Request payloads Response payloads Execution errors This dataset provides the forensic trail of everything an AI agent performed. 3.2 Agent Governance and Asset Inventory Sentinel Data Connector Microsoft Agent Identities Provides visibility into: Agent inventory Agent blueprint inventory Ownership Relationships Governance metadata Risk context This allows SOC teams to answer: Who owns this agent? What permissions does it have? Which business unit deployed it? Which related agents exist? 3.3 Copilot Audit and Usage Monitoring Sentinel Data Connector Microsoft Copilot Logs Connector Table CopilotActivity Provides: Copilot usage auditing Operational visibility User interaction tracking Useful for governance, compliance and adoption reporting. 3.4 AI Safety Telemetry Sentinel Data Connector Microsoft Defender XDR Connector Table CloudAppEvents CloudAppEvents provides AI safety signals such as: Prompt Shield detections Prompt Injection attempts Cross Prompt Injection Attacks (XPIA) Jailbreak-related verdicts Unsafe prompt classifications Think of CloudAppEvents as answering: "Was the prompt malicious?" 3.5 AI Security Alerts Sentinel Data Connectors Microsoft Defender XDR Microsoft Defender for Cloud Tables SecurityAlert SecurityIncident Used for: AI attack detections Security incidents Correlated investigation workflows 4. Understanding the Two Most Important AI Tables CloudAppEvents Focuses on AI Safety Questions answered: Was Prompt Shield triggered? Was this a jailbreak attempt? Was XPIA detected? Was the prompt suspicious? UnifiedAgentObservability Focuses on Agent Runtime Behavior Questions answered: What tool was invoked? Which connector executed? What arguments were passed? What data was returned? What actions did the agent perform? 5. Advanced Threat Hunting Scenarios The Agent365 Observability hunting guide contains several investigation scenarios that can be used directly in Microsoft Sentinel. Reference: Agent 365 Observability — AI Agent Telemetry Hunting https://github.com/SCStelz/security-investigator/blob/main/queries/cloud/agent365_observability.md 5.1 Prompt Injection Detection Detect prompts containing indicators such as: Ignore previous instructions Reveal system prompt Developer mode Disregard safety controls Investigation workflow: Review Tool Activity This allows analysts to determine whether a suspicious prompt resulted in downstream actions. 5.2 Session Reconstruction One of the most powerful capabilities of UnifiedAgentObservability is session reconstruction. Analysts can correlate: This creates complete forensic timelines. 5.3 MCP Tool Auditing Monitor all MCP activity including: query_lake Graph API tools ServiceNow connectors SharePoint connectors Custom enterprise tools Questions answered: Which tool was used? Who triggered it? What parameters were supplied? What data was returned? 5.4 Sensitive Data Access Monitoring Monitor AI agent interaction with: Employee records Customer data Financial information SharePoint repositories HR databases Useful for identifying: Data exfiltration attempts Excessive access patterns Sensitive data exposure 5.5 Query Lake Monitoring The GitHub hunting guide introduces monitoring of: query_lake RunAdvancedHuntingQuery Analysts can inspect: Actual KQL submitted Target workspaces Data sources queried Scope of access This provides visibility into AI-driven security investigations. 5.6 New Tool Detection Identify newly observed tool usage. Examples: Unauthorized MCP servers Newly registered connectors Unapproved tools Unexpected integrations This use case is particularly useful for governance programs. 5.7 Tool Failure Monitoring Monitor: Permission failures Connector failures Application errors Access-denied responses A sudden increase in failures may indicate: Reconnaissance activity Misconfiguration Privilege abuse attempts 6. Detection Engineering Opportunities Organizations can create Sentinel Analytics Rules for: 6.1 Prompt Injection Detection Developer Mode prompts Prompt Override attempts System Prompt disclosure requests 6.2 Jailbreak Attempt Detection Safety bypass attempts Role manipulation prompts Instruction override patterns 6.3 Unauthorized Tool Usage New MCP tools High-risk connectors Rare tool executions 6.4 Sensitive Data Access HR data queries Identity information retrieval Large-volume exports 6.5 Agent Identity Abuse Ownership changes Unexpected agent activity Agent-to-agent anomalies 7. Data Lake Exploration and Long-Term Analytics Because agent telemetry resides within Sentinel Data Lake, organizations can perform: Long-term AI investigations Historical AI attack analysis Agent baselining Governance reporting Trend analysis Tool inventory reporting Example dashboards include: Top Prompt Injection Attempts Most Active Agents High-Risk MCP Tools Agent Ownership Analysis AI Security Incidents Sensitive Data Access Trends 8. Summary AI agents represent the next major computing platform, but they also introduce a completely new attack surface. To effectively secure enterprise AI solutions, organizations require visibility across: User interactions Agent execution paths MCP tool usage Prompt safety signals Agent identities Security detections Microsoft Sentinel provides this unified view by integrating: Agent 365 Observability UnifiedAgentObservability Microsoft Agent Identities Microsoft Copilot Logs CloudAppEvents Defender XDR Defender for AI By combining AI runtime telemetry with AI safety signals and Defender detections, security teams can move beyond traditional monitoring and build a modern SOC capability for threat hunting, incident response, governance and forensic investigations across Microsoft 365 Copilot, Copilot Studio, Azure AI Foundry and future AI agent ecosystems. Reference: https://github.com/SCStelz/security-investigator/blob/main/queries/cloud/agent365_observability.md
Ingesting Logs through Azure Private Link
Hi, We are currently using Azure Private Link within our environment and we are attempting to ingest logs into Log Analytics. When I reached out to Microsoft Support, it appears that the CCF connectors will not work using Private Link and the Azure Functions connectors are becoming depricated. Has anyone else run into this issue and what is the solution for getting logs into Sentinel through the Private Link, specifically API log sources? Did this require a custom app for each of these log sources or some sort of custom script that lives on an AMA host within the Private Link to ingest the logs? Any advice here would be greatly appeciated. Thank you,Security Copilot RBAC for Embedded Experience in Unified Security Platform
Introduction The evolution of Security Operations Centers (SOC) is increasingly driven by AI-powered capabilities that improve efficiency, accuracy, and response time. Microsoft Security Copilot represents a significant advancement in this space by embedding AI-driven assistance directly within security platforms such as Microsoft Defender XDR, Microsoft Sentinel, and Microsoft Entra. The concept of embedded experience is central to this transformation. Rather than operating as a standalone interface, Security Copilot is integrated within existing security tools, allowing analysts to invoke AI-generated insights directly during investigations. This reduces the need for tool switching and accelerates decision-making. The purpose of this document is to define and explain the Role-Based Access Control (RBAC) model required to securely enable this embedded experience. It provides a structured understanding of how access is governed across multiple layers, how these layers interact, and how organizations can align permissions with SOC workflows while maintaining a least-privilege security posture. Understanding Embedded Experience Security Copilot in embedded mode operates within the context of the host platform. When invoked from Defender or Sentinel, it does not function independently but instead consumes data already accessible to the user. This model ensures that Copilot enhances visibility without expanding access boundaries. This behavior is governed by an On-Behalf-Of (OBO) model, where Security Copilot leverages the permissions of the authenticated user. It does not introduce new entitlements or override existing RBAC configurations. As a result, the insights generated by Copilot are always limited to what the user is already authorized to see, reinforcing Zero Trust principles and preventing unauthorized data exposure. Prerequisites for Embedded Experience To enable Security Copilot in an embedded environment, organizations must establish foundational prerequisites that ensure seamless and secure operation. First, access to underlying platforms such as Microsoft Defender XDR, Microsoft Sentinel, and Microsoft Entra must already be provisioned. Since Copilot is not a standalone data source, it cannot function without these integrations. Second, RBAC alignment across identity, platform, and service layers must be configured correctly. Misalignment can lead to incomplete results, restricted functionality, or inconsistent analyst experiences. Finally, governance processes such as access review, monitoring, and adherence to least privilege principles should be implemented. These controls ensure that Copilot usage remains compliant, auditable, and aligned with organizational security policies. RBAC Framework for Security Copilot Security Copilot adopts a multi-layer RBAC model consisting of three tightly integrated layers. These layers collectively determine whether a user can access Copilot features and what data they can retrieve. RBAC Layer Mapping RBAC Layer Role Type Purpose Example Roles Access Impact Security Copilot Platform Feature access control Determines who can use Copilot capabilities Security Copilot Owner, Security Copilot Contributor Enables use of Copilot features but does not grant data access Microsoft Entra ID Identity and directory governance Controls access to identity data and reports Security Reader, Reports Reader, Security Administrator Governs identity insights and directory visibility Service-Specific RBAC Data access control Defines access to security data within services Defender Security Reader, Sentinel Reader Determines what Copilot can retrieve and present This layered approach ensures that no single role grants full access. All three layers must align for complete functionality. Security Copilot Platform Roles Security Copilot platform roles control who can interact with the Copilot interface and execute AI-driven workflows. The Security Copilot Owner role provides administrative control over Copilot configuration, including access management and platform-level settings. This role is typically assigned to administrators responsible for governance and operational enablement. The Security Copilot Contributor role enables analysts to run prompts, perform investigations, and interact with Copilot features during daily SOC operations. However, this role does not grant visibility into security data by itself. This clear separation ensures that Copilot remains a controlled interface layer rather than a source of privilege escalation. Microsoft Entra ID Roles Microsoft Entra roles govern access to identity-related data, which is critical for security operations involving user behavior, sign-in logs, and directory insights. Roles such as Security Reader provide read-only visibility into security data, while Reports Reader enables access to reporting and analytics capabilities. In certain advanced cases, the Security Administrator role may be required for configuration-level actions. The document emphasizes avoiding excessive privilege assignment, particularly the use of Global Administrator roles for daily operations, as this conflicts with least privilege principles. Service-Specific RBAC Roles Service-level roles determine the data sources that Security Copilot can access when embedded in platforms. In Microsoft Defender XDR, roles such as Security Reader allow access to alerts, incidents, and endpoint data. In Microsoft Sentinel, Sentinel Reader provides access to log data, analytics, and incidents. In Microsoft Entra, roles like Reports Reader provide access to identity insights. Copilot cannot retrieve or analyze data beyond what these roles permit. The output it generates is always constrained to the user’s effective permissions across these services. Unified RBAC Behavior in Embedded Experience In an embedded scenario, all three RBAC layers are evaluated simultaneously. When a SOC analyst invokes Copilot in Defender, the system validates whether the user has permission to use Copilot, access identity data, and retrieve Defender-specific insights. Only when all these conditions are satisfied does Copilot provide a comprehensive output. This ensures that Copilot responses are both contextually rich and access-compliant, eliminating the risk of unauthorized data exposure while maintaining operational efficiency. Security Copilot Core Use Cases Security Copilot enables a layered set of capabilities that span both analyst interaction patterns and agent-driven execution models. These use cases collectively enhance SOC efficiency, decision-making, and operational scalability. Use Case Mapping Table Use Case Description Embedded / Agent Example Value to SOC Summarization Transforms complex alerts, incidents, and telemetry into structured, human-readable insights by correlating signals across multiple sources Summarizing a Defender XDR incident involving endpoint, identity, and cloud alerts into a unified attack narrative Reduces analyst fatigue and significantly accelerates triage by eliminating manual data aggregation Guided Response Provides contextual, step-by-step investigative guidance and recommended remediation actions based on observed patterns and threat intelligence Suggesting investigation paths in Sentinel, including pivoting to identity logs, device timeline, and lateral movement indicators Improves consistency in investigations and enables less experienced analysts to operate effectively Script Analysis Evaluates scripts, queries, and command-line activities to identify malicious patterns, errors, or optimization opportunities Analyzing PowerShell scripts or KQL queries used in threat hunting scenarios to detect obfuscation or suspicious logic Enhances detection accuracy and reduces the risk of missing critical indicators Reporting Generates structured incident summaries, executive reports, and compliance-ready documentation with contextual insights Producing incident summaries for leadership or compliance teams with both technical and business context Improves communication, supports audit readiness, and reduces manual reporting overhead Agent-Driven SOC Use Cases (Expanded Capabilities) With the introduction of Security Copilot agents, the platform extends beyond assistance into orchestrated, intelligence-driven operations across SOC workflows. Agent-Based Use Case Description Real Agent Example SOC Impact Dynamic Threat Detection Continuously analyzes telemetry to identify previously undetected or weak signals across the attack surface Dynamic Threat Detection Agent correlates signals across Defender workload telemetry to surface hidden threats Improves detection coverage and reduces the likelihood of missed attacks Threat Intelligence Correlation & Briefing Aggregates internal and external intelligence sources to generate contextual threat insights aligned to organizational risk Threat Intelligence Briefing Agent produces structured intelligence reports based on attack patterns and exposure context Enhances situational awareness and supports proactive defense strategies Advanced Threat Hunting Enables hypothesis-driven and AI-assisted threat hunting by generating queries, exploring telemetry, and correlating historical data Advanced Threat Hunting Agent builds and executes queries across Defender and Sentinel datasets for proactive investigation and telemetry exploration Accelerates threat discovery and reduces reliance on manual query development Security Analysis & Threat Prioritization Performs AI-driven analysis of security telemetry to identify high-risk patterns, prioritize threats, assess risk exposure, and recommend investigative actions Security Analyst Agent analyses password spray attacks, ransomware activity, malware campaigns, identity abuse, and other security risks by generating telemetry-driven assessments and recommendations Improves analyst productivity, prioritizes high-impact threats, and enables faster decision making Security Triage Automation Automates alert prioritization and classification by adding contextual enrichment and reducing noise Security Triage Agent / Phishing Triage Agent evaluates alerts and distinguishes between real threats and false positives Reduces alert fatigue and improves prioritization accuracy in high-volume environments End-to-End Investigation Orchestration Performs multi-step investigation by gathering signals, correlating activity, and building attack timelines Security Analyst Agent investigates incidents across identity, endpoint, email, cloud, and data signals to produce a consolidated incident narrative Reduces Mean Time to Investigate (MTTI) and ensures consistent investigation outcomes Cross-Domain Threat Correlation Connects signals across identity, endpoint, cloud, email, and data domains to identify multi-stage attack chains Agents operating across Defender, Entra, Sentinel, and Security Copilot correlate activities such as phishing leading to identity compromise and lateral movement Breaks down silos and enables holistic threat visibility across the environment Remediation & Response Enablement Identifies vulnerable assets and supports remediation workflows through contextual recommendations Agents integrated with endpoint and policy systems suggest patching actions, containment actions, and configuration changes based on detected risks Improves response effectiveness and strengthens overall security posture Each of these use cases operates within the RBAC boundaries defined earlier, ensuring secure and context-aware outputs. Mapping Use Cases to SOC Processes The four core use cases align directly with SOC operational stages, enabling a consistent and repeatable analysis model. Summarization plays a significant role during the detection and triage phase, where analysts need quick clarity on incoming alerts. Instead of manually analyzing raw data, Copilot provides a structured overview, helping analysts determine priority and relevance. Guided response becomes critical during the investigation and response phase, where decision-making speed is essential. By suggesting next steps and correlating data points, Copilot assists analysts in navigating complex attack scenarios. Script analysis supports both threat hunting and investigation, allowing analysts to validate scripts, queries, or automation logic. This reduces the risk of overlooking malicious behavior embedded in scripts. Reporting aligns with the post-incident and compliance phase, where structured documentation is required. Copilot generates summaries that can be shared with leadership or compliance teams, ensuring clarity and consistency. Together, these use cases create a continuous cycle of detection, investigation, response, and reporting, fully integrated with SOC workflows. Summary Security Copilot’s embedded experience represents a transformative shift in how AI is integrated into security operations. By embedding intelligence directly within platforms such as Defender and Sentinel, it enhances analyst productivity while maintaining strict governance controls. The three-layer RBAC model, consisting of Security Copilot roles, Microsoft Entra roles, and service-specific roles, ensures that access is both secure and compliant with least privilege principles. The On-Behalf-Of model further guarantees that Copilot does not expand access beyond existing permissions. The inclusion of structured use cases such as summarization, guided response, script analysis, and reporting enables organizations to operationalize Copilot effectively across SOC processes. When RBAC is properly aligned and integrated with SOC workflows, Security Copilot becomes a powerful enabler of faster investigations, improved accuracy, and enhanced security posture—all while maintaining strict control over data access and governance.Splitting single-tenant Microsoft Defender XDR Sentinel logs in multiple company scenarios
This article describes a simple, yet effective solution for the problem of segregating Microsoft Defender XDR and Entra ID Sentinel logs ingestion in a single-tenant with multiple companies scenario, leveraging Log Analytics workspace transformations and some simple KQL query statements.“Automating Export of Microsoft Sentinel Analytic Rules mapped to MITRE Tactics & Techniques”
1. Introduction In modern Security Operations Centers (SOCs), mapping detections to the MITRE ATT&CK framework is critical. MITRE ATT&CK provides a structured, globally recognized model of adversary behavior, categorized into Tactics (goals) and Techniques (methods). Microsoft Sentinel analytic rules frequently include MITRE mappings, but viewing or exporting these at scale isn’t straightforward within the portal. Security teams often need: • A centralized view of existing detections mapped to MITRE ATT&CK • A CSV export for reporting, audits, and threat coverage assessments • Insights for SOC maturity, gap analysis, and threat-informed defense Having an automated way to extract this information ensures accuracy, consistency, and faster operational insights — all essential for high-performing SOCs. 2) Why This Script Is Required While Sentinel analytic rules individually display MITRE mappings, organizations typically need a workspace-wide export for: Detection Coverage & Gap Analysis Identify which Tactics & Techniques are covered. Highlight missing ATT&CK areas. Support threat modelling or purple team exercises. Security Operations Reporting Governance and oversight meetings Compliance documentation SOC KPI reporting and dashboards. Detection Engineering Lifecycle Maintaining a detection catalogue. Versioning and documentation Supporting change management and audits Exporting rules into a CSV using automation avoids manual errors, saves analyst time, and ensures accurate, up-to-date data. 3) Script to Export Microsoft Sentinel Analytic Rules with MITRE Tactics & Techniques (TO BE RUN FROM AZURE CLI) Important: This Bash script must be executed from Azure CLI, either: Azure Cloud Shell, or A workstation/server with Azure CLI installed (Linux, macOS, or WSL on Windows) The script uses az rest and jq to pull analytic rules and generate a CSV containing MITRE Tactics, Techniques, Severity, Enabled state, and KQL query. Bash Script (Run in Azure CLI) ------------------------------------------------------------------------------------------------------------------------ #!/bin/bash # Variables export SUB="99005f96-e572-4035-b476-836fd9d83d64" export RG="CyberSOC" export WS="CyberSOC" API="2024-03-01" # Step 1: Set subscription az account set --subscription "$SUB" # Step 2: Fetch alert rules echo "Fetching alert rules..." az rest \ --method GET \ --uri "https://management.azure.com/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.OperationalInsights/workspaces/$WS/providers/Microsoft.SecurityInsights/alertRules?api-version=$API" \ --output json > rules.json # Step 3: Validate file if [ ! -s rules.json ]; then echo "Error: rules.json is empty or missing." exit 1 fi echo "Total rules found:" jq '.value | length' rules.json # Step 4: Generate CSV with MITRE mapping, severity, enabled echo "Generating CSV..." jq -r ' (["RuleName","Tactics","Techniques","MITRE_Map","Severity","Enabled","Query"] | @csv), ( .value[] | select(.kind == "Scheduled") | . as $r | ($r.properties // {}) as $p | [ ($p.displayName // $r.name // "N/A"), (( $p.tactics // $p.attackTactics // [] ) | join(";")), (( $p.techniques // $p.attackTechniqueIds // [] ) | join(";")), ( ( ($p.tactics // []) | map("TA" + (.[2:]? // "")) ) as $tacs | ( ($p.techniques // []) | map(.) ) as $techs | ( [$tacs[], $techs[]] | join(";")) ), ($p.severity // "N/A"), ($p.enabled | tostring), (( $p.query // "" ) | gsub("\\r?\\n"; " ")) ] | @csv ) ' rules.json > Scheduled_Rules_TTP_Query.csv ------------------------------------------------------------------------------------------------------------------------ 4) Summary Mapping detections to MITRE ATT&CK is a cornerstone of threat-informed defense. This script simplifies the process of exporting Microsoft Sentinel analytic rules with their MITRE mappings into a CSV — enabling SOC teams to: Analyze coverage across ATT&CK Identify detection gaps. Strengthen red–blue team collaboration. Build dashboards and ATT&CK heatmaps. Enhance SOC governance & reporting. Automating this export ensures faster insights, reduces manual workload, and supports a mature detection engineering program.Optimizing Microsoft Sentinel: Resolving AMA-Induced Syslog & CEF Duplicates
2) Recommended Solutions When collecting both Syslog and CEF logs from the same Linux collector using the Azure Monitor Agent (AMA) in Microsoft Sentinel, duplicate log entries can occur. These duplicates arise because the same event may be ingested through both the Syslog and CEF pipelines, leading to redundancy in the Log Analytics Workspace (LAW). The following solutions aim to eliminate or reduce duplicate log ingestion, ensuring that: CEF events are parsed correctly and only once. Syslog data remains clean and non-redundant. Storage and analytics efficiency is improved. Alerting and incident investigation are not skewed by duplicate entries. Each option provides a different strategy based on your environment’s flexibility and configuration capabilities—from facility-level separation, to ingestion-time filtering, to daemon-side log routing. Option 1: Facility Separation (Preferred) Configure devices to emit CEF logs on a dedicated facility (for example, 'local4'), and adjust the Data Collection Rules (DCRs) so that the CEF stream includes only that facility, while the Syslog stream excludes it. This ensures CEF events are parsed once into 'CommonSecurityLog' and never land in 'Syslog'. CEF via AMA DCR (include only CEF facility): { "properties": { "dataSources": { "syslog": [ { "streams": ["Microsoft-CommonSecurityLog"], "facilityNames": ["local4"], "logLevels": ["*"], "name": "cefDataSource" } ] }, "dataFlows": [ { "streams": ["Microsoft-CommonSecurityLog"], "destinations": ["laDest"] } ] } } Syslog via AMA DCR (exclude CEF facility): { "properties": { "dataSources": { "syslog": [ { "streams": ["Microsoft-Syslog"], "facilityNames": [ "auth","authpriv","cron","daemon","kern","mail", "syslog","user","local0","local1","local2","local3", "local5","local6","local7" ], "logLevels": ["*"], "name": "syslogDataSource" } ] }, "dataFlows": [ { "streams": ["Microsoft-Syslog"], "destinations": ["laDest"] } ] } } Option 2: Ingest-time Transform (Drop CEF from Syslog) If facility separation is not feasible, apply a transformation to the Syslog stream in the DCR so that any CEF-formatted messages are dropped during ingestion. Syslog stream transformKql: { "properties": { "dataFlows": [ { "streams": ["Microsoft-Syslog"], "transformKql": "source | where not(SyslogMessage startswith 'CEF:')", "destinations": ["laDest"] } ] } } Option 3: Daemon-side Filtering/Rewriting (rsyslog/syslog-ng) Filter or rewrite CEF messages before AMA sees them. For example, route CEF messages to a dedicated facility using syslog-ng and stop further processing: # Match CEF filter f_cef { message("^CEF:"); }; # Send CEF to local5 and stop further processing log { source(s_src); filter(f_cef); rewrite { set_facility(local5); }; destination(d_azure_mdsd); flags(final); } 3) Verification Steps with KQL Queries Detect CEF messages that leaked into Syslog: Syslog | where TimeGenerated > ago(1d) | where SyslogMessage startswith "CEF:" | summarize count() by Computer | order by count_ desc Estimate duplicate count across Syslog and CommonSecurityLog: let sys = Syslog | where TimeGenerated > ago(1d) | where SyslogMessage startswith "CEF:" | extend key = hash_sha256(SyslogMessage); let cef = CommonSecurityLog | where TimeGenerated > ago(1d) | extend key = hash_sha256(RawEvent); cef | join kind=innerunique (sys) on key | summarize duplicates = count() Note : You should identify the RawEvent that might be causing the duplicates. 3.1) Duplicate Detection Query Explained This query helps quantify duplicate ingestion when both Syslog and CEF connectors ingest the same events. It works as follows: Build the Syslog set (sys): Filter the 'Syslog' table for the last day and keep only messages that start with 'CEF:'. Compute a SHA-256 hash of the entire message as a stable join key ("key"). Build the CEF set (cef): Filter the 'CommonSecurityLog' table for the last day and compute a SHA-256 hash of the 'RawEvent' field as the same-style join key. Join on the key: Use 'join kind=innerunique' to find messages that exist in both sets (i.e., duplicates). Summarize: Count the number of matching rows to get a duplicate total. 4) Common Pitfalls - Overlapping DCRs applied to the same collector VM causing overlapping facilities/severities. - CEF and Syslog using the same facility on sources, leading to ingestion on both streams. - rsyslog/syslog-ng filters placed after AMA’s own configuration include (ensure your custom rules run before '10-azuremonitoragent.conf'). 5) References - Microsoft Learn: Ingest syslog and CEF messages to Microsoft Sentinel with AMA (https://learn.microsoft.com/en-us/azure/sentinel/connect-cef-syslog-ama)
KQL: setting query time leads to problem in watchlist column projecting
Hello to the community! I have stumbled upon a very strange issue when using watchlists. I have a watchlist with 2 columns (userPrincipalName,allowedActivity) that I am then using to whitelist activities. Watchlist is imported using: let WhitelistedUsers = _GetWatchlist("testQuery") | project userPrincipalName, allowedActivity; Then I wanted to set it to a specific time frame to test it on given data set: set query_now = datetime("1/14/2022, 1:45:46.556 PM"); Problem is that when setting my query for a specific time, I get the following error from the watchlist: 'project' operator: Failed to resolve scalar expression named 'userPrincipalName'. Commenting the set query_now solves the project problem (not my problem though). I tried to set the time before and after watchlist import but that does not solve the issue. I could not find any posts around the topic (quite a specific one), so anyone observed similar behaviors or has a possible explanation? I can probably work around the set query_now with other functions but I gotten used to it, and find this behavior extremely strangeSentinel-Threat Intelligence Feeds Integration to strengthen Threat Detection & Proactive Hunting.
Combining threat intelligence feeds is important for detecting threats and identifying Indicators of Compromise (IOCs) in various scenarios. Here are some key situations where this approach is advantageous: Comprehensive Threat Detection Integrating multiple threat intelligence feeds can cover a wider range of threats. Different feeds may provide unique insights into malicious activities, IP addresses, domain names, and other IOCs. Reducing False Positives Combining feeds helps cross-verify data, decreasing the likelihood of false positives. This ensures that security teams focus on actual threats rather than inaccurate alerts. Enhanced Contextual Analysis Multiple feeds can offer richer context around threats, including tactics, techniques, and procedures (TTPs) used by attackers. This helps in understanding the threat landscape better and making informed decisions. Real-Time Threat Response Integrating feeds allows for real-time updates on emerging threats. This enables security teams to respond swiftly to new threats and mitigate potential damage. Proactive Threat Hunting Threat hunters can use combined feeds to identify patterns and anomalies that might indicate a threat. This proactive approach assists in detecting threats before they can cause significant harm. Improved Threat Intelligence Sharing Combining feeds from different sources, such as government agencies, commercial vendors, and open-source communities, enhances the overall quality and reliability of threat intelligence. Example Query in Microsoft Sentinel Here's an example of how you might combine two threat intelligence feeds using the coalesce function in KQL: _______________________________________________________________________________________ ThreatIntelFeed1 | extend CombinedIndicator = coalesce(ThreatIntelFeed1.Indicator, ThreatIntelFeed2.Indicator) | extend CombinedDescription = coalesce(ThreatIntelFeed1.Description, ThreatIntelFeed2.Description) | project CombinedIndicator, CombinedDescription _________________________________________________________________________________________ In the above example coalsce function is used. The coalesce function in Kusto Query Language (KQL) is used to evaluate a list of expressions and return the first non-null (or non-empty for strings) expression. This function is particularly useful in Microsoft Sentinel for handling data where some fields might be missing or null. Syntax coalesce(arg, arg_2, [arg_3, ...]) arg: The expression to be evaluated. All arguments must be of the same type. Maximum of 64 arguments is supported. Functions of coalesce in Sentinel Threat Intelligence Feeds Handling Missing Data: It helps in filling gaps where data might be missing by providing a fallback value. For example, if one threat intelligence feed lacks an IP address, coalesce can pull it from another feed. Data Normalization: Combines multiple fields into one, ensuring that you always have a value to work with. This is useful when different feeds provide similar data in different fields. Simplifying Queries: Reduces the need for complex conditional logic to handle null values, making queries more readable and maintainable. Let’s look at Threat Intelligence Analytic rule where caolsec function is used. The query combines threat intelligence indicators with DNS data to identify potential malicious activity. It ensures that only relevant and recent indicators are considered and matches them with DNS queries to detect suspicious behavior. This query ensures that you obtain the most comprehensive data by taking the first non-null value from either feed. Let's break down this KQL query step by step: Define Lookback Periods dt_lookBack: Sets a lookback period of 1 hour for DNS data. ioc_lookBack: Sets a lookback period of 14 days for threat intelligence indicators. Extract Relevant Threat Intelligence Indicators ThreatIntelligenceIndicator: Filters threat intelligence indicators generated within the last 14 days and not expired. arg_max(TimeGenerated, *) by IndicatorId: Summarizes to get the latest indicator for each IndicatorId. Active == true: Filters only active indicators. coalesce(NetworkIP, NetworkDestinationIP, NetworkSourceIP, EmailSourceIpAddress, "NO_IP"): Combines various IP fields into a single IoC field, defaulting to "NO_IP" if none are present. where IoC != "NO_IP": Filters out entries without valid IP addresses. Join with DNS Data join kind=innerunique: Joins the threat intelligence indicators with DNS data using an inner unique join to keep performance fast and result set low. _Im_Dns(starttime=ago(dt_lookBack)): Retrieves DNS data from the last hour. where isnotempty(DnsResponseName): Filters DNS records with non-empty response names. summarize imDns_mintime=min(TimeGenerated), imDns_maxtime=max(TimeGenerated) by SrcIpAddr, DnsQuery, DnsResponseName, Dvc, EventProduct, EventVendor: Summarizes DNS data by various fields. extract_all(@'(\d+\.\d+\.\d+\.\d+)', DnsResponseName): Extracts all IP addresses from the DNS response name. mv-expand IoC = addresses to typeof(string): Expands the extracted IP addresses into individual rows. Combined KQL looks like below _________________________________________________________________________________________ let dt_lookBack = 1h; let ioc_lookBack = 14d; let IP_TI = ThreatIntelligenceIndicator | where TimeGenerated >= ago(ioc_lookBack) and ExpirationDateTime > now() | summarize LatestIndicatorTime = arg_max(TimeGenerated, *) by IndicatorId | where Active == true | extend IoC = coalesce(NetworkIP, NetworkDestinationIP, NetworkSourceIP,EmailSourceIpAddress,"NO_IP") | where IoC != "NO_IP" ; IP_TI | join kind=innerunique // using innerunique to keep perf fast and result set low, we only need one match to indicate potential malicious activity that needs to be investigated ( _Im_Dns(starttime=ago(dt_lookBack)) | where isnotempty(DnsResponseName) | summarize imDns_mintime=min(TimeGenerated), imDns_maxtime=max(TimeGenerated) by SrcIpAddr, DnsQuery, DnsResponseName, Dvc, EventProduct, EventVendor | extend addresses = extract_all (@'(\d+\.\d+\.\d+\.\d+)', DnsResponseName) | mv-expand IoC = addresses to typeof(string) ) on IoC _________________________________________________________________________________________ Summary This article explores the importance of combining threat intelligence feeds to improve security operations. Key benefits include extending threat coverage, reducing false positives, and enhancing contextual analysis through detailed insights into attackers' tactics and techniques. The integration process also facilitates real-time threat updates and enables better collaboration between different intelligence sources. An example is provided using KQL (Kusto Query Language) to demonstrate how threat intelligence feeds can be combined effectively within Microsoft Sentinel. The query showcases steps like defining lookback periods, extracting relevant indicators, and correlating them with DNS data through an inner unique join. By leveraging this method, organizations can efficiently identify potential malicious activities and strengthen their threat response capabilities. The content emphasizes that integrating threat feeds is not just a technical function but a strategic necessity to fortify organizations against evolving cyber threats.Sentinel Notebook: Guided Hunting - Domain Generation Algorithm (DGA) Detection
Overview This notebook, titled “Guided Hunting - Domain Generation Algorithm (DGA) Detection”, provides a framework for investigating anomalous network activity by identifying domains generated by algorithms, which are often used by malware to evade detection. It integrates data from Log Analytics (DeviceNetworkEvents) and employs Python-based tools and libraries such as “msticpy”, “pandas”, and “scikit-learn” for analysis and visualization. DGA detection is crucial for cybersecurity as it helps identify and mitigate threats like botnets and malware that use dynamically generated domains for command-and-control communication, making it a key component in proactive threat hunting and network defense. Link: https://github.com/GonePhishing402/SentinelNotebooks/blob/main/DGA_Detection_ManagedIdentity.ipynb What is Domain Generation Algorithm and How to Detect it? A Domain Generation Algorithm (DGA) is a technique used by malware to create numerous domain names for communicating with Command and Control (C2) servers, ensuring continued operation even if some domains are blocked. DGAs evade static signature detection by dynamically generating unpredictable domain names, making it hard for traditional security methods to identify and blacklist them. Machine learning models can effectively detect DGAs by analyzing patterns and features in domain names, leveraging techniques like deep learning to adapt to new variants and identify anomalies that static methods may miss. How to Run the Notebook Log in with Managed Identity This notebook requires you to authenticate with a managed identity. The managed identity can be created from the Azure portal and must have the following RBAC: - Sentinel Contributor - Log Analytics Contributor - AzureML Data Scientist - AzureML Compute Operator Replace the [CLIENT_ID] with the client ID for your managed identity. This can be obtained from the Azure portal under Managed Identities -> Select the identity -> Overview. Note: This notebook will still work if you choose to authenticate with just an azure user using the CLI method as well. Import Libraries This code block is used to import the necessary libraries and label the “credential” variable to use the ManagedIdentityCredential() library. Setup msticpyconfig.yaml This section just pulls the msticpyconfig.yaml to use later on in the notebook. Ensure this is setup before running this notebook and in your current working directory. Setup QueryProvider The query provider is setup for Azure Sentinel. This does not need to be changed unless you want to use a different query provider from msticpy. Connect to Sentinel This code block is used to connect to Sentinel with the managed identity to the workspace specified in your msticpyconfig.yaml. You should see a “connected” after running this code block. DGA Model Creation This code block is designed to use CountVectorizer() and MultinomialNB() to create a model called dga_model.joblib and save it to the path specified in the “model_filename” variable. It is important to change this path specific to your environment. You must give the algorithm data to learn from in order to be effective. Download the domain.csv located here and upload to your current working directory on Azure Machine Learning Workspace: DGA_Detection/data/domain.csv at master · hmaccelerate/DGA_Detection You must also change line 10 in this code block to have the “labeled_domains_df” point to the domain.csv in your environment. Once you run the code block, you should see the model saved and the model accuracy. This number will vary depending on the data you are giving it. Apply dga_model.joblib to Sentinel Data This code block uses the model that we generated in the previous block and runs it against our data we specific in the “query” variable. This is using domain names from the “DeviceNetworkEvents” table in MDE events. The “parse_json” was used in our KQL to extract the appropriate sub-field needed for this search. When this model is run against the data, it will try to determine if any domains in our environment are associated with domain generation algorithms (DGA). If the “IsDGA” column contains a value of “True”, the model has determined that the characteristics of that domain matches a DGA. Here is what the output will look like: Output All Results to CSV This code block will output all the results above to a CSV called “dgaresults.csv”. Change the “output_path” variable to match your environment. Filter DGA Results to CSV This code block will output just the DGA results above to a CSV called “dgaresults2.csv”. Change the “output_path” variable to match your environment. How to Investigate these Results Further You can take the domain results that match DGA and find the correlating IP to see if it matches any threat intelligence. Correlate findings with other security logs, such as firewall or endpoint data, to uncover patterns of malicious behavior. This approach helps pinpoint potential threats and enables proactive mitigation. We can also create logic apps to automate follow-on analysis of these notebooks. This will be covered in a later blog.AMA: Microsoft SIEM & XDR: unified security operations
At Microsoft Ignite we announced that we are bringing our Microsoft Sentinel and Microsoft Defender XDR products together to deliver an optimized and unified security operations experience. We are combining the full power of these products into a single portal enhanced with more comprehensive features, automation, guided experiences, and Microsoft Security Copilot. Bring your questions to this Ask Microsoft Anything (AMA) as members of our Microsoft Security engineering team bring clarity and insights about this new experience. This session is part of the Microsoft Security Tech Accelerator. RSVP for event reminders, add it to your calendar, and post your questions and comments below! This session will also be recorded and available on demand shortly after conclusion of the live event.
