Ingesting Google Cloud Logs into Microsoft Sentinel: Native vs. Custom Architectures
Overview of GCP Log Types and SOC Value Modern Security Operations Centers (SOCs) require visibility into key Google Cloud Platform logs to detect threats and suspicious activities. The main log types include: GCP Audit Logs – These encompass Admin Activity, Data Access, and Access Transparency logs for GCP services. They record every administrative action (resource creation, modification, IAM changes, etc.) and access to sensitive data, providing a trail of who did what and when in the cloud. In a SOC context, audit logs help identify unauthorized changes or anomalous admin behavior (e.g. an attacker creating a new service account or disabling logging). They are essential for compliance and forensics, as they detail changes to configurations and access patterns across GCP resources. VPC Flow Logs – These logs capture network traffic flow information at the Virtual Private Cloud (VPC) level. Each entry typically includes source/destination IPs, ports, protocol, bytes, and an allow/deny action. In a SOC, VPC flow logs are invaluable for network threat detection: they allow analysts to monitor access patterns, detect port scanning, identify unusual internal traffic, and profile ingress/egress traffic for anomalies. For example, a surge in outbound traffic to an unknown IP or lateral movement between VMs can be spotted via flow logs. They also aid in investigating data exfiltration and verifying network policy enforcement. Cloud DNS Logs – Google Cloud DNS query logs record DNS requests/responses from resources, and DNS audit logs record changes to DNS configurations. DNS query logs are extremely useful in threat hunting because they can reveal systems resolving malicious domain names (C2 servers, phishing sites, DGA domains) or performing unusual lookups. Many malware campaigns rely on DNS; having these logs in Sentinel enables detection of known bad domains and anomalous DNS traffic patterns. DNS audit logs, on the other hand, track modifications to DNS records (e.g. newly added subdomains or changes in IP mappings), which can indicate misconfigurations or potential domain hijacking attempts. Together, these GCP logs provide comprehensive coverage: audit logs tell what actions were taken in the cloud, while VPC and DNS logs tell what network activities are happening. Ingesting all three into Sentinel gives a cloud security architect visibility to detect unauthorized access, network intrusions, and malware communication in a GCP environment. Native Microsoft Sentinel GCP Connector: Architecture & Setup Microsoft Sentinel offers native data connectors to ingest Google Cloud logs, leveraging Google’s Pub/Sub messaging for scalable, secure integration. The native solution is built on Sentinel’s Codeless Connector Framework (CCF) and uses a pull-based architecture: GCP exports logs to Pub/Sub, and Sentinel’s connector pulls from Pub/Sub into Azure. This approach avoids custom code and uses cloud-native services on both sides. Supported GCP Log Connectors: Out of the box, Sentinel provides connectors for: GCP Audit Logs – Ingests the Cloud Audit Logs (admin activity, data access, transparency). GCP Security Command Center (SCC) – Ingests security findings from Google SCC for threat and vulnerability management. GCP VPC Flow Logs – (Recently added) Ingests VPC network flow logs. GCP Cloud DNS Logs – (Recently added) Ingests Cloud DNS query logs and DNS audit logs. Others – Additional connectors exist for specific GCP services (Cloud Load Balancer logs, Cloud CDN, Cloud IDS, GKE, IAM activity, etc.) via CCF, expanding the coverage of GCP telemetry in Sentinel. Each connector typically writes to its own Log Analytics table (e.g. GCPAuditLogs, GCPVPCFlow, GCPDNS, etc.) and comes with built-in KQL parsers. Architecture & Authentication: The native connector uses Google Pub/Sub as the pipeline for log delivery. On the Google side, you will set up a Pub/Sub Topic that receives the logs (via Cloud Logging exports), and Sentinel will subscribe to that topic. Authentication is handled through Workload Identity Federation (WIF) using OpenID Connect: instead of managing static credentials, you establish a trust between Azure AD and GCP so that Sentinel can impersonate a GCP service account. The high-level architecture is: GCP Cloud Logging (logs from services) → Log Router (export sink) → Pub/Sub Topic → (Secure pull over OIDC) → Azure Sentinel Data Connector → Log Analytics Workspace. This ensures a secure, keyless integration. The Azure side (Sentinel) authenticates as a Google service account via OIDC tokens issued by Azure AD, which GCP trusts through the Workload Identity Provider. Below are the detailed setup steps: GCP Setup (Publishing Logs to Pub/Sub) Enable Required APIs: Ensure the GCP project hosting the logs has the IAM API and Cloud Resource Manager API enabled (needed for creating identity pools and roles). You’ll also need owner/editor access on the project to create the resources below. Create a Workload Identity Pool & Provider: In Google Cloud IAM, create a new Workload Identity Pool (e.g. named “Azure-Sentinel-Pool”) and then a Workload Identity Provider within that pool that trusts your Azure AD tenant. Google provides a template for Azure AD OIDC trust – you’ll supply your Azure tenant ID and the audience and issuer URIs that Azure uses. For Azure Commercial, the issuer is typically https://sts.windows.net/<TenantID>/ and audience api://<some-guid> as documented. (Microsoft’s documentation or Terraform scripts provide these values for the Sentinel connector.) Create a Service Account for Sentinel: Still in GCP, create a dedicated service account (e.g. email address removed for privacy reasons). This account will be what Sentinel “impersonates” via the WIF trust. Grant this service account two key roles: Pub/Sub Subscriber on the Pub/Sub Subscription that will be created (allows pulling messages). You can grant roles/pubsub.subscriber at the project level or on the specific subscription. Workload Identity User on the Workload Identity Pool. In the pool’s permissions, add a principal of the form principalSet://iam.googleapis.com/projects/<WIF_project_number>/locations/global/workloadIdentityPools/<Pool_ID>/* and grant it the role roles/iam.workloadIdentityUser on your service account. This allows the Azure AD identity to impersonate the GCP service account. Note: GCP best practice is often to keep the identity pool in a centralized project and service accounts in separate projects, but as of late 2023 the Sentinel connector UI expected them in one project (a limitation under review). It’s simplest to create the WIF pool/provider and the service account within the same GCP project to avoid connectivity issues (unless documentation confirms support for cross-project). Create a Pub/Sub Topic and Subscription: Open the GCP Pub/Sub service and create a Topic (for example, projects/yourproject/topics/sentinel-logs). It’s convenient to dedicate one topic per log type or use case (e.g. one for audit logs). As you create the topic, you can add a Subscription to it in Pull mode (since Sentinel will pull messages). Use a default subscription with an appropriate name (e.g. sentinel-audit-sub). You can leave the default settings (ack deadline, retention) as is, or extend retention if you want messages to persist longer in case of downtime (default is 7 days). Create a Logging Export (Sink): In GCP Cloud Logging, set up a Log Sink to route the desired logs into the Pub/Sub topic. Go to Logging > Logs Router and create a sink: Give it a descriptive name (e.g. audit-logs-to-sentinel). Choose Cloud Pub/Sub as the Destination and select the topic you created (or use the format pubsub.googleapis.com/projects/yourproject/topics/sentinel-logs). Scope and filters: Decide which logs to include. For Audit Logs, you might include ALL audit logs in the project (the sink can be set to include admin activity, data access, etc., by default for the whole project or even entire organization if created at org level). For other log types like VPC Flow Logs or DNS, you’d set an inclusion filter for those specific log names (e.g. logName:"compute.googleapis.com/vpc_flows" to capture VPC Flow Logs). You can also create organization-level sinks to aggregate logs from all projects. Permissions: When creating a sink, GCP will ask to grant the sink service account publish rights to the Pub/Sub topic. Accept this so logs can flow. Once the sink is created, verify logs are flowing: in Pub/Sub > Subscriptions, you can “Pull” messages manually to see if any logs appear. Generating a test event (e.g., create a VM to produce an audit log, or make a DNS query) can help confirm. At this point, GCP is set up to export logs. All requisite GCP resources (IAM federation, service account, topic/subscription, sink) are ready. Google also provides Terraform scripts (and Microsoft supplies Terraform templates in their GitHub) to automate these steps. Using Terraform, you can stand up the IAM and Pub/Sub configuration quickly if comfortable with code. Azure Sentinel Setup (Connecting the GCP Connector) With GCP publishing logs to Pub/Sub, configure Sentinel to start pulling them: Install the GCP Solution: In the Azure portal, navigate to your Sentinel workspace. Under Content Hub (or Data Connectors), find Google Cloud Platform Audit Logs (for example) and click Install. This deploys any needed artifacts (the connector definition, parser, etc.). Repeat for other GCP solutions (like GCP VPC Flow Logs or GCP DNS) as needed. Open Data Connector Configuration: After installation, go to Data Connectors in Sentinel, search for “GCP Pub/Sub Audit Logs” (or the relevant connector), and select it. Click Open connector page. In the connector page, click + Add new (or Add new collector) to configure a new connection instance. Enter GCP Parameters: A pane will prompt for details to connect: you need to supply the Project ID (of the GCP project where the Pub/Sub lives), the Project Number, the Topic and Subscription name, and the Service Account Email you created. You’ll also enter the Workload Identity Provider ID (the identifier of the WIF provider, usually in format projects/<proj>/locations/global/workloadIdentityPools/<pool>/providers/<provider>). All these values correspond to the GCP resources set up earlier – the UI screenshot in docs shows sample placeholders. Make sure there are no typos (a common error is mixing up project ID (name) with project number, or using the wrong Tenant ID). Data Collection Rule (DCR): The connector may also ask for a Data Collection Rule (DCR) and DCE (Data Collection Endpoint) names. Newer connectors based on CCF use the Log Ingestion API, so behind the scenes a DCR is used. If required, provide a name (the docs often suggest prefixing with “Microsoft-Sentinel-” e.g., Microsoft-Sentinel-GCPAuditLogs-DCR). The system will create the DCR and a DCE for you if not already created. (If you installed via Content Hub, this is often automated – just ensure the names follow any expected pattern.) Connect: Click Connect. Sentinel will attempt to use the provided info to establish connectivity. It performs checks like verifying the subscription exists and that the service account can authenticate. If everything is set up properly, the connector will connect and start streaming data. In case of an error, you’ll receive a detailed message. For example, an error about WIF Pool ID not found or subscription not found indicates an issue in the provided IDs or permissions. Double-check those values if so. Validation: After ~15-30 minutes, verify that logs are arriving. You can run a Log Analytics query on the new table, for example: GCPAuditLogs | take 5 (for audit logs) or GCPVPCFlow | take 5 for flow logs. You should see records if ingestion succeeded. Sentinel also provides a “Data connector health” feature – enable it to get alerts or status on data latency and volume for this connector. Data Flow and Ingestion: Once connected, the system works continuously and serverlessly: GCP Log Router pushes new log entries to Pub/Sub as they occur. The Sentinel connector (running in Azure’s cloud) polls the Pub/Sub subscription. It uses the service account credentials (via OIDC token) to call the Pub/Sub API and pull messages in batches. This happens at a defined interval (typically very frequently, e.g. every few seconds). Each message (which contains a log entry in JSON) is then ingested into Log Analytics. The CCF connector uses the Log Ingestion API on the backend, mapping the JSON fields to the appropriate table schema. The logs appear in the respective table (with columns for each JSON field or a dynamic JSON column, depending on the connector design). Sentinel’s built-in parser or Normalized Schemas (ASIM) can be used to query these logs in a friendly way. For instance, the Audit Logs solution includes KQL functions to parse out common fields like user, method, status, etc., from the raw JSON. This native pipeline is fully managed – you don’t have to run any servers or code. The use of Pub/Sub and OIDC makes it scalable and secure by design. Design Considerations & Best Practices for the Native Connector: Scalability & Performance: The native connector approach is designed for high scale. Pub/Sub itself can handle very large log volumes with low latency. The Sentinel CCF connectors use a SaaS, auto-scaling model – no fixed infrastructure means they will scale out as needed to ingest bursts of data. This is a significant advantage over custom scripts or function apps which might need manual scaling. In testing, the native connector can reliably ingest millions of log events per day. If you anticipate extremely high volumes (e.g. VPC flow logs from hundreds of VMs), monitor the connector’s performance but it should scale automatically. Reliability: By leveraging Pub/Sub’s at-least-once delivery, the integration is robust. Even if Azure or the connector has a transient outage, log messages will buffer in Pub/Sub. Once the connector resumes, it will catch up on the backlog. Ensure the subscription’s message retention is adequate (the default 7 days is usually fine). The connector acknowledges messages only after they’re ingested into Log Analytics, which prevents data loss on failures. This reliability is achieved without custom code – reducing the chance of bugs. Still, it’s good practice to use Sentinel’s connector health metrics to catch any issues (e.g., if the connector hasn’t pulled data in X minutes, indicating a problem). Security: The elimination of persistent credentials is a best practice. By using Workload Identity Federation, the Azure connector obtains short-lived tokens to act as the GCP service account. There is no need to store a GCP service account key file, which could be a leak risk. Ensure that the service account has the minimal roles needed. Typically, it does not need broad viewer roles on all GCP resources – it just needs Pub/Sub subscription access (and logging viewer only if you choose to restrict log export by IAM – usually not necessary since the logs are already exported via the sink). Keep the Azure AD application’s access limited too: the Azure AD app (which underpins the Sentinel connector) only needs to access the Sentinel workspace and doesn’t need rights in GCP – the trust is handled by the WIF provider. Filtering and Log Volume Management: A common best practice is to filter GCP logs at the sink to avoid ingesting superfluous data. For instance, if only certain audit log categories are of interest (e.g., Admin Activity and security-related Data Access), you could exclude noisy Data Access logs like storage object reads. For VPC Flow Logs, you might filter on specific subnetworks or even specific metadata (though typically you’d ingest all flows and use Sentinel for filtering). Google’s sink filters allow you to use boolean expressions on log fields. The community recommendation for Firewall or VPC logs, for example, is to set a filter so that only those logs go into that subscription. This reduces cost and noise in Sentinel. Plan your log sinks carefully: you may create multiple sinks if you want to separate log types (one sink to Pub/Sub for audit logs, another sink (with its own topic) for flow logs, etc.). The Sentinel connectors are each tied to one subscription and one table, so separating by log type can help manage parsing and access. Coverage Gaps: Check what the native connectors support as of the current date. Microsoft has been rapidly adding GCP connectors (VPC Flow Logs and DNS logs were in Preview in mid-2025 and are likely GA by now). If a needed log type is not supported (for example, if you have a custom application writing logs to Google Cloud Logging), you might consider the custom ingestion approach (see next section). For most standard infrastructure logs, the native route is available and preferable. Monitoring and Troubleshooting: Familiarize yourself with the connector’s status in Azure. In the Sentinel UI, each configured GCP connector instance will show a status (Connected/Warning/Error) and possibly last received timestamp. If there’s an issue, gather details from error messages there. On GCP, monitor the Pub/Sub subscription: pubsub subscriptions list --filter="name:sentinel-audit-sub" can show if there’s a growing backlog (unacked message count). A healthy system should have near-zero backlog with steady consumption. If backlog is growing, it means the connector isn’t keeping up or isn’t pulling – check Azure side for throttling or errors. Multi-project or Org-wide ingestion: If your organization has many GCP projects, you have options. You could deploy a connector per project, or use an organization-level log sink in GCP to funnel logs from all projects into a single Pub/Sub. The Sentinel connector can pull organization-wide if the service account has rights (the Terraform script allows an org sink by providing an organization-id). This centralizes management but be mindful of very large volumes. Also, ensure the service account has visibility on those logs (usually not an issue if they’re exported; the sink’s own service account handles the export). In summary, the native GCP connector provides a straightforward and robust way to get Google Cloud logs into Sentinel. It’s the recommended approach for supported log types due to its minimal maintenance and tight integration. Custom Ingestion Architecture (Pub/Sub to Azure Event Hub, etc.) In cases where the built-in connector doesn’t meet requirements – e.g., unsupported log types, custom formats, or corporate policy to use an intermediary – you can design a custom ingestion pipeline. The goal of custom architectures is the same (move logs from GCP to Sentinel) but you can incorporate additional processing or routing. One reference pattern is: GCP Pub/Sub → Azure Event Hub → Sentinel, which we’ll use as an example among other alternatives. GCP Export (Source): This part remains the same as the native setup – you create log sinks in GCP to continuously export logs to Pub/Sub topics. You can reuse what you’ve set up or create new, dedicated Pub/Sub topics for the custom pipeline. For instance, you might have a sink for Cloud DNS query logs if the native connector wasn’t used, sending those logs to a topic. Ensure you also create subscriptions on those topics for your custom pipeline to pull from. If you plan to use a GCP-based function to forward data, a Push subscription could be used instead (which can directly call an HTTP endpoint), but a Pull model is more common for custom solutions. Bridge / Transfer Component: This is the core of the custom pipeline – a piece of code that reads from Pub/Sub and sends data to Azure. Several implementation options: Google Cloud Function or Cloud Run (in GCP): You can deploy a Cloud Function that triggers on new Pub/Sub messages (using Google’s EventArc or a Pub/Sub trigger). This function will execute with the message as input. Inside the function, you would parse the Pub/Sub message and then forward it to Azure. This approach keeps the “pull” logic on the GCP side – effectively GCP pushes to Azure. For example, a Cloud Function (Python) could be subscribed to the sentinel-logs topic; each time a log message arrives, the function runs, authenticates to Azure, and calls the ingestion API. Cloud Functions can scale out automatically based on the message volume. Custom Puller in Azure (Function App or Container): Instead of running the bridging code in GCP, you can run it in Azure. For instance, an Azure Function with a timer trigger (running every minute) or an infinite-loop container in Azure Kubernetes Service could use Google’s Pub/Sub client library to pull messages from GCP. You would provide it the service account credentials (likely a JSON key) to authenticate to the Pub/Sub pull API. After pulling a batch of messages, it would send them to the Log Analytics workspace. This approach centralizes everything in Azure but requires managing GCP credentials securely in Azure. Using Google Cloud Dataflow (Apache Beam): For a heavy-duty streaming solution, you could write an Apache Beam pipeline that reads from Pub/Sub and writes to an HTTP endpoint (Azure). Google Dataflow runs Beam pipelines in a fully managed way and can handle very large scale with exactly-once processing. However, this is a complex approach unless you already use Beam – it’s likely overkill for most cases and involves significant development. No matter which method, the bridge component must handle reading, transforming, and forwarding logs efficiently. Destination in Azure: There are two primary ways to ingest the data into Sentinel: Azure Log Ingestion API (via DCR) – This is the modern method (introduced in 2022) to send custom data to Log Analytics. You’ll create a Data Collection Endpoint (DCE) in Azure and a Data Collection Rule (DCR) that defines how incoming data is routed to a Log Analytics table. For example, you might create a custom table GCP_Custom_Logs_CL for your logs. Your bridge component will call the Log Ingestion REST API endpoint (which is a URL associated with the DCE) and include a shared access signature or Azure AD token for auth. The payload will be the log records (in JSON) and the DCR rule ID to apply. The DCR can also perform transformations if needed (e.g., mappings of fields). This API call will insert the data into Log Analytics in real-time. This approach is quite direct and is the recommended custom ingestion method (it replaces the older HTTP Data Collector API). Azure Event Hub + Sentinel Connector – In this approach, instead of pushing directly into Log Analytics, you use an Event Hub as an intermediate buffer in Azure. Your GCP bridge will act as a producer, sending each log message to an Event Hub (over AMQP or using Azure’s SDK). Then, you need something to get data from Event Hub into Sentinel. There are a couple of options: Historically, Sentinel provided an Event Hub data connector (often used for Azure Activity logs or custom CEF logs). This connector can pull events from an Event Hub and write to Log Analytics. However, it typically expects the events to be in a specific format (like CEF or JSON with a known structure). If your logs are raw JSON, you might need to wrap them or use a compatible format. This method is somewhat less flexible unless you tailor your output to what Sentinel expects. Alternatively (and more flexibly), you can write a small Azure Function that triggers on the Event Hub (using Event Hub trigger binding). When a message arrives, the function takes it and calls the Log Ingestion API (similar to method (a) above) to put it into Log Analytics. Essentially, this just decouples the pulling from GCP (done by the first function) and the pushing to Sentinel (done by the second function). This two-stage design might be useful if you want to do more complex buffering or retries, but it does introduce more components. Using an Event Hub in the pipeline can be beneficial if you want a cloud-neutral queue between GCP and Azure (maybe your organization already consolidates logs in an Event Hub or Kafka). It also allows reusing any existing tools that read off Event Hubs (for example, maybe feeding the same data to another system in parallel to Sentinel). This pattern – Cloud Logging → Pub/Sub → Event Hub → Log Analytics – has been observed in real-world multi-cloud deployments, essentially treating Pub/Sub + Event Hub as a bridging message bus between clouds. Data Transformation: With a custom pipeline, you have full control (and responsibility) for any data transformations needed. Key considerations: Message Decoding: GCP Pub/Sub messages contain the log entry in the data field, which is a base64-encoded string of a JSON object. Your code must decode that (it’s a one-liner in most languages) to get the raw JSON log. After decoding, you’ll have a JSON structure identical to what you’d see in Cloud Logging. For example, an audit log entry JSON has fields like protoPayload, resourceName, etc. Schema Mapping: Decide how to map the JSON to your Log Analytics table. You could ingest the entire JSON as a single column (and later parse in KQL), but it’s often better to map important fields. For instance, for VPC Flow Logs, you might extract src_ip, dest_ip, src_port, dest_port, bytes_sent, action and map each to a column in a custom table. This requires that you create the custom table with those columns and configure the DCR’s transformation schema accordingly. If using the Log Ingestion API, the DCR can transform the incoming JSON to the table schema. If using the Data Collector API (legacy, not recommended now), your code would need to format records as the exact JSON that Log Analytics expects. Enrichment (optional): In a custom pipeline, you could enrich the logs before sending to Sentinel. For example, performing IP geolocation on VPC flow logs, or tagging DNS logs with threat intel (if a domain is known malicious) – so that the augmented information is stored in Sentinel. Be cautious: enrichment adds processing time and potential failure points. If it’s light (like a dictionary lookup), it might be fine; if heavy, consider doing it after ingestion using Sentinel analytics instead. Filtering: Another advantage of custom ingestion is that you can filter events at the bridge. You might decide to drop certain events entirely (to save cost or noise). For example, if DNS query logs are too verbose, you might only forward queries for certain domains or exclude known benign domains. Or for audit logs, you might exclude read-only operations. This gives flexibility beyond what the GCP sink filter can do, since you have the full event content to decide. The trade-off is complexity – every filter you implement must be maintained/justified. Batching: The Log Ingestion API allows sending multiple records in one call. It’s more efficient to batch a bunch of log events (say 100 at a time) into one API request rather than call per event. Your function can accumulate a short batch (with some timeout or max size) and send together. This improves throughput and lowers overhead. Ensure the payload stays within API limits (~1 MB per post, and 30,000 events per post for Log Ingestion API). Pub/Sub and Event Hub also have batch capabilities – you may receive multiple messages in one invocation or read them in a loop. Design your code to handle variable batch sizes. Authentication & Permissions (Custom Pipeline): You will effectively need to handle two authentications: GCP → Bridge, and Bridge → Azure: GCP to Bridge: If using a GCP Cloud Function triggered by Pub/Sub, GCP handles the auth for pulling the message (the function is simply invoked with the data). If pulling from Azure, you’ll need GCP credentials. The most secure way is to use a service account key with minimal permissions (just Pub/Sub subscriber on the subscription). Store this key securely (Azure Key Vault or as an App Setting in the Azure Function, possibly encrypted). The code uses this key (a JSON file or key string) to initialize the Pub/Sub client. Google’s libraries support reading the key from an environment variable. Alternatively, you could explore using the same Workload Identity Federation concept in reverse (Azure to GCP), but that’s non-trivial to set up manually for custom code. A service account key is straightforward but do rotate it periodically. On GCP’s side, you might also restrict the service account so it cannot access anything except Pub/Sub. Bridge to Azure: To call the Log Ingestion API, you need an Azure AD App Registration (client ID/secret) with permissions or a SAS token for the DCR. The modern approach: create an AAD app, grant it the role Monitoring Data Contributor on your Sentinel workspace or explicitly grant the DCR permissions (Log Ingestion Data Contributor). Then your code can use the app’s client ID and secret to get a token and call the API. This is a secure, managed way. Alternatively, the DCR can be configured with a shared access signature (SAS) that you generate in Azure – your code could use that SAS token in the API URL (so that no interactive auth is needed). The older Data Collector API used the workspace ID and a primary key for auth (HMAC SHA-256 header) – some existing solutions still use that, but since that API is being deprecated, it’s better to use the new method. In summary: ensure the Azure credentials are stored safely (Key Vault or GCP Secret Manager if function is in GCP) and that you follow principle of least privilege (only allow ingest, no read of other data). End-to-End Data Flow Example: To make this concrete, consider an example where we ingest Firewall/VPC logs using a custom pipeline (this mirrors a solution published by Microsoft for when these logs weren’t yet natively supported): A GCP Log Sink filters for VPC Firewall logs (the logs generated by GCP firewall rules, which are part of VPC flow logging) and exports them to a Pub/Sub topic. An Azure Function (in PowerShell, as in the example, or any language) runs on a timer. Every minute, it pulls all messages from the Pub/Sub subscription (using the Google APIs). The function authenticates with a stored service account key to do this. It then decodes each message’s JSON. The function constructs an output in the required format for Sentinel’s Log Ingestion API. In this case, they created a custom Log Analytics table (say GCPFirewall_CL) with columns matching the log fields (source IP, dest IP, action, etc.). The function maps each JSON field to a column. For instance, json.payload.sourceIp -> src_ip column. It then calls the Log Ingestion REST API to send a batch of log records. The call is authorized with an Azure AD app’s client ID/secret which the function has in its config. Upon successfully POSTing the data, the function sends an acknowledgment back to Pub/Sub for those messages (or, if using the Pub/Sub client in pull mode, it acks as it pulls). This removal is important to ensure the messages don’t get re-delivered. If the send to Azure fails for some reason, the function can choose not to ack, so that the message remains in Pub/Sub and will be retried on the next run (ensuring reliability). The logs show up in Sentinel under the custom table, and can now be used in queries and analytics just like any other log. The entire process from log generation in GCP to log ingestion in Sentinel can be only a few seconds of latency in this design, effectively near-real-time. Tooling & Infrastructure: When implementing the above, some recommended tools: Use official SDKs where possible. For example, Google Cloud has a Pub/Sub client library for Python, Node.js, C#, etc., which simplifies pulling messages. Azure has an SDK for the Monitor Ingestion API (or you can call the REST endpoints directly with an HTTP client). This saves time versus manually crafting HTTP calls and auth. Leverage Terraform or IaC for repeatability. You can automate creation of Azure resources (Function App, Event Hub, etc.) and even the GCP setup. For instance, the community SCC->Sentinel example provides Terraform scripts. This makes it easier to deploy the pipeline in dev/test and prod consistently. Logging and monitoring: implement robust logging in your function code. In GCP Cloud Functions, use Cloud Logging to record errors (so you can see if something fails). In Azure Functions, use Application Insights to track failures or performance metrics. Set up alerts if the function fails repeatedly or if an expected log volume drops (which could indicate a broken pipeline). Essentially, treat your custom pipeline as you would any production integration – monitor its health continuously. Example Use-Case – Ingesting All Custom GCP Logs: One key advantage of a custom approach is flexibility. Imagine you have a custom application writing logs to Google Cloud Logging (Stackdriver) that has no out-of-the-box Sentinel connector. You can still get those logs into Sentinel. As one cloud architect noted, they built a fully custom pipeline with GCP Log Sink -> Pub/Sub -> Cloud Function -> Sentinel, specifically to ingest arbitrary GCP logs beyond the built-in connectors. This unlocked visibility into application-specific events that would otherwise be siloed. While doing this, they followed many of the steps above, demonstrating that any log that can enter Pub/Sub can ultimately land in Sentinel. This extensibility is a major benefit of a custom solution – you’re not limited by what Microsoft or Google have pre-integrated. In summary, the custom ingestion route requires more effort up front, but it grants complete control. You can tune what you collect, transform data to your needs, and integrate logs that might not be natively supported. Organizations often resort to this if they have very specific needs or if they started building ingestion pipelines before native connectors were available. Many will start with custom for something like DNS logs and later switch to a native connector once available. A hybrid approach is also possible (using native connector for audit logs, but custom for a niche log source). Comparison of Native vs. Custom Ingestion Methods Both native and custom approaches will get your GCP logs into Microsoft Sentinel, but they differ in complexity and capabilities. The table below summarizes the trade-offs to help choose the right approach: Aspect Native GCP Connector (Sentinel Pub/Sub Integration) Custom Ingestion Pipeline (DIY via Event Hubs or API) Ease of Setup Low-Code Setup: Requires configuration in GCP and Azure, but no custom code. You use provided Terraform scripts and a UI wizard. In a few hours you can enable the connector if prerequisites (IAM, Pub/Sub) are met. Microsoft’s documentation guides the process step by step. High-Code Setup: Requires designing and writing integration code (function or app) and configuring cloud services (Function Apps, Event Hub, etc.). More moving parts mean a longer setup time – possibly days or weeks to develop and thoroughly test. Suitable if your team has cloud developers or if requirements demand it. Log Type Coverage Supported Logs: Out-of-the-box support for standard GCP logs (audit, SCC findings, VPC flow, DNS, etc.). However, it’s limited to those data types Microsoft has released connectors for. (As of 2025, many GCP services are covered, but not necessarily all Google products.) If a connector exists, it will reliably ingest that log type. Any Log Source: Virtually unlimited – you can ingest any log from GCP, including custom application logs or niche services, as long as you can export it to Pub/Sub. You define the pipeline for each new log source. This is ideal for custom logs beyond built-ins. The trade-off is you must build parsing/handling for each log format yourself. Development & Maintenance Minimal Maintenance: After initial setup, the connector runs as a service. No custom code to maintain; Microsoft handles updates/improvements. You might need to update configuration if GCP projects or requirements change, but generally it’s “configure and monitor.” Support is available from Microsoft for connector issues. Ongoing Maintenance: You own the code. Updates to log schemas, API changes, or cloud platform changes might require code modifications. You need to monitor the custom pipeline, handle exceptions, and possibly update credentials regularly. This approach is closer to software maintenance – expect to allocate effort for bug fixes or improvements over time. Scalability Cloud-Scale (Managed): The connector uses Azure’s cloud infrastructure which auto-scales. High volumes are handled by scaling out processing nodes behind the scenes. GCP Pub/Sub will buffer and deliver messages at whatever rate the connector can pull, and the connector is optimized for throughput. There’s effectively no hard limit exposed to you (aside from Log Analytics ingestion rate limits, which are very high). Custom Scaling Required: Scalability depends on your implementation. Cloud Functions and Event Hubs can scale, but you must configure them (e.g., set concurrency, ensure enough throughput units on Event Hub). If logs increase tenfold, you may need to tweak settings or upgrade plans. There’s more possibility of bottlenecks (e.g., a single-threaded function might lag). Designing for scale (parallelism, batching, multi-partition processing) is your responsibility. Reliability & Resilience Reliable by Design: Built on proven Google Pub/Sub durability and Azure’s reliable ingestion pipeline. The connector handles retries and acknowledgements. If issues occur, Microsoft likely addresses them in updates. Also, you get built-in monitoring in Sentinel for connector health. Reliability Varies: Requires implementing your own retry and error-handling logic. A well-built custom pipeline can be very reliable (e.g., using Pub/Sub’s ack/retry and durable Event Hub storage), but mistakes in code could drop logs or duplicate them. You need to test failure scenarios (network blips, API timeouts, etc.). Additionally, you must implement your own health checks/alerts to know if something breaks. Flexibility & Transformation Standardized Ingestion: Limited flexibility – it ingests logs in their native structure into pre-defined tables. Little opportunity to transform data (beyond what the connector’s mapping does). Essentially “what GCP sends is what you get,” and you rely on Sentinel’s parsing for analysis. All logs of a given type are ingested (you control scope via GCP sink filters, but not the content). Highly Flexible: You can customize everything – which fields to ingest, how to format them, and even augment logs with external data. For example, you could drop benign DNS queries or mask sensitive fields before sending. You can consolidate multiple GCP log types into one table or split one log type into multiple tables if desired. This freedom lets you tailor the data to your environment and use cases. The flip side is complexity: every transformation is custom logic to maintain. Cost Considerations Cost-Efficient Pipeline: There is no charge for the Sentinel connector itself (it’s part of the service). Costs on GCP: Pub/Sub charges are minimal (especially for pulling data) and logging export has no extra cost aside from the egress of the data. On Azure: you pay for data ingestion and storage in Log Analytics as usual. No need to run VMs or functions continuously. Overall, the native route avoids infrastructure costs – you’re mainly paying for the data volume ingested (which is unavoidable either way) and a tiny cost for Pub/Sub (pennies for millions of messages). Additional Costs: On top of Log Analytics ingestion costs, you will incur charges for the components you use. An Azure Function or Cloud Function has execution costs (though modest, they add up with high volume). An Event Hub has hourly charges based on throughput units and retention. Data egress from GCP to Azure will be charged by Google (network egress fees) – this also applies to the native connector, though in that case GCP egress is typically quite small cost. If your pipeline runs 24h, ensure to factor in those platform costs. Custom pipelines can also potentially reduce Log Analytics costs by filtering out data (saving money by not ingesting noise), so there’s a trade-off: spend on processing to save on storage, if needed. Support & Troubleshooting Vendor-Supported: Microsoft supports the connector – if things go wrong, you can open a support case. Documentation covers common setup issues. The connector UI will show error messages (e.g., authentication failures) to guide troubleshooting. Upgrades/improvements are handled by Microsoft (e.g., if GCP API changes, Microsoft will update the connector). Self-Support: You build it, you fix it. Debugging issues might involve checking logs across two clouds. Community forums and documentation can help (e.g., Google’s docs for Pub/Sub, Azure docs for Log Ingestion API). When something breaks, your team must identify whether it’s on the GCP side (sink or Pub/Sub) or Azure side (function error or DCR issue). This requires familiarity with both environments. There’s no single vendor to take responsibility for the end-to-end pipeline since it’s custom. In short, use the native connector whenever possible – it’s easier and reliably maintained. Opt for a custom solution only if you truly need the flexibility or to support logs that the native connectors can’t handle. Some organizations start with custom ingestion out of necessity (before native support exists) and later migrate to native connectors once available, to reduce their maintenance burden. Troubleshooting Common Issues Finally, regardless of method, you may encounter some hurdles. Here are common issues and ways to address them in the context of GCP-to-Sentinel log integration: No data appearing in Sentinel: If you’ve set up a connector and see no logs, first be patient – initial data can take ~10–30 minutes to show up. If nothing appears after that, verify the GCP side: Check the Log Router sink status in GCP (did you set the correct inclusion filters? Are logs actually being generated? You can view logs in Cloud Logging to confirm the events exist). Go to Pub/Sub and use the “Pull” option on the subscription to see if messages are piling up. If you can pull messages manually but Sentinel isn’t getting them, the issue is likely on the Azure side. In Sentinel, ensure the connector shows as Connected. If it’s in an error state, click on it to see details. A common misconfiguration is an incorrect Project Number or Service Account in the connector settings – one typo in those will prevent ingestion. Update the parameters if needed and reconnect. Authentication or Connectivity errors (native connector): These show up as errors like “Workload Identity Pool ID not found” or “Subscription does not exist” in the connector page. This usually means the values entered in the connector are mismatched: Double-check the Workload Identity Provider ID. It must exactly match the one in GCP (including correct project number). If you created the WIF pool in a different project than the Pub/Sub, remember the connector (until recently) expected them in one project. Ensure you used the correct project ID/number for all fields. Verify the service account email is correct and that you granted the Workload Identity User role on it. If not, the Azure identity cannot assume the service account. Check that the subscription name is correct and that the service account has roles/pubsub.subscriber on it. If you forgot to add that role, Azure will be denied access to Pub/Sub. Ensure the Azure AD app (which is automatically used by Sentinel) wasn’t deleted or disabled in your tenant. The Sentinel connector uses a multi-tenant app provided by Microsoft (identified by the audience GUID in the docs), which should be fine unless your org blocked third-party Azure apps. If you have restrictions, you might need to allow Microsoft’s Sentinel multi-cloud connector app. Tip: Try running the Terraform scripts provided by Microsoft if you did things manually and it’s failing. The scripts often can pinpoint what’s missing by setting everything up for you. Partial data or specific logs missing: If some expected events are not showing up: Revisit your sink filter in GCP. Perhaps the filter is too narrow. For example, for DNS logs, you might need to include both _Default logs and DNS-specific log IDs. Or for audit logs, remember that Data Access logs for certain services might be excluded by default (you have to enable Data Access audit logs in GCP for some services). If those aren’t enabled in GCP, they won’t be exported at all. If using the SCC connector, ensure you enabled continuous export of findings in SCC to Pub/Sub – those findings won’t flow unless explicitly configured. Check Sentinel’s table for any clues – sometimes logs might arrive under a slightly different table or format. E.g., if the connector was set up incorrectly initially, it might have sent data to a custom table with a suffix. Use Log Analytics query across all tables (or search by a specific IP or timestamp) to ensure the data truly isn’t there. Duplicate logs or high event counts (custom ingestion): If your custom pipeline isn’t carefully handling acknowledgments, you might ingest duplicates. For instance, if your function crashes after sending data to Sentinel but before acking Pub/Sub, the message will be retried later – resulting in the same log ingested twice. Over time this could double-count events. Solution: Ensure idempotency or proper ack logic. One way is to include a unique ID with each log (GCP audit logs have an insertId which is unique per log event; VPC flow logs have unique flowID for each flow chunk). You could use that as a de-duplication key on the Sentinel side (e.g., ingest it and deduplicate in queries). Or design the pipeline to mark messages as processed in an external store. However, the simplest is to acknowledge only after successful ingestion and let Pub/Sub handle retries. If you notice duplicates in Sentinel, double-check that your code isn’t calling ack too early or multiple times. Log Ingestion API errors (custom pipeline): When calling the Log Ingestion API, you might encounter HTTP errors: 400 Bad Request – often schema mismatch. This means the JSON you sent doesn’t match the DCR’s expected format. Check the error details; the API usually returns a message indicating which field is wrong. Common issues: sending a string value for a column defined as integer, missing a required column, or having an extra column that’s not in the table. Adjust your transformation or DCR accordingly. 403 Forbidden – authentication failure. Your Azure AD token might be expired or your app doesn’t have rights. Make sure the token is fresh (fetch a new one for each function run, or use a managed identity if supported and authorized). Also verify the app’s role assignments. 429 Too Many Requests / Throttling – you might be sending data too fast. The Log Ingestion API has throughput limits (per second per workspace). If you hit these, implement a backoff/retry and consider batching more. This is rare unless you have a very high log rate. Azure Function timeouts – if using Functions, sometimes the default timeout (e.g., 5 minutes for an HTTP-triggered function) might be hit if processing a large batch. Consider increasing the timeout setting or splitting work into smaller chunks. Connector health alerts: If you enabled the health feature for connectors, you might get alerted that “no logs received from GCP Audit Logs in last X minutes” etc. If this is a false alarm (e.g., simply that there were genuinely no new logs in GCP during that period), you can adjust the alert logic or threshold. But if it’s a real issue, treat it as an incident: check GCP’s Cloud Logging to ensure new events exist (if not, maybe nothing happened – e.g., no admin activity in the last hour). If events do exist in GCP but none in Sentinel, you have a pipeline problem – refer to the earlier troubleshooting steps for auth/connectivity. Updating or Migrating pipelines: Over time, you might replace a custom pipeline with a native connector (or vice versa). Be cautious of duplicate ingestion if both are running simultaneously. For example, if you enable the new GCP DNS connector while your custom DNS log pipeline is still on, you’ll start ingesting DNS logs twice. Plan a cutover: disable one before enabling the other in production. Also, if migrating, note that the data may land in a different table (the native connector might use GCPDNS table whereas your custom went to GCP_DNS_Custom_CL). You may need to adjust your queries and workbooks to unify this. It could be worthwhile to backfill historical data for continuity if needed. By following these practices and monitoring closely, you can ensure a successful integration of GCP logs into Microsoft Sentinel. The end result is a centralized view in Sentinel where your Azure, AWS, on-prem, and now GCP logs all reside – empowering your SOC to run advanced detections and investigations across your multi-cloud environment using a single pane of glass.
MTO Portal MFA Prompt Not Loading
Hi We are using the mto portal to hunt across multiple tenants. My team get the "loading completed with errors" message and the prompt for "MFA Login Required". When they select this the window to authenticate opens and then closes instantly. When selecting the tenant name they can authenticate in a new tab directly to Defender in this tenant without any issue (but this does not carry over to the MTO portal). The old behaviour was that they selected "MFA Login Required" and they could authenticate to the tenants they needed to at that time. Is this happening to anyone else? Does anyone have any tips for managing multiple Defender instances using MTO? ThanksFake Employees, Real Threat: Decentralized Identity to combat Deepfake Hiring?
In recent months, cybersecurity experts have sounded the alarm on a surge of fake “employees” – job candidates who are not who they claim to be. These fraudsters use everything from fabricated CVs and stolen identities to AI-generated deepfake videos in interviews to land jobs under false pretenses. It’s a global phenomenon making headlines on LinkedIn and in the press. With the topic surfacing everywhere, I wanted to take a closer look at what’s really going on — and explore the solutions that could help organizations respond to this growing challenge. And as it happens, one solution is finally reaching maturity at exactly the right moment: decentralized identity. Let me walk you through it. But first, let’s look at a few troubling facts: Even tech giants aren’t immune. Amazon’s Chief Security Officer revealed that since April 2024 the company has blocked over 1,800 suspected North Korean scammers from getting hired, and that the volume of such fake applicants jumped 27% each quarter this year (1.1). In fact, a coordinated scheme involving North Korean IT operatives posing as remote workers has infiltrated over 300 U.S. companies since 2020, generating at least $6.8 million in revenue for the regime (2.1). CrowdStrike also reported more than 320 confirmed incidents in the past year alone, marking a 220% surge in activity (2.2). And it’s not just North Korea: organised crime groups globally are adopting similar tactics. This trend is not a small blip; it’s likely a sign of things to come. Gartner predicts that by 2028, one in four job applicant profiles could be fake in some way (3). Think about that – in a few years, 25% of the people applying to your jobs might be bots or impostors trying to trick their way in. We’re not just talking about exaggerated resumes; we’re talking about full-scale deception: people hiring stand-ins for interviews, AI bots filling out assessments, and deepfake avatars smiling through video calls. It’s a hiring manager’s nightmare — no one wants to waste time interviewing bots or deepfakes — and a CISO’s worst-case scenario rolled into one. The Rise of the Deepfake Employee What does a “fake employee” actually do? In many cases, these impostors are part of organized schemes (even state-sponsored) to steal money or data. They might forge impressive résumés and create a minimal but believable online presence. During remote interviews, some have been caught using deepfake video filters – basically digital masks – to appear as someone else. In one case, Amazon investigators noticed an interviewee’s typing did not sync with the on-screen video (the keystrokes had a 110ms lag); it turned out to be a North Korean hacker remotely controlling a fake persona on the video call (1.2). Others refuse video entirely, claiming technical issues, so you only hear a voice. Some even hire proxy interviewees – a real person who interviews in their place. The level of creativity is frightening. Once inside, a fake employee can do serious damage. They gain legitimate access to internal systems, data, and tools. Some have stolen sensitive source code and threatened to leak it unless the company paid a ransom (1). Others quietly set up backdoor access for future cyberattacks. And as noted, if they’re part of a nation-state operation, the salary you pay them is funding adversaries. The U.S. Department of Justice recently warned that many North Korean IT workers send the majority of their pay back to the regime’s illicit weapons programs (1)(2.3). Beyond the financial angle, think of the security breach: a malicious actor is now an “insider” with an access badge. No sector is safe. While tech companies with lots of remote jobs were the first targets, the scam has expanded. According to the World Economic Forum, about half of the companies targeted by these attacks aren’t in the tech industry at all (4). Financial services, healthcare, media, energy – any business that hires remote freelancers or IT staff could be at risk. Many Fortune 500 firms have quietly admitted to Charles Carmakal (Chief Technology Officer at Google Cloud’s Mandiant) that they’ve encountered fake candidates (2.3). Brandon Wales — former Executive Director of the Cybersecurity and Infrastructure Security Agency (CISA) and now VP of Cybersecurity Strategy at SentinelOne — warned that the “scale and speed” of these operations is unlike anything seen before (2.3). Rivka Little, Chief Growth Officer at Socure, put it bluntly: “Every Fortune 100 and potentially Fortune 500 has a pretty high number of risky employees on their books” right now (1). If you’re in charge of security or IT, this should send a chill down your spine. How do you defend against an attack that walks in through your front door (virtually) with HR’s approval? It calls for rethinking some fundamental practices, which leads us to the biggest gap these scams have exposed: identity verification in the hiring process. The Identity Verification Gap in Hiring Let’s face it: traditional hiring and onboarding operate on a lot of trust. You collect a résumé, maybe call some references, do a background check that might catch a criminal record but won’t catch a well-crafted fake identity. You might ask for a copy of a driver’s license or passport to satisfy HR paperwork, but how thoroughly is it checked? And once the person is hired and given an employee account, how often do we re-confirm that person’s identity in the months or years that follow? Almost never. Now let’s look at the situation from the reverse perspective: During your last recruitment, or when you became a new vendor for a client, were you asked to send over a full copy of your ID via email? Most likely, yes. You send a scan of your passport or ID card to an HR representative or a partner’s portal, and you have no idea where that image gets stored, who can see it, or how long it will sit around. It feels uncomfortable, but we do it because we need to prove who we are. In reality, we’re making a leap of faith that the process is secure. This is the identity verification gap. Companies are trusting documents and self-assertions that can be forged, and they rarely have a way to verify those beyond a cursory glance. Fraudsters exploit this gap mercilessly. They provide fake documents that look real, or steal someone else’s identity details to pass background checks. Once they’ve cleared that initial hurdle, the organization treats them as legit. IT sets up accounts, security gives them access, and from then on the “user identity” is assumed to be genuine. Forever. Moreover, once an employee is on board, internal processes often default to trust. Need a password reset? The helpdesk might ask for your birthdate or employee ID – pieces of info a savvy attacker can learn or steal. We don’t usually ask an employee who calls IT to re-prove that they are the same person HR hired months or years ago. All of this stands in contrast to the principle of Zero Trust security that many companies are now adopting. Thanks to John Kindervag (Forrester, 2009), Zero Trust says “never trust, always verify” each access request. But how can you verify if the underlying identity was fake to start with? As part of Microsoft, we often say that “identity is the new perimeter” – meaning the primary defense line is verifying identities, not just securing network walls. If that identity perimeter is built on shaky ground (unverified people), the whole security model is weak. So, what can be done? Security leaders and even the World Economic Forum are advocating for stronger identity proofing in hiring. The WEF specifically recommends “verifiable government ID checks at multiple stages of recruitment and into employment” (4). In other words, don’t just verify once and forget it – verify early, verify often. That might mean an ID and background check when offering the job, another verification during onboarding, and perhaps periodic re-checks or at least on certain events (like when the employee requests elevated privileges). Amazon’s CSO, S. Schmidt, echoed this after battling North Korean fakes; he advised companies to “Implement identity verification at multiple hiring stages and monitor for anomalous technical behavior” as a key defense (1). Of course, doing this manually is tough. You can’t very well ask each candidate to fly in their first day just to show their passport in person, especially with global and remote workforces. That’s where technology is stepping up. Enter the world of Verified ID and decentralized identity. Enter Microsoft Entra Verified ID: proving Identity, not just Checking a Box Imagine if, instead of emailing copies of your passport to every new employer or partner, you could carry a digital identity credential that is already verified and can be trusted by others instantly. That’s the idea behind Microsoft Entra Verified ID. It’s essentially a system for issuing and verifying cryptographically-secure digital identity credentials. Let’s break down what that means in plain terms. At its core, a Verified ID credential is like a digital ID card that lives in an app on your phone. But unlike a photocopy of your driver’s license (which anyone could copy, steal or tamper with), this digital credential is signed with cryptographic keys that make it tamper-proof and verifiable. It’s based on open standards. Microsoft has been heavily involved in the development of Decentralized Identifiers (DID) and W3C Verifiable Credentials standards over the past few years (7). The benefit of standards is that this isn’t a proprietary Microsoft-only thing – it’s part of a broader move toward decentralized identity, where the user is in control of their own credentials. Here’s a real-life analogy: When you go to a bar and need to prove you’re over 18, you show your driver’s license, National ID or Passport. The bouncer checks your birth date and maybe the hologram, but they don’t photocopy your entire ID and keep it; they just verify it and hand it back. You remain in possession of your ID. Now translate that to digital interactions: with Verified ID, you could have a credential on your phone that says “Government ID verified: [Your Name], age 25”. When a verifier (like an employer or service) needs proof, you share that credential through a secure app. The verifier’s system checks the credential’s digital signature to confirm it was issued by a trusted authority (for example, a background check company or a government agency) and that it hasn’t been altered. You don’t have to send over a scan of your actual passport or reveal extra info like your full birthdate or address – the credential can be designed to reveal only the necessary facts (e.g. “is over 18” = yes). This concept is called selective disclosure, and it’s a big win for privacy. Crucially, you decide which credentials to share and with whom. You might have one that proves your legal name and age (from a government issuer), another that proves your employment status (from your employer), another that proves a certification or degree (from a university). And you only share them when needed. They can also have expiration dates or be revoked. For instance, an employment credential could automatically expire when you leave the company. This means if someone tries to use an old credential, it would fail verification – another useful security feature. Now, how do these credentials get issued in the first place? This is where the integration of our Microsoft Partner IDEMIA comes in, which was a highlight of Microsoft Ignite 2025. IDEMIA is a company you might not have heard of, but they’re a huge player in the identity world – they’re the folks behind many government ID and biometric systems (think passport chips, national ID programs, biometric border control, etc.). Microsoft announced that Entra Verified ID now integrate IDEMIA’s identity verification services. In practice, this means when you need a high-assurance credential (like proving your real identity for a job), the system can invoke IDEMIA to do a thorough check. For example, as part of a remote onboarding process, an employer using Verified ID could ask the new hire to verify their identity through IDEMIA. The new hire gets a link or prompt, and is guided to scan their official government ID and take a live selfie video. IDEMIA’s system checks that the ID is authentic (not a forgery) and matches the person’s face, doing so in a privacy-protecting way (for instance, biometric data might be used momentarily to match and then not stored long-term, depending on the service policies). This process confirms “Yes, this is Alice, and we’ve confirmed her identity with a passport and live face check.” At that point, Microsoft Entra Verified ID can issue a credential to Alice, such as “Alice – identity verified by Contoso Corp on [Date]”. Alice stores this credential in her digital wallet (for instance, the Microsoft Authenticator app). Now Alice can present that credential to apps or IT systems to prove it’s really Alice. The employer might require it to activate her accounts, or later if Alice calls IT support, they might ask her to present the credential to prove her identity for a password reset. The verification of the credential is cryptographically secure and instantaneous – the IT system just checks the digital signature. There’s no need to manually pull up Alice’s passport scan from HR files or interrogate her with personal questions. Plus, Alice isn’t repeatedly sending sensitive personal documents; she shared them once with a trusted verifier (IDEMIA via the Verified ID app flow), not with every individual who asks for ID. This reduces the exposure of her personal data. From the company’s perspective, this approach dramatically improves security and streamlines processes. During onboarding, it’s actually faster to have someone go through an automated ID verification flow than to coordinate an in-person verification or trust slow manual checks. Organizations also avoid collecting and storing piles of personal documents, which is a compliance headache and a breach risk. Instead, they get a cryptographic assurance. Think of it like the difference between keeping copies of everyone’s credit card versus using a payment token – the latter is safer and just as effective for the transaction. Microsoft has been laying the groundwork for this for years. Back in 2020 (and even 2017....), Microsoft discussed decentralized identity concepts where users own their identity data and apps verify facts about you through digital attestations (7). Now it’s reality: Entra Verified ID uses those open standards (DID and Verifiable Credentials) under the hood. Plus, the integration with IDEMIA and others means it’s not just theoretical — it’s operational and scalable. As Ankur Patel, one of our product leaders for Microsoft Entra, said about these integrations: it enables “high assurance verification without custom business contracts or technical implementations” (6). In other words, companies can now easily plug this capability in, rather than building their own verification processes from scratch. Before moving on, let’s not forget to include the promised quote from IDEMIA’s exec that really underscores the value: “With more than 40 years of experience in identity issuance, verification and advanced biometrics, our collaboration with Microsoft enables secure authentication with verified identities organizations can rely on to ensure individuals are who they claim to be and critical services can be accessed seamlessly and securely.” – Amit Sharma, Head of Digital Strategy, IDEMIA (6) That quote basically says it all: verified identities that organizations can rely on, enabling seamless and secure access. Now, let’s see how that translates into real-world usage. Use Cases and Benefits: From Onboarding to Recovery How can Verified ID (plus IDEMIA’s) be applied in day-to-day business? There are several high-impact use cases: Remote Employee Onboarding (aka Hire with Confidence): This is the most straightforward scenario. When bringing in a new hire you haven’t met in person, you can integrate an identity verification step. As described earlier, the new employee verifies their government ID and face once, gets a credential, and uses that to start their work. The hiring team can trust that “this person is real and is who they say they are.” This directly prevents many fake-employee scams. In fact, some companies have already tried informal versions of this: The Register reported a story of an identity verification company (ironically) who, after seeing suspicious candidates, told one applicant “next interview we’ll do a document verification, it’s easy, we’ll send you a barcode to scan your ID” – and that candidate never showed up for the next round because they knew they’d be caught (1). With Verified ID, this becomes a standard, automated part of the process, not an ad-hoc test. As a bonus, the employee’s Verified ID credential can also speed up IT onboarding (auto-provisioning accounts when the verified credential is presented) and even simplify things like proving work authorization to other services (think how you often have to send copies of IDs to benefits providers or background screeners – a credential could replace that). The new hire starts faster, and with less anxiety because they know there’s a strong proof attached to their identity, and the company has less risk from day one. Oh, and HR isn’t stuck babysitting sensitive documents – governance and privacy risk go down. Stronger Helpdesk and Support Authentication: Helpdesk fraud is a common way attackers exploit weak verification. Instead of asking employees for their first pet’s name or a short code (which an attacker might phish), support can use Verified ID to confirm the person’s identity. For example, if someone calls IT saying “I’m locked out of my account,” the support portal can send a push notification asking the user to present their Verified Employee credential or do a quick re-verify via the app. If the person on the phone is an impostor, they’ll fail this check. If it’s the real employee, it’s an easy tap on their phone to prove it’s them. This approach secures processes like password resets, unlocking accounts, or granting temporary access. Think of it as caller-ID on steroids. Instead of taking someone’s word that “I am Alice from Finance,” the system actually asks for proof. And because the proof is cryptographically verified, it’s much harder to trick than a human support agent with a sob story. This reduces the burden on support too – less time playing detective with personal questions, more confidence in automating certain requests. Account Recovery and On-Demand Re-Verification: We’ve all dealt with the hassle of account recovery when we lose a password or device. Often it’s a weak link: backup codes, personal Q&A, the support team asking some manager who can’t even tell if it’s really you, or asking for a copy of your ID… With Verified ID, organizations can offer a secure self-service recovery that doesn’t rely on shared secrets. For instance, if you lose access to your multi-factor auth and need to regain entry, you could be prompted to verify your identity with a government ID check through the Verified ID system. Once you pass, you might be allowed to reset your authentication methods. Microsoft is already moving in this direction – there’s talk of replacing security questions with Verified ID checks for Entra ID account recovery (6). The benefit here is you get high assurance that the person recovering the account is the legitimate owner. This is especially important for administrators or other highly privileged users. And it’s still faster for the user than, say, waiting days for IT to manual vet and approve a request. Additionally, companies could have policies where every X months, employees might get a prompt to reaffirm their identity if they’re engaging in sensitive work. It keeps everyone honest and catches any anomalies (like, imagine an attacker somehow compromised an account – when faced with an unexpected ID check, they wouldn’t be able to comply, raising a red flag). Step-Up Authentication for Sensitive Actions: Not every action an employee takes needs this level of verification, but some absolutely do. For example, a finance officer making a $10 million wire transfer, or an engineer pushing code to a production environment, or an HR admin downloading an entire employee database – these could all trigger a step-up authentication that includes verifying the user’s identity credential. In practice, the user might get a pop-up saying “Please present your Verified ID to continue.” It might even ask for a quick fresh selfie depending on the sensitivity, which can be matched against the one on file (using Face Match in a privacy-conscious way). This is like saying: “We know you logged in with your password and MFA earlier, but this action is so critical that we want to double-check you are still the one executing it – not someone who stole your session or is using your computer.” It’s analogous to how some banks send a one-time code for high-value transactions, but instead of just a code (which could be stolen), it’s verifying you. This dramatically reduces the risk of insider threats and account takeovers causing catastrophic damage. And for the user, it’s usually a simple extra step that they’ll understand the importance of, especially in high-stakes fields. It builds trust – both that the company trusts them enough to give access, but also verifies them to ensure no one is impersonating them. In all these cases, Verified ID is adding security without a huge usability cost. In fact, many users might prefer it to the status quo: I’d rather verify my identity once properly than have to answer a bunch of security questions or have an IT person eyeballing my ID over a grainy video call. It also introduces transparency and control. As an employee, if I’m using a Verified ID, I know exactly what credential I’m sharing and why, and I have a log of it. It’s not an opaque process where I send documents into a void. From a governance perspective, using Verified ID means less widespread personal data to protect, and a clearer audit trail of “this action was taken by Alice, whose identity was verified by method X at time Y.” It can even help with regulatory compliance – for instance, proving that you really know who has access to sensitive financial data (important for things like SOX compliance or other audits). And circling back to the theme of fake employees, if such a system is in place, it’s a massive deterrent. The barrier to entry for fraudsters becomes much higher. It’s not impossible (nothing is, and you still need to Assume breach), but now they’d have to fool a top-tier document verification and biometric check – not just an overworked recruiter. That likely requires physical presence and high-quality fake documents, which are riskier and more costly for attackers. The more companies adopt such measures, the less “return on investment” these hiring scams will have for cybercriminals. The Bigger Picture: Verified Identity as the New Security Frontier The convergence of trends here is interesting. On one hand, we have digital transformation and remote work which opened the door to these novel attacks. On the other hand, we have new security philosophies like Zero Trust that emphasize continuous verification of identity and context. Verified ID is essentially Zero Trust for the hiring and identity side of things: “never trust an identity claim, always verify it.” What’s exciting is that this can now be done without turning the enterprise into a surveillance state or creating unbearable friction for legitimate users. It leverages cryptography and user-centric design to raise security and preserve privacy. Microsoft’s involvement in decentralized identity and the integration of partners like IDEMIA signals that this approach is maturing. It’s moving from pilot projects to being built into mainstream products (Entra ID, Microsoft 365, LinkedIn even offers verification badges via Entra Verified ID now (5)). It’s worth noting LinkedIn’s angle here: job seekers can verify where they work or their government ID on their LinkedIn profile, which could also help employers spot fakes (though it’s voluntary and early-stage). For CISOs and identity architects, Verified ID offers a concrete tool to address what was previously a very squishy problem. Instead of just crossing your fingers that employees are who they say they are, you can enforce it. It’s analogous to the evolution of payments security: we moved from signatures (which were rarely checked) to PIN codes and chips, and now to contactless cryptographic payments. Hiring and access management can undergo a similar upgrade from assumption-based to verification-based. Of course, adopting Verified ID or any new identity tech requires planning. Organizations will need to update their onboarding processes, train HR and IT staff on the new procedure, and ensure employees are comfortable with it. Privacy considerations must be addressed (e.g., clarify that biometric data used for verification isn’t stored indefinitely, etc.). But compared to the alternative – doing nothing and hoping to avoid being the next company in a scathing news headline about North Korean fake workers – the effort is worthwhile. In summary, human identity has become the new primary perimeter for cybersecurity. We can build all the firewalls and endpoint protections we want, but if a malicious actor can legitimately log in through the front door as an employee, those defenses may not matter. Verified identity solutions like Microsoft Entra Verified ID (with partners like IDEMIA) provide a way to fortify that perimeter with strong, real-time checks. They bring trust back into remote interactions by shifting from “trust by default” to “trust because verified.” This is not just a theoretical future; it’s happening now. As of late 2025, these tools are generally available and being rolled out in enterprises. Early adopters will likely be those in highly targeted sectors or with regulatory pressures – think defense contractors, financial institutions, and tech companies burned by experience. But I suspect it will trickle into standard best practices over the next few years, much like multi-factor authentication did. The fight against fake employees and deepfake hiring scams will continue, and attackers will evolve new tricks (perhaps trying to fake the verifications themselves). But having this layer in place tilts the balance back in favor of the defenders. It forces attackers to take more risks and expend more resources, which in turn dissuades many from even trying. To end on a practical note: If you’re a security decision-maker, now is a good time to evaluate your organization’s hiring and identity verification practices. Conduct a risk assessment – do you have any way to truly verify a new remote hire’s identity? How confident are you that all your current employees are real? If those questions make you uncomfortable, it’s worth looking into solutions like Verified ID. We’re entering an era where digital identity proofing will be as standard as background checks in HR processes. The technology has caught up to the threat, and embracing it could save your company from a very costly “lesson learned.” Remember: trust is good, but verified trust is better. By making identity verification a seamless part of the employee lifecycle, we can help ensure that the only people on the payroll are the ones we intended to hire. In a world of sophisticated fakes, that confidence is priceless. Sources: (1.1) The Register – Amazon blocked 1,800 suspected North Korean scammers seeking jobs (Dec 18, 2025) – S. Schmidt comments on DPRK fake workers and advises multi-stage identity verification. https://www.theregister.com/2025/12/18/amazon_blocked_fake_dprk_workers ("We believe, at this point, every Fortune 100 and potentially Fortune 500 has a pretty high number of risky employees on their books" Socure Chief Growth Officer Rivka Little) & https://www.linkedin.com/posts/stephenschmidt1_over-the-past-few-years-north-korean-dprk-activity-7407485036142276610-dot7 (“Implement identity verification at multiple hiring stages and monitor for anomalous technical behavior”, Amazon’s CSO, S. Schmidt) | (1.2) Heal Security – Amazon Catches North Korean IT Worker by Tracking Tiny 110ms Keystroke Delays (Dec 19, 2025). https://healsecurity.com/amazon-catches-north-korean-it-worker-by-tracking-tiny-110ms-keystroke-delays/ (2.1) U.S. Department of Justice – “Charges and Seizures Brought in Fraud Scheme Aimed at Denying Revenue for Workers Associated with North Korea” (May 16, 2024). https://www.justice.gov/usao-dc/pr/charges-and-seizures-brought-fraud-scheme-aimed-denying-revenue-workers-associated-north | (2.2) PCMag – “Remote Scammers Infiltrate 300+ Companies” (Aug 4, 2025). https://www.pcmag.com/news/is-your-coworker-a-north-korean-remote-scammers-infiltrate-300-plus-companies | (2.3) POLITICO – Tech companies have a big remote worker problem: North Korean operatives (May 12 2025). https://www.politico.com/news/2025/05/12/north-korea-remote-workers-us-tech-companies-00340208 ("I’ve talked to a lot of CISOs at Fortune 500 companies, and nearly every one that I’ve spoken to about the North Korean IT worker problem has admitted they’ve hired at least one North Korean IT worker, if not a dozen or a few dozen,” Charles Carmakal, Chief Technology Officer at Google Cloud’s Mandiant) & North Koreans posing as remote IT workers infiltrated 136 U.S. companies (Nov 14, 2025). https://www.politico.com/news/2025/11/14/north-korean-remote-work-it-scam-00652866 HR Dive – By 2028, 1 in 4 candidate profiles will be fake, Gartner predicts (Aug 8, 2025) – Gartner research highlighting rising candidate fraud and 25% fake profile forecast. https://www.hrdive.com/news/fake-job-candidates-ai/757126/ World Economic Forum – Unmasking the AI-powered, remote IT worker scams threatening businesses (Dec 15, 2025) – Overview of deepfake hiring threats; recommends government ID checks at multiple hiring stages. https://www.weforum.org/stories/2025/12/unmasking-ai-powered-remote-it-worker-scams-threatening-businesses-worldwide/ The Verge – LinkedIn gets a free verified badge that lets you prove where you work (Apr 2023) – Describes LinkedIn’s integration with Microsoft Entra for profile verification. https://www.theverge.com/2023/4/12/23679998/linkedin-verification-badge-system-clear-microsoft-entra Microsoft Tech Community – Building defense in depth: Simplifying identity security with new partner integrations (Nov 24, 2025 by P. Nrisimha) – Microsoft Entra blog announcing Verified ID GA, includes IDEMIA integration and quotes (Amit Sharma, Ankur Patel). https://techcommunity.microsoft.com/t5/microsoft-entra-blog/building-defense-in-depth-simplifying-identity-security-with-new/ba-p/4468733 & https://www.linkedin.com/posts/idemia-public-security_synced-passkeys-and-high-assurance-account-activity-7407061181879709696-SMi7 & https://www.linkedin.com/posts/4ankurpatel_synced-passkeys-and-high-assurance-account-activity-7406757097578799105-uFZz ("high assurance verification without custom business contracts or technical implementations", Ankur Patel) Microsoft Entra Blog – Building trust into digital experiences with decentralized identities (June 10, 2020 by A. Simons & A. Patel) – Background on Microsoft’s approach to decentralized identity (DID, Verifiable Credentials). https://techcommunity.microsoft.com/t5/microsoft-entra-blog/building-trust-into-digital-experiences-with-decentralized/ba-p/1257362 & Decentralized digital identities and blockchain: The future as we see it. https://www.microsoft.com/en-us/microsoft-365/blog/2018/02/12/decentralized-digital-identities-and-blockchain-the-future-as-we-see-it/ & Partnering for a path to digital identity (Janv 22, 2018) https://blogs.microsoft.com/blog/2018/01/22/partnering-for-a-path-to-digital-identity/ About the Author I'm Samuel Gaston-Raoul, Partner Solution Architect at Microsoft, working across the EMEA region with the diverse ecosystem of Microsoft partners—including System Integrators (SIs) and strategic advisory firms, Independent Software Vendors (ISVs) / Software Development Companies (SDCs), and Startups. I engage with our partners to build, scale, and innovate securely on Microsoft Cloud and Microsoft Security platforms. With a strong focus on cloud and cybersecurity, I help shape strategic offerings and guide the development of security practices—ensuring alignment with market needs, emerging challenges, and Microsoft’s product roadmap. I also engage closely with our product and engineering teams to foster early technical dialogue and drive innovation through collaborative design. Whether through architecture workshops, technical enablement, or public speaking engagements, I aim to evangelize Microsoft’s security vision while co-creating solutions that meet the evolving demands of the AI and cybersecurity era.
Unifying AWS and Azure Security Operations with Microsoft Sentinel
The Multi-Cloud Reality Most modern enterprises operate in multi-cloud environments using Azure for core workloads and AWS for development, storage, or DevOps automation. While this approach increases agility, it also expands the attack surface. Each platform generates its own telemetry: Azure: Activity Logs, Defender for Cloud, Entra ID sign-ins, Sentinel analytics AWS: CloudTrail, GuardDuty, Config, and CloudWatch Without a unified view, security teams struggle to detect cross-cloud threats promptly. That’s where Microsoft Sentinel comes in, bridging Azure and AWS into a single, intelligent Security Operations Center (SOC). Architecture Overview Connect AWS Logs to Sentinel AWS CloudTrail via S3 Connector Enable the AWS CloudTrail connector in Sentinel. Provide your S3 bucket and IAM role ARN with read access. Sentinel will automatically normalize logs into the AWSCloudTrail table. AWS GuardDuty Connector Use the AWS GuardDuty API integration for threat detection telemetry. Detected threats, such as privilege escalation or reconnaissance, appear in Sentinel as the AWSGuardDuty table. Normalize and Enrich Data Once logs are flowing, enrich them to align with Azure activity data. Example KQL for mapping CloudTrail to Sentinel entities: AWSCloudTrail | extend AccountId = tostring(parse_json(Resources)[0].accountId) | extend User = tostring(parse_json(UserIdentity).userName) | extend IPAddress = tostring(SourceIpAddress) | project TimeGenerated, EventName, User, AccountId, IPAddress, AWSRegion Then correlate AWS and Azure activities: let AWS = AWSCloudTrail | summarize AWSActivity = count() by User, bin(TimeGenerated, 1h); let Azure = AzureActivity | summarize AzureActivity = count() by Caller, bin(TimeGenerated, 1h); AWS | join kind=inner (Azure) on $left.User == $right.Caller | where AWSActivity > 0 and AzureActivity > 0 | project TimeGenerated, User, AWSActivity, AzureActivity Automate Cross-Cloud Response Once incidents are correlated, Microsoft Sentinel Playbooks (Logic Apps) can automate your response: Example Playbook: “CrossCloud-Containment.json” Disable user in Entra ID Send a command to the AWS API via Lambda to deactivate IAM key Notify SOC in Teams Create ServiceNow ticket POST https://api.aws.amazon.com/iam/disable-access-key PATCH https://graph.microsoft.com/v1.0/users/{user-id} { "accountEnabled": false } Build a Multi-Cloud SOC Dashboard Use Sentinel Workbooks to visualize unified operations: Query 1 – CloudTrail Events by Region AWSCloudTrail | summarize Count = count() by AWSRegion | render barchart Query 2 – Unified Security Alerts union SecurityAlert, AWSGuardDuty | summarize TotalAlerts = count() by ProviderName, Severity | render piechart Scenario Incident: A compromised developer account accesses EC2 instances on AWS and then logs into Azure via the same IP. Detection Flow: CloudTrail logs → Sentinel detects unusual API calls Entra ID sign-ins → Sentinel correlates IP and user Sentinel incident triggers playbook → disables user in Entra ID, suspends AWS IAM key, notifies SOC Strengthen Governance with Defender for Cloud Enable Microsoft Defender for Cloud to: Monitor both Azure and AWS accounts from a single portal Apply CIS benchmarks for AWS resources Surface findings in Sentinel’s SecurityRecommendations table
Step-by-Step Guide: Integrating Microsoft Purview with Azure Databricks and Microsoft Fabric
Co-Authored By: aryananmol, laurenkirkwood and mmanley This article provides practical guidance on setup, cost considerations, and integration steps for Azure Databricks and Microsoft Fabric to help organizations plan for building a strong data governance framework. It outlines how Microsoft Purview can unify governance efforts across cloud platforms, enabling consistent policy enforcement, metadata management, and lineage tracking. The content is tailored for architects and data leaders seeking to execute governance in scalable, hybrid environments. Note: this article focuses mainly on Data Governance features for Microsoft Purview. Why Microsoft Purview Microsoft Purview enables organizations to discover, catalog, and manage data across environments with clarity and control. Automated scanning and classification build a unified view of your data estate enriched with metadata, lineage, and sensitivity labels, and the Unified Catalog gives business-friendly search and governance constructs like domains, data products, glossary terms, and data quality. Note: Microsoft Purview Unified Catalog is being rolled out globally, with availability across multiple Microsoft Entra tenant regions; this page lists supported regions, availability dates, and deployment plans for the Unified Catalog service: Unified Catalog Supported Regions. Understanding Data Governance Features Cost in Purview Under the classic model: Data Map (Classic), users pay for an “always-on” Data Map capacity and scanning compute. In the new model, those infrastructure costs are subsumed into the consumption meters – meaning there are no direct charges for metadata storage or scanning jobs when using the Unified Catalog (Enterprise tier). Essentially, Microsoft stopped billing separately for the underlying data map and scan vCore-hours once you opt into the new model or start fresh with it. You only incur charges when you govern assets or run data processing tasks. This makes costs more predictable and tied to governance value: you can scan as much as needed to populate the catalog without worrying about scan fees and then pay only for the assets you actively manage (“govern”) and any data quality processes you execute. In summary, Purview Enterprise’s pricing is usage-based and divided into two primary areas: (1) Governed Assets and (2) Data Processing (DGPUs). Plan for Governance Microsoft Purview’s data governance framework is built on two core components: Data Map and Unified Catalog. The Data Map acts as the technical foundation, storing metadata about assets discovered through scans across your data estate. It inventories sources and organizes them into collections and domains for technical administration. The Unified Catalog sits on top as the business-facing layer, leveraging the Data Map’s metadata to create a curated marketplace of data products, glossary terms, and governance domains for data consumers and stewards. Before onboarding sources, align Unified Catalog (business-facing) and Data Map (technical inventory) and define roles, domains, and collections so ownership and access boundaries are clear. Here is a documentation that covers roles and permissions in Purview: Permissions in the Microsoft Purview portal | Microsoft Learn. The imageabove helps understand therelationship between the primary data governance solutions, Unified Catalog and Data Map, and the permissions granted by the roles for each solution. Considerations and Steps for Setting up Purview Steps for Setting up Purview: Step 1: Create a Purview Account. In the Azure Portal, use the search bar at the top to navigate to Microsoft Purview Accounts. Once there, click “Create”. This will take you to the following screen: Step 2: Click Next: Configuration and follow the Wizard, completing the necessary fields, including information on Networking, Configurations, and Tags. Then click Review + Create to create your Purview account. Consideration: Private networking: Use Private Endpoints to secure Unified Catalog/Data Map access and scan traffic; follow the new platform private endpoints guidance in the Microsoft Purview portal or migrate classic endpoints. Once your Purview Account is created, you’ll want to set up and manage your organization’s governance strategy to ensure that your data is classified and managed according to the specific lifecycle guidelines you set. Note: Follow the steps in this guide to set up Microsoft Purview Data Lifecycle Management: Data retention policy, labeling, and records management. Data Map Best Practices Design your collections hierarchy to align with organizational strategy—such as by geography, business function, or data domain. Register each data source only once per Purview account to avoid conflicting access controls. If multiple teams consume the same source, register it at a parent collection and create scans under subcollections for visibility. The imageaboveillustrates a recommended approach for structuring your Purview DataMap. Why Collection Structure Matters A well-structured Data Map strategy, including a clearly defined hierarchy of collections and domains, is critical because the Data Map serves as the metadata backbone for Microsoft Purview. It underpins the Unified Catalog, enabling consistent governance, role-based access control, and discoverability across the enterprise. Designing this hierarchy thoughtfully ensures scalability, simplifies permissions management, and provides a solid foundation for implementing enterprise-wide data governance. Purview Integration with Azure Databricks Databricks Workspace Structure In Azure Databricks, each region supports a single Unity Catalog metastore, which is shared across all workspaces within that region. This centralized architecture enables consistent data governance, simplifies access control, and facilitates seamless data sharing across teams. As an administrator, you can scan one workspace in the region using Microsoft Purview to discover and classify data managed by Unity Catalog, since the metastore governs all associated workspaces in a region. If your organization operates across multiple regions and utilizes cross-region data sharing, please review the consideration and workaround outlined below to ensure proper configuration and governance. Follow pre-requisite requirements here, before you register your workspace: Prerequisites to Connect and manage Azure Databricks Unity Catalog in Microsoft Purview. Steps to Register Databricks Workspace Step 1: In the Microsoft Purview portal, navigate to the Data Map section from the left-hand menu. Select Data Sources. Click on Register to begin the process of adding your Databricks workspace. Step 2: Note: There are two Databricks data sources, please review documentation here to review differences in capability: Connect to and manage Azure Databricks Unity Catalog in Microsoft Purview | Microsoft Learn. You can choose either source based on your organization’s needs. Recommended is “Azure Databricks Unity Catalog”: Step 3: Register your workspace. Here are the steps to register your data source: Steps to Register an Azure Databricks workspace in Microsoft Purview. Step 4: Initiate scan for your workspace, follow steps here: Steps to scan Azure Databricks to automatically identify assets. Once you have entered the required information test your connection and click continue to set up scheduled scan trigger. Step 5: For Scan trigger, choose whether to set up a schedule or run the scan once according to your business needs. Step 6: From the left pane, select Data Map and select your data source for your workspace. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab. Review further options here: Manage and Review your Scans. You can review your scanned data sources, history and details here: Navigate to scan run history for a given scan. Limitation: The “Azure Databricks Unity Catalog” data source in Microsoft Purview does not currently support connection via Managed Vnet. As a workaround, the product team recommends using the “Azure Databricks Unity Catalog” source in combination with a Self-hosted Integration Runtime (SHIR) to enable scanning and metadata ingestion. You can find setup guidance here: Create and manage SHIR in Microsoft Purview Choose the right integration runtime configuration Scoped scan support for Unity Catalog is expected to enter private preview soon. You can sign up here: https://aka.ms/dbxpreview. Considerations: If you have delta-shared Databricks-to-Databricks workspaces, you may have duplication in your data assets if you are scanning both Workspaces. The workaround for this scenario is as you add tables/data assets to a Data Product for Governance in Microsoft Purview, you can identify the duplicated tables/data assets using their Fully Qualified Name (FQN). To make identification easier: Look for the keyword “sharing” in the FQN, which indicates a Delta-Shared table. You can also apply tags to these tables for quicker filtering and selection. The screenshot highlights how the FQN appears in the interface, helping you confidently identify and manage your data assets. Purview Integration with Microsoft Fabric Understanding Fabric Integration: Connect Cross-Tenant: This refers to integrating Microsoft Fabric resources across different Microsoft Entra tenants. It enables organizations to share data, reports, and workloads securely between separate tenants, often used in multi-organization collaborations or partner ecosystems. Key considerations include authentication, data governance, and compliance with cross-tenant policies. Connect In-Same-Tenant: This involves connecting Fabric resources within the same Microsoft Entra tenant. It simplifies integration by leveraging shared identity and governance models, allowing seamless access to data, reports, and pipelines across different workspaces or departments under the same organizational umbrella. Requirements: An Azure account with an active subscription. Create an account for free. An active Microsoft Purview account. Authentication is supported via: Managed Identity. Delegated Authentication and Service Principal. Steps to Register Fabric Tenant Step 1: In the Microsoft Purview portal, navigate to the Data Map section from the left-hand menu. Select Data Sources. Click on Register to begin the process of adding your Fabric Tenant (which also includes PowerBI). Step 2: Add in Data Source Name, keep Tenant ID as default (auto-populated). Microsoft Fabric and Microsoft Purview should be in the same tenant. Step 3: Enter in Scan name, enable/disable scanning for personal workspaces. You will notice under Credentials automatically created identity for authenticating Purview account. Note: If your Purview is behind Private Network, follow the guidelines here: Connect to your Microsoft Fabric tenant in same tenant as Microsoft Purview. Step 4: From your Microsoft Fabric, open Settings, Click on Tenant Settings and enable “Service Principals can access read-only admin APIs”, “Enhanced admin API responses within detailed metadata” and “Enhance Admin API responses with DAX and Mashup Expressions” within Admin API Settings section. Step 5: You will need to create a group, add the Purviews' managed identity to the group and add the group under “Service Principals can access read-only admin APIs” section of your tenant settings inside Microsoft Fabric Step 6: Test your connection and setup scope for your scan. Select the required workspaces, click continue and automate a scan trigger. Step 7: From the left pane, select Data Map and select your data source for your workspace. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab. Review further options here: Manage and Review your Scans. You can review your scanned data sources, history and details here: Navigate to scan run history for a given scan. Why Customers Love Purview Kern County unified its approach to securing and governing data with Microsoft Purview, ensuring consistent compliance and streamlined data management across departments. EY accelerated secure AI development by leveraging the Microsoft Purview SDK, enabling robust data governance and privacy controls for advanced analytics and AI initiatives. Prince William County Public Schools created a more cyber-safe classroom environment with Microsoft Purview, protecting sensitive student information while supporting digital learning. FSA (Food Standards Agency) helps keep the UK food supply safe using Microsoft Purview Records Management, ensuring regulatory compliance and safeguarding critical data assets. Conclusion Purview’s Unified Catalog centralizes governance across Discovery, Catalog Management, and Health Management. The Governance features in Purview allow organizations to confidently answer critical questions: What data do we have? Where did it come from? Who is responsible for it? Is it secure and compliant? Can we trust its quality? Microsoft Purview, when integrated with Azure Databricks and Microsoft Fabric, provides a unified approach to cataloging, classifying, and governing data across diverse environments. By leveraging Purview’s Unified Catalog, Data Map, and advanced governance features, organizations can achieve end-to-end visibility, enforce consistent policies, and improve data quality. You might ask, why does data quality matter? Well, in today’s world, data is the new gold. References Microsoft Purview | Microsoft Learn Pricing - Microsoft Purview | Microsoft Azure Use Microsoft Purview to Govern Microsoft Fabric Connect to and manage Azure Databricks Unity Catalog in Microsoft Purview
Permissions to see and manage sentinel workspace in Defender XDR
Hi Team, One of my customers recently completed their Sentinel → Defender portal migration. Initially, I didn’t have access to view the Defender portal, but after the migration I was assigned the Security Operator role in Entra (via PIM), which now allows me to access the Defender portal.However, when I navigate to: Defender portal → System → Settings → Microsoft Sentinel → Workspaces. I’m unable to view the available workspaces. The portal shows an insufficient permissions error, and I also cannot switch the primary/secondary workspace. Could you please advise on the exact permissions/roles required to: View the Sentinel workspace list in Defender, and Switch the primary workspace? Thanks in advanceXDR advanced hunting region specific endpoints
Hi, I am exploring XDR advanced hunting API to fetch data specific to Microsoft Defender for Endpoint tenants. The official documentation (https://learn.microsoft.com/en-us/defender-xdr/api-advanced-hunting) mentions to switch to Microsoft Graph advanced hunting API. I had below questions related to it: 1. To fetch the region specific(US , China, Global) token and Microsoft Graph service root endpoints(https://learn.microsoft.com/en-us/graph/deployments#app-registration-and-token-service-root-endpoints ) , is the recommended way to fetch the OpenID configuration document (https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols-oidc#fetch-the-openid-configuration-document) for a tenant ID and based on the response, the region specific SERVICE/TOKEN endpoints could be fetched? Since using it, there is no need to maintain different end points for tenants in different regions. And do we use the global service URL https://login.microsoftonline.com to fetch OpenID config document for a tenantID in any region? 2. As per the documentation, Microsoft Graph Advanced hunting API is not supported in China region (https://learn.microsoft.com/en-us/graph/api/security-security-runhuntingquery?view=graph-rest-1.0&tabs=http). In this case, is it recommended to use Microsoft XDR Advanced hunting APIs(https://learn.microsoft.com/en-us/defender-xdr/api-advanced-hunting) to support all region tenants(China, US, Global)?
Secure Score - Vulnerability Exceptions Not Registering
I have followed the guide to configure the proper permissions to manage within Defender. Device groups have been created based off tags we applied to the devices, and the device groups register the expected number of devices. We apply an exception to the vulnerability recommendation based off the device group, looking at the individual device pages we can confirm the recommendation is excluded and it all appears to work as intended up to this point. The problem starts on the vulnerability dashboard. The recommendation shows it is in partial exception status however none of the statistics or data reflect this including our secure score. I can confirm making a global exception works as expected and we can see the score adjust properly. Has anyone experienced this before or have any pointers? We have been working at this for weeks trying different things without luck, we are ensuring to leave adequate sync times.
Unable to add Endpoints and Vulnerability management in XDR Permissions
Hi, I have defender for endpoint running on obver 400 devices. I have 10 with Bus Premium, 5 with E5, and the rest E3. I am getting incidents for DFE, and this is being sent to my SOAR platform for analysis, but when I pivot back using client-sync, I cannot see DFE incidents. I have gone into Settings > XDR > Workload settings, and can only see the below There does not appear to be the option to grant the roles I have provided for my SOAR user the ability to see Endpoint and Vulnerability management. Really scratching my head here. Help?
