Safely activate your data estate with Microsoft Purview
60% of CDOs cite data integration challenges as a top pain-point due to lack of knowledge of where relevant data resides[1]. Companies operate on multi-platform, multi-cloud data estates making it harder than ever to seamlessly discover, secure, govern and activate data. This increases the overall complexity when enabling users to responsibly derive insights and drive business value from data. In the era of AI, data governance is no longer an afterthought, data security and data governance are now both table stakes. Data Governance is not a new concept but with the proliferation of AI and evolving regulatory landscape, data governance is critical for safeguarding data related to AI-driven business innovation. With 95% of organizations implementing or developing an AI strategy[2], customers are facing emerging governance challenges, such as: False signals: The lack of clean accurate data can cause false signals in AI which can trigger consequential business outcomes or lead to incorrect reported forecasting and regulatory fines. Time to insight: Data scientists and analysts spend 60-80% of their time on data access and preparation to feed AI initiatives which leads to staff frustration, increased OPEX, and delays in critical AI innovation priorities. Shadow innovation: Data innovation outside governance can increase business risks around data leakage, oversharing, or inaccurate outcomes. This is why federated governance has surfaced as a top priority across security and data leaders because it unlocks data innovation while maintaining appropriate data oversight to help minimize risks. Customers are seeking more unified solutions that enable data security and governance seamlessly across their complex data estate. To help customers better respond to these needs, Microsoft Purview unifies data security, data governance, and data compliance solutions across the heterogeneous data estate for the era of AI. Microsoft Purview also works closely with Microsoft Fabric to integrate capabilities that help seamlessly secure and govern data to help reduce risks associated with data activation across the Microsoft Intelligent Data Platform and across the Microsoft Cloud portfolio. Microsoft Fabricdelivers a pre-integrated and optimized SaaS environment for data teams to work faster together over secure and governed data within the Fabric environment. Combining the strengths of Microsoft Purview and Microsoft Fabric enables organizations to more confidently leverage Fabric to unlock data innovation across data engineers, analysts, data scientists, and developers whilst Purview enables data security teams to extend Purview advanced data security value and enables the central data office to extend Purview advanced data governance value across Fabric, Azure, M365, and the heterogenous data estate. Furthering this vision, today Microsoft is announcing 1. a new name for the Purview Data Governance solution, Purview Unified Catalog, to better reflect its growing catalog capabilities, 2. integration with new OneLake catalog, 3. a new data quality scan engine, 4. Purview Analytics in OneLake, and 5. expanded Data Loss Prevention (DLP) capabilities for Fabric lakehouse and semantic models. Introducing Unified Catalog: a new name for the visionary solution The Microsoft Purviewdata governance solution, made generally available in September, delivers comprehensive visibility, data confidence, and responsible innovation—for greater business value in the era of AI. The solution streamlines metadata from disparate catalogs and sources, like OneLake, Databricks Unity, and Snowflake Polaris, into a unified experience. To better reflect these comprehensive customer benefits, Microsoft Purview Data Catalog is being renamed to Microsoft Purview Unified Catalog to exemplify the growing catalog capabilities such as deeper data quality support for more cloud sources, and Purview Analytics in OneLake. Adata catalogserves as a comprehensive inventory of an organization's data assets. As the Microsoft Purview Unified Catalog continues to add on capabilities within curation, data quality, and third-party platform integration, the new Unified Catalog name reflects the current cross-cloud capability. This cross-cloud capability is illustrated in the figure below. This data product contains data assets from multiple different sources, including a Fabric lakehouse table, Snowflake Table and Azure Databricks Table. With the proper curation of analytics into data products, data users can govern data assets easier than ever. Figure 1: Curation of a data product from disparate data sources within Purview’s Unified Catalog Introducing OneLake catalog (Preview) As announced in the Microsoft Fabric blog earlier today, the OneLake catalog is a solution purpose-built for data engineers, data scientists, developers, analysts, and data consumers to explore, manage, and govern data in Fabric. The new OneLake catalog works with Purview by seamlessly connecting data assets governed by OneLake catalog into Purview Unified Catalog, enabling the central data office to centrally govern and manage data assets. The Purview Unified Catalog offers data stewards and data owners advanced capabilities for data curation, advanced data quality, end-to-end data lineage, and an intuitive global catalog that spans the data estate. For data leaders, Unified Catalog offers built-in reports for actionable insights into data health and risks and the ability to confidently govern data across the heterogeneous data estate. In figure 2, you can see how Fabric data is seamlessly curated into the Corporate Emissions Created by AI for CY2024 Data Product, built with data assets from OneLake. Figure 2: Data product curated with Fabric assets Introducing a new data quality scan engine for deeper data quality (Preview) Purview offers deeper data quality support, through a new data quality scan engine for big data platforms, including: Microsoft Fabric, Databricks Unity Catalog, Snowflake, Google Big Query, and Amazon S3, supporting open standard file and table formats. In short, this new scan engine allows businesses to centrally perform rich data quality management from within the Purview Unified Catalog. In Figure 3, you can see how users can run different data quality rules on a particular asset, in this case, a table hosted in OneLake, and when users click on “run quality scan”, the scanner runs a deep scan on the data itself, running the data quality rules in real time, and updating the quality score for that particular asset. Figure 3: Running a data quality scan on an asset living in OneLake Introducing Purview Analytics in OneLake (Preview) To further an organization’s data quality management practice, data stewards can now leverage a new Purview Analytics in OneLake capability, in preview, to extract tenant-specific metadata from the Purview Unified Catalog and publish to OneLake. This new capability enables deeper data quality and lineage investigation using the rich capabilities in Power BI within Microsoft Fabric. Figure 4: In Unified Catalog settings, a user can add self-serve analytics to Microsoft Fabric Figure 5: Curated metadata from Purview within Fabric Expanded Data Loss Prevention (DLP) capabilities for Fabric lakehouse and semantic models To broaden Purview data security features for Fabric, today we are announcing that the restrict access action in Purview DLP policies now extends to Fabric semantic models. With the restrict access action, DLP admins can configure policies to detect sensitive information in semantic models and limit access to only internal users or data owners. This control is valuable for when a Fabric tenant includes guest users and you want to limit unnecessary access to internal proprietary data. The addition of the restrict access action for Fabric semantic models augments the existing ability to detect upload of sensitive data to Fabric lakehouses announced earlier this year. Learn more about the new Purview DLP capabilities for Fabric lakehouses and semantic models in the DLP blog. Figure 6: Example of restricted access to a Fabric semantic model enforced through a Purview DLP policy. Summary With these investments in security and governance, Microsoft Purview is delivering on its vision to extend data protection customer value and innovation across your heterogenous data estate for reduced complexities and improved risk mitigation. Together Purview and Fabric set the foundations for a modern intelligent data platform with seamless security and governance to drive AI innovation you can trust. Learn more As we continue to innovate our products to expand the security and governance capabilities, check out these resources to stay informed. https://aka.ms/Try-Purview-Governance https://www.microsoft.com/en-us/security/business/microsoft-purview https://aka.ms/try-fabric [1] Top 7 Challenges in Data Integration and How to Solve Them | by Codvo Marketing | Medium [2] Microsoft internal research May 2023, N=638Creating Endpoint DLP Rules using PowerShell - Part 1
This blog is Part 1 of our multi-part series on managing Endpoint DLP Rules using PowerShell. In Part 1, we will demonstrate how we can use PowerShell to create Endpoint DLP Rules with AdvancedRule, AlertProperties and EndpointDLPRestrctions Parameter. In Part 2, we will cover the same for EndpointDLPBrowserRestrictions. Step 1: Create the text file with complex condition as per the requirements and save it. Here is a sample for reference: { "Version": "1.0", "Condition": { "Operator": "And", "SubConditions": [ { "ConditionName": "ContentContainsSensitiveInformation", "Value": [ { "Groups": [ { "Name": "Default", "Operator": "Or", "Sensitivetypes": [ { "Name": "Credit Card Number", "Mincount": 1, "Maxcount": 5, "Confidencelevel": "Low", }, { "Name": "U.S. Bank Account Number", "Mincount": 5, "Confidencelevel": "Medium", } ] } ], "Operator": "And" } ] } ] } } In the above example, we are using the conditionContent Contains Sensitive Information with SIT’s Credit Card or Bank Account Number. You can choose to add/remove additional SIT’s/conditions as needed along with the desired operator. You can also change the Confidence level to Low/Medium/High as per the requirements and update the Min/Max count. We have saved it as advancedrule.txt in our example. Note: If you do not specify the Min/Max attribute, the value is taken as any by default. In our example we have not specified the Max attribute for the Bank Account Number, hence it would take the default value i.e. Any. Here is another example: { "Version": "1.0", "Condition": { "Operator": "And", "SubConditions": [ { "ConditionName": "ContentContainsSensitiveInformation", "Value": [ { "Groups": [ { "Name": "Default", "Operator": "Or", "Labels": [ { "Name": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "Id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "Type": "Sensitivity" } ] } ], "Operator": "And" } ] }, { "ConditionName": "ContentFileTypeMatches", "Value": [ "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" ] } ] } } In this example we are using the conditionContent Contains Sensitive Level with a specific label and Content matches a specific file type. Please ensure to replace the ID’s with the appropriate values before saving the file. Step 2: Define the parameters for endpointDlpRestrictions or create a text file for complex restrictions. Here is an example for a simple restriction: $endpointDlpRestrictions = @(@{"Setting"="Print"; "Value"="Block"},@{"Setting"="RemovableMedia"; "Value"="Warn"}) In this case we are setting the Print action toBlock and Copy to removable USB Device to Warn. We can configure the value to Block/Warn/Audit as per our requirements. Here is an example to create a text file with complex condition: [ { "defaultmessage": "none", "setting": "Print", "value": "Block", "appgroup": "none", "networkLocation": [ { "priority": "1", "type": "vpn", "action": "Audit" } ], "printerGroup": [ { "priority": "1", "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "action": "Audit" } ] }, { "setting": "RequireBusinessJustification", "value": "Required" }, { "setting": "RemovableMedia", "defaultmessage": "none", "value": "Warn", "appgroup": "none" }, { "setting": "CloudEgress", "defaultmessage": "none", "cloudEgressGroup": [ { "priority": "1", "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "action": "Audit" } ], "value": "Warn", "appgroup": "none" }, { "setting": "PasteToBrowser", "defaultmessage": "none", "pasteSensitiveDomainsGroup": [ { "priority": "1", "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "action": "Audit" } ], "value": "Block", "appgroup": "none" }, { "setting": "CopyPaste", "defaultmessage": "none", "value": "Warn", "appgroup": "none", "networkLocation": [ { "priority": "1", "type": "corporateNetwork", "action": "Audit" } ] }, ] We are setting the below restrictions in the above example. The Action and restrictions can be modified as per the requirements. We have saved it as endpointdlprestrictions.txt in our example. Activity Action Network Restrictions Group Restrictions Print Block VPN is set to Audit A custom Printer Group with Action as Audit The group ID can be retrieved from the Endpoint DLP Settings using PowerShell. Make sure to update the ID before saving the file. Copy to Removable USB Device Warn Upload to restricted cloud service domain Warn A custom Sensitive service domain Group with Action as Audit The group ID can be retrieved from the Endpoint DLP Settings using PowerShell. Paste to browser Block A custom Sensitive service domain Group with Action as Audit The group ID can be retrieved from the Endpoint DLP Settings using PowerShell. Copy to clipboard Warn CorporateNetwork is set to Audit Step 3: Define the Parameters: # Define the parameters to read complex condition from the file we created in Step 1 $data = Get-Content -Path "C:\temp\advancedrule.txt" -ReadCount 0 $AdvancedRuleString = $data | Out-string # Define the parameters for the DLP rule with Simple restriction $ruleName = "Endpoint Rule - Restrict Financial Information Sharing Rule" $PolicyName = "Endpoint Policy - Restrict Financial Information Sharing" $endpointDlpRestrictions = @(@{"Setting"="Print"; "Value"="Block"},@{"Setting"="RemovableMedia"; "Value"="Block"}) $Notifyendpointuser = @{NotificationContent = "default:The sharing is blocked, please contact the helpdesk for more details" ; NotificationTitle = "default:Restricted"} $alertProperties = @{AggregationType = "SimpleAggregation" ; VolumeThreshold = "5" ; AlertBy = "Tenant"; Threshold = "15"; TimeWindow = "60"} Note: The values in bold for notification content can be changed as per the notification you would like to configure. Similarly, the values in Alert properties can also be changed to meet different requirements. Step 4 Create the DLP rule: #Create the DLP rule New-DlpComplianceRule -Name $ruleName -Policy $PolicyName -GenerateAlert admin@xxxx.onmicrosoft.com -ReportSeverityLevel "Medium" -Notifyendpointuser $Notifyendpointuser -EndpointDlpRestrictions $endpointDlpRestrictions -AlertProperties $alertProperties -AdvancedRule $AdvancedRuleString You can use the below if you want to create a DLP rule with complex EDLP Restriction: # Define the parameters to read complex condition from a file we created in Step 1 $data = Get-Content -Path "C:\temp\advancedrule.txt" -ReadCount 0 $AdvancedRuleString = $data | Out-string # Define the parameters for the DLP rule with Simple restriction $ruleName = "Endpoint Rule - Restrict Financial Information Sharing Rule" $PolicyName = "Endpoint Policy - Restrict Financial Information Sharing" $Notifyendpointuser = @{NotificationContent = "default:The sharing is blocked, please contact the helpdesk for more details" ; NotificationTitle = "default:Restricted"} $alertProperties = @{AggregationType = "SimpleAggregation" ; VolumeThreshold = "5" ; AlertBy = "Tenant"; Threshold = "15"; TimeWindow = "60"} # Create the DLP rule using the EndpointDlpRestrictions file we created in Step 2. New-DlpComplianceRule -Name $ruleName -Policy $PolicyName -GenerateAlert admin@xxxx.onmicrosoft.com -ReportSeverityLevel "Medium" -AlertProperties $alertProperties -Notifyendpointuser $Notifyendpointuser -AdvancedRule $AdvancedRuleString -EndpointDlpRestrictions (Get-Content -Raw ("C:\temp\endpointdlprestrictions.txt") | ConvertFrom-Json -AsHashtable) Note: PowerShell 7 is a must for this to work.Bulk Import Endpoint DLP Global Settings
Updating the eDLP settings can be a tedious task when managing an extensive list of Service Domains, File Path Exclusions, Unallowed apps and browsers, Unallowed Bluetooth Apps, and Network Path Exclusions. In this blog, we will demonstrate how to efficiently bulk import these settings and maintain an ongoing list. Pre-requisites Visual Studio Code with Extension to convert csv to json. We are using the below extension in our example. Step 1: Create a csv file with the required parameters and values. Here is a sample table with all the parameters for eDLP Global Settings: Setting Value Executable CloudAppMode Block CloudAppRestrictionList yahoo.com CloudAppRestrictionList hotmail.com PathExclusion /Users/*/Desktop/Folder1 PathExclusion /Users/*/Desktop/Folder2 MacPathExclusion /Users/*/Downloads/Folder1 MacPathExclusion /Users/*/Downloads/Folder2 UnallowedApp testapp1 testapp1.exe UnallowedApp testapp2 testapp2.exe UnallowedBrowser Avast Secure Browser avastbrowser.exe UnallowedBrowser Firefox firefox.exe UnallowedBluetoothApp bluetoothapp1 bluetoothapp1.exe UnallowedBluetoothApp bluetoothapp2 bluetoothapp1.exe UnallowedCloudSyncApp Notepad++ notepad++.exe EvidenceStoreSettings { "FileEvidenceIsEnabled": true, "NumberOfDaysToRetain": 30, "StorageAccounts": [ { "Name": "Test", "BlobUri": "https://test.blob.windows.core.net/" } ], "Store": "CustomerManaged" } VPNSettings { "serverAddress": [ "test.vpnus.contoso.com", "test.vpnin.contoso.com" ] } serverDlpEnabled TRUE CustomBusinessJustificationNotification 1 MacDefaultPathExclusionsEnabled TRUE AdvancedClassificationEnabled TRUE BandwidthLimitEnabled TRUE DailyBandwidthLimitInMB 1000 IncludePredefinedUnallowedBluetoothApps TRUE NetworkPathEnforcementEnabled TRUE NetworkPathExclusion \\TestShare\MyFolder NetworkPathExclusion \\TestShare\MyFolder1 You can make the necessary changes and add additional rows to add more values per setting as needed. Copy the table to a csv file, make the necessary changes, and save it. Step 2: Convert csv to json. Open the csv file in Visual Studio Code Press Ctrl + Shift + P Select convert csv to json in the pop that appears. A new file will be created in VS Code in JSON format Step 3: Remove the unwanted values. Remove the unwanted values such as below using the Find and Replace All (Replace with blank) option in VS Code and save the file in json Format. We have saved it as eDLPGlobalSettings.json in our case. , "Executable": "\n" , "Executable\r": "\r\n" , "Executable\r": "\r" \r Step 4: Validate if the value TRUE is in lower-case in the json file, if not please replace it using txt editor to lower-case and save the file. Step 5: Run the below command to update the eDLP Global Settings. Sst-PolicyConfig -EndpointDlpGlobalSettings (Get-Content -Raw ("C:\temp\eDLPGlobalSettings.json") | ConvertFrom-Json -AsHashtable) Note: Set-PolicyConfig will always override the existing data hence the recommendation is to have a running csv that can be edited, converted, and imported every time. PS: Please ensure to test it in a test environment before executing it in prod and always take a backup of the current settings before importing the new one.Creating Endpoint DLP Rules using PowerShell - Part 2
This blog is Part 2 of our multi-part series on managing Endpoint DLP Rules using PowerShell. In Part 1, we demonstrated how we can use PowerShell to create Endpoint DLP Rules with AdvancedRule, AlertProperties and EndpointDLPRestrctions Parameter. In this blog, we will cover the same for EndpointDLPBrowserRestrictions. Step 1: Create a text file with condition to restrict browser access. Here is a sample for reference: { "Version": "1.0", "Condition": { "Operator": "And", "SubConditions": [ { "ConditionName": "RestrictBrowserAccess", "Value": true } ] } } We have saved the file as advancedrule.txt in our example. Step 2: Create a text file with endpoint Dlp Browser restrictions. Here is an example for a restriction: [ { "setting": "WebPagePrint", "defaultmessage": "none", "sitegroup": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "value": "Block" }, { "setting": "WebPageCopyPaste", "defaultmessage": "none", "sitegroup": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "value": "Warn" }, { "setting": "WebPageSaveToLocal", "defaultmessage": "none", "sitegroup": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "value": "Audit" }, { "setting": "WebPagePrint", "defaultmessage": "none", "sitegroup": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "value": "Block" }, { "setting": "WebPageCopyPaste", "defaultmessage": "none", "sitegroup": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "value": "Warn" }, { "setting": "WebPageSaveToLocal", "defaultmessage": "none", "sitegroup": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "value": "Audit" } ] We are setting the below Sensitive Site Restrictions in the above example. The Action and group can be modified as per the requirements, we can also choose to add more groups and remove one out of the two. We have saved the file as EndpointDlpbrowserRestrictions.txt in our example. Note: Please ensure to replace the SiteGroupID before saving the file. Activity CustomSensitiveGroup1 Action CustomSensitiveGroup2 Action Print the site Block Block Copy the date from the site Warn Warn Save the site as local files (Save-As) Audit Audit Step 3: Define the Parameters: # Define the parameters to read condition from the file we created in Step 1 $data = Get-Content -Path "C:\temp\advancedrule.txt" -ReadCount 0 $AdvancedRuleString = $data | Out-string # Define the parameters for the DLP rule $ruleName = "Endpoint Rule – Sensitive Site Restrictions" $PolicyName = "Endpoint Policy - Sensitive Site Restrictions" $alertProperties = @{AggregationType = "SimpleAggregation" ; VolumeThreshold = "5" ; AlertBy = "Tenant"; Threshold = "15"; TimeWindow = "60"} $Notifyendpointuser = @{NotificationContent = "default:The sharing is blocked, please contact the helpdesk for more details" ; NotificationTitle = "default:Restricted"} The values in bold for notification content can be changed as per the notification you would like to configure. Similarly, the values in Alert properties can also be changed to meet different requirements. Step 4: Create the DLP rule: New-DlpComplianceRule -Name $ruleName -Policy $PolicyName -GenerateAlert admin@xxxx.onmicrosoft.com -ReportSeverityLevel "Medium" -Notifyendpointuser $Notifyendpointuser -AlertProperties $alertProperties -AdvancedRule $AdvancedRuleString -EndpointDlpbrowserRestrictions (Get-Content -Raw ("C:\temp\EndpointDlpbrowserRestrictions.txt") | ConvertFrom-Json -AsHashtable) Note: PowerShell 7 is a must for this to work.Navigating the New Frontier: Information Security in the Era of M365 Copilot
Explore the intersection of AI and security in our latest feature, where Microsoft Purview meets M365 Copilot. Dive into the critical role of sensitivity labels, advanced data classification, and encryption in shaping a secure digital workspace. Gain expert insights from industry professionals and discover practical strategies for balancing innovative AI tools with rigorous security protocols.
