User Profile
nsakthi
MicrosoftJoined 3 years ago
User Widgets
Recent Discussions
Re: Exporting Microsoft Purview Data Assets using the REST API
Hi Yuvaraja1 Kindly check your firewall settings and API version. Additionally, if you have Private Endpoints enabled, please verify your DNS configuration. https://docs.azure.cn/en-us/purview/migrate-to-governance-private-endpoints#:~:text=If%20you%20configured%20firewall%20allowlist%20rules%20for%20your%20account%20endpoints Configure Microsoft Purview firewall - Microsoft Purview | Azure Docs Confirm that your firewall allows these global and tenant-specific endpoints (replacing theaccountnameandtenantidwith your values): api.purview-service.microsoft.com accountname.purview.azure.com tenantid-api.purview-service.microsoft.com I hope this helps.Extracting tenantSettings via REST API from Microsoft Fabric to SharePoint directly using PowerShell
Microsoft Fabric is a unified platform that integrates data and services, encompassing data science and data lakes, to enhance your team’s data utilization. Discover how to leverage Fabric’s features like OneLake, Data Factory, Synapse, Data Activator, Power BI, and Microsoft Purview to usher your data into the AI era. Extracting tenantSettings via REST API from Microsoft Fabric to SharePoint directly using PowerShell/CloudShell. To generate a tenant settings report using REST API, you can use the following steps: Obtain an access token from Microsoft Entra (aka Azure Active Directory) by following this guide:https://docs.microsoft.com/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow Use the access token to call the Fabric REST API endpoint for tenant settings:https://learn.microsoft.com/rest/api/fabric/admin/tenants/get-tenant-settings Authenticate SharePoint Teams Site using Service Principal and Certificate/Thumbprint and stream the REST API response directly to SharePoint file. Parse the JSON response and extract the relevant information for your report. Here’s a samplePowerShell/CloudShell scriptthat can produce a report on Microsoft Fabric (including Power BI) tenant settings using the REST API: Download complete code snippet:fabric/admin/tenant_settings_SP.ps1 (github)Microsoft Fabric Tenant Settings v2 - Current & Snapshot Comparison
Preserve the API responses as JSON files either daily or weekly and use Power BI to analyze these snapshot files. This approach also enables you to subscribe to email notifications or alerts based on certain metrics as needed. Loading JSON Files: To import your local JSON files, select the JSON option in the Get Data menu and utilize the JSON connector in Power Query. This action prompts a local file browser for you to choose your JSON files. Transforming JSON Data: Power Query identifies tables within the JSON data and automatically converts them into a tabular format. You can then employ the editor to further modify the data as required, or simply close and apply. Data Analysis: Once all the JSON files have been imported and transformed into tables, Power BI’s data modeling and visualization tools can be used to analyze the data. You can formulate calculated columns, measures, and visuals to contrast the data across different JSON files. Email Subscriptions: Power BI offers the feature to regularly receive email updates for reports and dashboards. You can adjust the frequency of these emails to be daily, weekly, or triggered by data changes. This functionality allows you to stay informed with the most recent insights without the need to manually check Power BI. Alert Creation: Power BI also provides the capability to set up data-driven alerts, enabling you to receive notifications when your data surpasses predefined limits. Here are several methods for retrieving Microsoft Fabric tenant settings via the REST API Python, PowerShell and Curl. Extracting tenant settings from Microsoft Fabric using PowerShell and the REST API Extracting tenant settings from Microsoft Fabric using Python and the REST API You can obtain the Power BI Template files from this location: v1 Live/Current Tenant Settings:Microsoft Fabric Tenant Settings.pbit (github) v2 Live/Current Tenant Settings + Snapshot Settings:Microsoft Fabric Tenant Settings v2.pbit (github) v3 Snapshot Settings:Microsoft Fabric Tenant Settings v3.pbit (github) Provide the necessary parameters to the template file, save it as a PBIX file, and then publish it to the Power BI service. Provide the Tenant ID, Client ID, and Client Secret for the service principal that can access the Tenant Settings. Also, specify the folder path where the exported snapshot files are stored, so they can be read and compared. Please refer to the following documents to understand how to set up a service principal.Power BI REST APIsandEnable service principal authentication for read-only admin APIs Later, these reports can be shared with others and alerts can be created or subscribed to as required. The below is a version 1 or part 1 article that you can use as a reference.Microsoft Fabric Tenant Settings - Reports - Microsoft Community HubMicrosoft Fabric Tenant Settings - Reports
Microsoft Fabric is a unified platform that combines data and services, encompassing data science, data engineering, data lakehouses, data warehouses and visualizations, to enhance your team’s data utilization. Discover how to leverage Fabric’s features like OneLake, Data Factory, Synapse, Data Activator, Power BI, and Microsoft Purview to usher your data into the AI era. Tenant settings provide a detailed level of control over the features accessible to your organization. If you’re worried about sensitive data, some features may not be suitable for your organization, or you may want to limit certain features to specific groups. While tenant settings that govern the availability of features in the Power BI user interface can aid in setting up governance policies, they do not serve as a security measure. For instance, the ‘Export data’ setting does not limit a Power BI user’s permissions on a semantic model. Power BI users with read access to a semantic model have the right to query this model and may be able to save the results without utilizing the ‘Export data’ feature in the Power BI user interface. By extracting and externally visualizing tenant settings in Power BI reports, stakeholders can view, archive, and compare these settings with historical data. This approach negates the need for higher privileges and access to the Microsoft Fabric (previously Power BI) admin portal. Learn more about the tenant settings:About tenant settings - Microsoft FabricandTenant settings index - Microsoft Fabric To generate a tenant settings report using REST API, you can use the following steps: Obtain an access token from Microsoft Entra (aka Azure Active Directory) by following this guide: OAuth 2.0 client credentials flow on the Microsoft identity platform Use the access token to call the Fabric REST API endpoint for tenant settings: Tenants - Get Tenant Settings - REST API Parse the JSON response and extract the relevant information for your report. You can obtain the Power BI Template file from this location:Microsoft Fabric Tenant Settings.pbit (github) Enter the Tenant ID, Client ID, and Client Secret associated with the service principal that has the necessary permissions to retrieve the Tenant Settings. Please refer to the following documents to understand how to set up a service principal. Power BI REST APIsandEnable service principal authentication for read-only admin APIs Save the file as PBXI and upload it to the relevant workspace for sharing with stakeholders or co-administrators. Take advantage of the ability to capture snapshots of the settings for audit purposes and historical comparison Set up alerts for any new additions or modifications Set up automatic email exports for record-keeping Here are several methods for retrieving Microsoft Fabric tenant settings via the REST API using M query, Python, and PowerShell. Extracting tenant settings from Microsoft Fabric using PowerShell and the REST API Extracting tenant settings from Microsoft Fabric using Python and the REST APIMicrosoft Fabric (including Power BI) Workspace Deployment Pipeline with Customized Stages
Deployment Pipelines - Customized Stages Previously: Development=> Testing=> Production(Only 3 Stages) Now: POC => Dev => SIT => UAT => Preprod => Prod(Upto 10 Stages) The pipeline is initially set up with three default stages: Development, Test, and Production. You have the option to either keep these default stages or modify them according to your preferences. You can customize the number of stages and their names within a range of 2-10 stages. To add another stage, click on the "+Add" button. If needed, you can delete stages or rename them by entering a new name in the designated box. Once you have made the desired changes, click on "Create" to finalize the configuration. Create a pipeline Name the pipeline Customize the stages Assign the workspaces: Some sample customized stages 🙂 General: Practical Data Science: Data Platform: Data Application: Quality Assurance and Quality Control: Reference:Get started using deployment pipelines, the Fabric Application lifecycle management (ALM) tool - Microsoft Fabric | Microsoft LearnExtracting tenant settings from Microsoft Fabric using Python and the REST API
Microsoft Fabric is a unified platform that integrates data and services, encompassing data science and data lakes, to enhance your team’s data utilization. Discover how to leverage Fabric’s features like OneLake, Data Factory, Synapse, Data Activator, Power BI, and Microsoft Purview to usher your data into the AI era. To generate a tenant settings report using REST API, you can use the following steps: Obtain an access token from Microsoft Entra (aka Azure Active Directory) by following this guide: https://docs.microsoft.com/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow Use the access token to call the Fabric REST API endpoint for tenant settings: https://learn.microsoft.com/rest/api/fabric/admin/tenants/get-tenant-settings Parse the JSON response and extract the relevant information for your report. Here’s a sample Python script that can produce a report on Microsoft Fabric (including Power BI) tenant settings using the REST API: Tenant Settings: Download complete code snippet:fabric/admin/tenant_settings.py (github)Extracting tenant settings from Microsoft Fabric using PowerShell and the REST API
Microsoft Fabric is a unified platform that integrates data and services, encompassing data science and data lakes, to enhance your team’s data utilization. Discover how to leverage Fabric’s features like OneLake, Data Factory, Synapse, Data Activator, Power BI, and Microsoft Purview to usher your data into the AI era. To generate a tenant settings report using REST API, you can use the following steps: Obtain an access token from Microsoft Entra (aka Azure Active Directory) by following this guide:https://docs.microsoft.com/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow Use the access token to call the Fabric REST API endpoint for tenant settings:https://learn.microsoft.com/rest/api/fabric/admin/tenants/get-tenant-settings Parse the JSON response and extract the relevant information for your report. Here’s a sample PowerShell script that can produce a report on Microsoft Fabric (including Power BI) tenant settings using the REST API: Download complete code snippet:fabric/admin/tenant_settings.ps1 (github)Enhance Large Language Models (LLM)/Azure OpenAI Performance and Cost Efficiency
Basic techniques to optimize performance and cost efficiency of Large Language Models (LLM) in Azure OpenAI. Selection of Optimal Models: Choosing the right model is crucial for achieving the best performance. This involves understanding the capabilities of each model and selecting the one that best suits the task at hand. Factors to consider include the model's language capabilities, its ability to generate creative content, and its performance on tasks similar to the one you're working on. Enhancing Prompts: The quality of the prompts you provide to the model can significantly impact its performance. A well-crafted prompt can guide the model to generate more relevant and useful responses. This might involve specifying the format you want the answer in, or asking the model to think step-by-step or debate pros and cons before settling on an answer. Controlling Call Rates: To optimize costs and performance, it's important to manage the rate at which you make API calls. This involves understanding the rate limits of the API and designing your application to stay within these limits. You might need to implement a queuing system or use exponential backoff in case of rate limit errors. Provisioned Throughput Units (PTUs): PTUs allow you to reserve capacity for your application, ensuring consistent performance even during peak times. By correctly provisioning PTUs, you can balance cost and performance to meet your application's needs. Integration with Networks and Security: Azure OpenAI can be integrated with your existing network and security infrastructure. This might involve setting up Virtual Networks, using Private Link for secure communication, and managing access with Azure Active Directory. Integration with Data Sources and Cognitive Enterprise Search: Azure OpenAI can be used in conjunction with various data sources and Azure's Cognitive Search service. This allows you to build powerful applications that can search through large amounts of data, understand it, and generate human-like text based on it. Efficient Search using Vector Search: Vector search is a method of retrieving information that involves converting text into high-dimensional vectors and searching for similar vectors. This can be much more efficient than traditional search methods, especially for large datasets. Azure OpenAI includes support for vector search, allowing you to build efficient, AI-powered search applications. Utilizing Agent Pools for Different Tasks: Agent pools are a powerful feature that allows you to manage and distribute workloads across different resources. By creating separate agent pools for different tasks, you can ensure that each task has the resources it needs to run efficiently. This can help to optimize performance and reduce costs. Use Clear Meta Prompts and System Messages for Desired Outputs: Meta prompts and system messages play a crucial role in guiding the AI to produce the desired outputs. A meta prompt is a directive that instructs the AI on the format or type of response required. Use AI model combination for simple tasks and LLM for complex tasks: Different AI models have different strengths and capabilities. For simple tasks, such as answering straightforward questions or generating short pieces of text, a combination of smaller, specialized AI models might be sufficient. These models can be faster and more efficient than larger models, and can often produce high-quality results for simple tasks. It's my hope that these 10 methods will deepen your understanding of how to optimize large Language Models, thereby improving performance and cost efficiency.Exporting Microsoft Purview Data Assets using the REST API
Exporting Data Assets from Microsoft Purview using the REST API in Python enables a streamlined process to retrieve structured metadata and asset information. By leveraging the REST API and Python, users can effortlessly access and export Data Assets, ensuring a programmatic and efficient approach. The powerful combination of the REST API and Python empowers users with flexibility and automation capabilities, facilitating the extraction of Microsoft Purview Data Assets and seamless integration with various data management and analytics workflows. I performed a sample search on the Microsoft Purview governance portal using "*" as the keyword to generate a list of all data assets. The accompanying screenshot from the Purview portal serves as a reference. The generated CSV file shown below is the output obtained from Microsoft Purview using the REST API. Here's a guide on exporting data assets from Microsoft Purview using the REST API in Python. To access Microsoft Purview through the Python SDK, please ensure that you install the following PyPI libraries: pip install azure-identity pip install azure-purview-scanning pip install azure-purview-administration pip install azure-purview-catalog pip install azure-purview-account pip install azure-core pip install pandas Important Your endpoint value will be different depending on which Microsoft Purview portal you are using. Endpoint for theclassic Microsoft Purview governance portal: https://{your_purview_account_name}.purview.azure.com/ Endpoint for the New Microsoft Purview portal: https://api.purview-service.microsoft.com Scan endpoint for theclassic Microsoft Purview governance portal: https://{your_purview_account_name}.scan.purview.azure.com/ Endpoint for the New Microsoft Purview portal: https://api.scan.purview-service.microsoft.com To create a Service Principal and grant Data Reader or Data Curator access to the Service Principal at the Microsoft Purview Collection Level, please refer to the instructions provided [here]. keywords = "*" tenant_id = "<Please update the Microsoft Purview tenant ID here>" client_id = "<Please provide the updated Service Principal client ID that has access to the Microsoft Purview account>" client_secret = "<Please update the Service Principal client secret for the aforementioned client ID>" purview_endpoint = "https://<Please provide the name of the Microsoft Purview account>.purview.azure.com/" purview_scan_endpoint = "https://<Please provide the name of the Microsoft Purview account>.scan.purview.azure.com/" Retrieve the entire notebook file from [GitHub]. from azure.purview.catalog import PurviewCatalogClient from azure.identity import ClientSecretCredential from azure.core.exceptions import HttpResponseError import pandas as pd from pandas.io.json import json_normalize keywords = "*" export_csv_path = "purview_search_export.csv" keywords = "*" tenant_id = "<Please update the Microsoft Purview tenant ID here>" client_id = "<Please provide the updated Service Principal client ID that has access to the Microsoft Purview account>" client_secret = "<Please update the Service Principal client secret for the aforementioned client ID>" purview_endpoint = "https://<Please provide the name of the Microsoft Purview account>.purview.azure.com/" purview_scan_endpoint = "https://<Please provide the name of the Microsoft Purview account>.scan.purview.azure.com/" def get_credentials(): credentials = ClientSecretCredential(client_id=client_id, client_secret=client_secret, tenant_id=tenant_id) return credentials def get_catalog_client(): credentials = get_credentials() client = PurviewCatalogClient(endpoint=purview_endpoint, credential=credentials, logging_enable=True) return client body_input={ "keywords": keywords } try: catalog_client = get_catalog_client() except ValueError as e: print(e) try: response = catalog_client.discovery.query(search_request=body_input) df = pd.DataFrame(response) jdf = pd.json_normalize(df.value) jdf.to_csv(export_csv_path, index=False) except HttpResponseError as e: print(e) The provided Python notebook or script is capable of exporting the following set of columns in the output CSV file. endorsement collectionId updateTime name description displayText label sensitivityLabelId objectType isIndexed assetType @search.score updateBy qualifiedName createBy owner id entityType createTime classification Additional Reference:Exploring Purview’s REST API with Python (microsoft.com)DataBot (aka DataConcierge) with Microsoft Purview
Microsoft Purview is a PaaS service that helps organizations manage and govern their data in hybrid and multi-cloud environments. By using the enterprise data catalog, organizations can easily track and catalog their data assets and sources. With Microsoft Teams and Power Automate, it is possible to create a simple bot that can access the catalog's REST API and perform various operations within the Teams interface. This can make it easier for teams to access and manage their data without having to switch between different tools. The additional objective is to enable seamless integration and collaboration within the enterprise business users and modern digital solutions. The Architecture is simple, as Microsoft Teams will be a primary interface and connect to Microsoft Purview via REST API using Power Virtual Agent and Power Automate. It also integrates with Azure Active Directory for authentication and authorization. You can take out Power Virtual Agent if you want to optimize the cost by going in a less conversational way. An interactive conversational agent can be embedded using Power Virtual Agent or traditional bot framework with Azure Solutions and this can be published as an app and consumed via different channels such as Microsoft Teams, direct Bot interface, Web Application, Mobile App etc., I have built a POC that demonstrates the basic functionalities (search data asset & glossary) of the Data Concierge, and this can be expanded based on the use cases and I have primarily used Microsoft teams with a private chat and teams channel for better collaboration. The Microsoft Teams integration enables better collaboration with enterprise teams/business units. All teams can interact with the same Data Bot just by typing @Data Concierge<topic keywords/response> with new/existing conversations. The Microsoft Purview documentation references for REST API. How to use REST APIs for Microsoft Purview Data Planes - Microsoft Purview | Microsoft Docs Discovery - Query - REST API (Microsoft Purview) | Microsoft Docs Quick Links: Microsoft Purview: https://web.purview.azure.com/ Azure Active Directory App or Service Principal: https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/RegisteredApps or https://aad.portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/RegisteredApps Microsoft Teams: https://teams.microsoft.com/ Power Virtual Agent: https://web.powerva.microsoft.com/ Power Automate: https://flow.microsoft.com/ Azure Portal: https://portal.azure.com/ Conclusion: Microsoft Purview REST API can be integrated with any custom applications. Some relevant use cases for Microsoft Purview REST API usage: Data Bot for an enterprise user with all data sources/assets Virtual Data Engineer for Data Analyst and Data Scientist Custom app with Data Lineage and Data Map Custom data assets registration and data enrichment Workflow automation and metadata policy management with REST API Glossary search and data assets scanning Enterprise Data Lake adoption through custom bot interface You may reachout to Microsoft team if you're already using Microsoft Azure Cloud & Microsoft Teams in your environment.How do we address today's data challenge using Microsoft Purview?
Microsoft Purview is a cloud-based data governance and catalog service that helps organizations discover, understand, and manage their data. It enables users to easily search for and find data across their organization, understand the content and context of the data, and ensure that the data is being used in a compliant and secure manner. Purview uses machine learning and natural language processing to automatically generate metadata for data assets, making it easier for users to find and understand the data they need. Without the right people, processes and capabilities in place, it can be difficult to answer fundamental questions like: What data do I have? Is it trustworthy? Can people access the data needed to make the right decisions? How can I enable faster business insights? What’s my compliance exposure? Let's try to see how Microsoft Purview addresses the fundamental questions. What data do I have? Is it trustworthy? Can people access the data needed to make the right decisions? How can I enable faster business insights? What’s my compliance exposure? Hope this helps.Adjust query timeout in Azure Data Explorer client tools/interfaces
Azure Data Explorer allows you to execute KQL queries via multiple interfaces/tools. This post helps you to adjust the query timeout with a few universally used tools/interfaces. ADX Web UI Kusto Explorer REST API SDK ADX Web UI: Navigate to setting icon ( ) in the top right before the user display name. (Please follow the instructions as shown in the below image to adjust the query timeout slider) Kusto Explorer: Navigate to tools menu( ) in the top menu bar next to View. (Please follow the instructions as shown in the below image to adjust the query server timeout window) REST API: Provide a request servertimout property (properties/options) in REST API (Please follow the instructions as shown in the below image to adjust the query server timeout period) Request properties and ClientRequestProperties - Azure Data Explorer | Microsoft Docs SDK: Add ClientRequestProperties options for norequesttiomeout or servertimout (properties) in client app. (Please follow the instructions as shown in the below image to manage the query server timeout period) Request properties and ClientRequestProperties - Azure Data Explorer | Microsoft Docs Hope this helps.
