Recent Discussions
Azure Virtual Desktop
Hi There Hope you are doing great! I am new to Azure and Virtual Desktop and require some help to set it up correctly. I have created and setup the Virtual Desktop but looking for an option to automatically shut it down and deallocate, there is only 1 machine running and usually 1 user connects to it. if that user is not accessing the machine it needs to be deallocated after 30 or 15 min. Is it something anyone can help? Many ThanksAnnouncing the winners of the December 2024 Innovation Challenge
The Innovation Challenge hackathon brings together developers from groups who are underrepresented in technology to solve for AI use cases from Azure customers. We’re proud to be supporting these organizations who helped to prepare the participants of our most recent hackathon: BITE-CON, Código Facilito, DIO, GenSpark, Microsoft Software and Systems Academy (MSSA), TechBridge, and Women in Cloud. In order to qualify for the hackathon, participants had to earn a Microsoft Applied Skills credential or one of these Azure certifications: Azure AI Engineer Associate,Azure Developer Associate,Azure Data Science Associate. Our goal for everyone who participates is to help them open up doors to new career opportunities by demonstrating highly in demand skills and the ability to work with a team to deliver a working proof of concept under a deadline. The winning projects worked to solve for a range of real world AI challenges. Observability for AI systems: ensure that systems operate effectively, ethically, and reliably by identifying issues like model drift, bias, performance degradation, and data quality problems. VoiceRAG: how do you implement retrieval-augmented generation (RAG), the prevailing pattern for combining language models with your own data, in a system that uses audio for input and output? Accessibility for state and local government websites: How can AI be used to ensure that both web content and documents that can be downloaded meet the Web Content Accessibility Guidelines (WCAG) international standard? Hallucination detection and context validation: How could output automatically cross-reference with a reliable knowledge base or API? How do you provide confidence scores and explanations for detected hallucinations? Role-Based content filtering for AI outputs: create an AI output moderation system that filters or adjusts generated content based on user roles and access levels to prevent misuse or exposure of restricted data. AI search innovation: our industry has only just begun to get started combining AI search with RAG. What can you build that demonstrates the possibilities for improving the ways we interact with information online? There were many very strong projects and the judges had to make some hard decisions. We’re sure that every team that submitted a project will be doing epic stuff in the near future! Here are the projects awarded by the judges First place $10,000 Azure Insight Lens: Model Monitoring and Observability a comprehensive AI model monitoring and observability solution, designed to enhance model performance and optimize efficiency Second $5,000 Edu Echo a voice-first education platform designed to help 4th, 5th, and 6th grade students excel in math and language arts AbleSphere an AI-powered educational support application that empowers students with disabilities by providing real-time, personalized assistance Third $2,500 FAITH : Framework for AI Integrity and Testing Hallucinations an Azure AI based web application used to find hallucinations and ensure integrity among various AI models and LLMs along with confidence scores, complete reasoning, detailed analytics and visualizations by comparing with external knowledge sources Content-o enables organizations, whether in the financial, health, or service sectors, to offer their employees, associates, and third parties a point of access to receive information aligned and adjusted to their roles AI Search for Agricultural Planning and Control an AI-powered assistant tailored to the Brazilian agricultural sector, adhering to local legislation We’ll have our next hackathon in March 2025! Looking forward to getting inspired by what this community can do!
Health state unavailable DomainTrustCheck failed
H, we are building an AVD environment (not for the first time) and sporadically VM's will show Health state in Host pool as unavailable, iv you click on the effected VM it will say DomainTruestCheck failed however if you log on the VM (via RDP as MTSC wont connect to it when in this state) and run domain join checks they all come back as no issue. If you shut the vm down to deallocated and turn on sometimes the same VM will come back as green available but then if you restart it, it will come back as unavailable for domain checks. We have built 4 different host pools using the store win11 multi session 23h2 some using "create Session Host Configuration" enabled (Preview) some disabled as the norm, they all do the same, we have tried removing from CA policies and sanity check on domain controller its the credentials work however if we look in sign in checks on Entra it says its failing Windows Sign in incorrect credentials, but the same credentials are being used to sign into azure sign into the VM sign into Domain controller and it never fails to log in at that point, also if the credentials were wrong surely it would fail health check every time not some of the time. Anyone any ideas, logged with Microsoft but they are on holiday for Christmas periodHow to connect ADF to AWS RDS
We are trying to connect an AZURE Data factory (ADF) to an AWS RDS (Relational Database Service)instance that does not have public access. We have a working VPN up and running however, we cannot connect over the private network to the AWS IP address/RDS server name. Looking online there is not a good way to do this without creating an additional NAT instance VM. Is there a way to do this with a cloud based ADF? Or does it make more sense to just create a self hosted ADF service either in AWS or in Azure to move the data into our data warehouse? Is it possible that fabric would have a solution for this problem?
Custom permission to enable diagnostic setting in Entra ID
Custom permissions doesnt works when tried to enable diagnostic settings, in Microsoft Entra ID portal. Error: "does not have authorisation to perform action 'microsoft.aadiam/diagnosticSettings/write' over scope '/providers/microsoft.aadiam/diagnostic Settings/resourcename" Selective permissions that I applied to user account. My approach is to use custom role specific permissions. Appreciate your help to knows the right permission required. Regards, Rajkumar
Getting error message "Invoking Azure function failed with HttpStatusCode - Unauthorized"
I have a synapse pipeline which contains a single component, an azure function activity component. The objective is to send a test JSON payload to a private endpoint using POST call. The azure function activity is configured to use the POST method and an azure function linked service has also been specified in the activity. We have a function app in premium plan, the linked service is pointing to the function app. Inside the function app, we have function which contains the main python code that makes the request. Function app stack is python and the function created inside is an HTTP trigger using V2 programming model and the authorization level selected is Function. When I debug the pipeline I am getting the error messageInvoking Azure function failed with HttpStatusCode - Unauthorized. Please support in resolving this. Thanks
Understanding Data Ingestion for Log Analytics and Sentinel Workspace
I'm trying to understand how data ingestion works for both Log Analytics and Microsoft Sentinel. Every time we notice a spike in data ingestion costs for Log Analytics, we see a similar increase in Sentinel costs as well. It seems like data is being ingested into both workspaces, potentially doubling the ingestion and driving up our costs. Can someone explain if this is expected behavior, or if there's a way to optimize and avoid duplicate data ingestion between Log Analytics and Sentinel?Android RD Client Mouse Middle Click Problems
I can use the RD Client application on my Android tablet. I can connect without any problems. I can also use it with a mouse and keyboard that I connect to my tablet via Bluetooth without any problems, but while all the buttons on the mouse work, only the middle button click works like the left mouse button instead of performing itsownfunction.Migrating Windows Server from 'Hybrid Azure AD Join' to 'Entra ID Join (Cloud-Only)'
Hi everyone, I need help with a scenario where I’m looking to move a server ( non DC ) from Microsoft Entra Hybrid Joined to cloud only (Entra ID) Joined only. The goal is to stop relying on the on-premises AD domain and make the server authenticate only through cloud-based accounts (Azure AD accounts). I’d appreciate any guidance, documentation or best practices on how to properly handle this migration while ensuring minimal disruption. Has anyone done this transition before? What steps and considerations should I be aware of in terms of network configuration, Cost.... etc? Thanks in advance for your help!Azure Course Blueprints
Overview The Course Blueprint is a comprehensive visual guide to the Azure ecosystem, integrating all the resources, tools, structures, and connections covered in the course into one inclusive diagram. It enables students to map out and understand the elements they've studied, providing a clear picture of their place within the larger Azure ecosystem. It serves as a 1:1 representation of all the topics officially covered in the instructor-led training. Links: Each icon in the blueprint has a hyperlink to the pertinent document in the learning path on Learn. Layers: You have the capability to filter layers to concentrate on segments of the course by modules. I.E.: Just day 1 of AZ-104, using filters in Visio and selecting modules 1-3 Enhanced Integration: The Visio Template+ for expert courses such as SC-100 and AZ-305 now features an additional layer that allows you to compare SC-100, AZ-500, and SC-300 within the same diagram. Similarly, you can compare any combination of AZ-305, AZ-204, AZ-700, and AZ-104 to identify differences and study gaps. Since SC-300 and AZ-500 are potential prerequisites for SC-100, and AZ-204 or AZ-104 for AZ-305, this comparison is particularly useful for understanding the extra knowledge or skills required to advance to the next level. Advantages for Students Defined Goals: The blueprint presents learners with a clear vision of what they are expected to master and achieve by the course’s end. Focused Learning: By spotlighting the course content and learning targets, it steers learners’ efforts towards essential areas, leading to more productive learning. Progress Tracking: The blueprint allows learners to track their advancement and assess their command of the course material. New Feature:A comprehensive list of topics for each slide deck is now available in a downloadable .xlsx file. Each entry includes a link to Learn and its dependencies. Download links Associate Level PDF Visio Released Updated Contents! AZ-104 Azure Administrator Associate Blueprint [PDF] Template 12/14/2023 10/28/2024 Contents AZ-204 Azure Developer Associate Blueprint [PDF] Template 11/05/2024 11/11/2024 Contents AZ-500 Azure Security Engineer Associate Blueprint [PDF] Template+ 01/09/2024 10/10/2024 Contents AZ-700 Azure Network Engineer Associate Blueprint [PDF] Template 01/25/2024 11/04/2024 Contents SC-300 Identity and Access Administrator Associate Blueprint [PDF] Template 10/10/2024 Contents Specialty PDF Visio Released Updated AZ-140 Azure Virtual Desktop Specialty Blueprint [PDF] Template 01/03/2024 02/05/2024 Expert level PDF Visio Released Updated AZ-305 Designing Microsoft Azure Infrastructure Solutions Blueprint [PDF] Template+ AZ-104 AZ-204 AZ-700 05/07/2024 12/09/2024 Contents SC-100 Microsoft Cybersecurity Architect Blueprint [PDF] Template+ AZ-500 SC-300 10/10/2024 Contents Skill based Credentialing PDF Visio Released Updated AZ-1002 Configure secure access to your workloads using Azure virtual networking Blueprint [PDF] Template 05/27/2024 Contents AZ-1003 Secure storage for Azure Files and Azure Blob Storage Blueprint [PDF] Template 02/07/2024 02/05/2024 Contents Benefits for Trainers: Trainers can follow this plan to design a tailored diagram for their course, filled with notes. They can construct this comprehensive diagram during class on a whiteboard and continuously add to it in each session. This evolving visual aid can be shared with students to enhance their grasp of the subject matter. Introduction to Course Blueprint for Trainers [10 minutes + comments] Real life demo AZ-104 Advanced Networking section [3 minutes] Visio stencilsAzure icons - Azure Architecture Center | Microsoft Learn Subscribe if you want to get notified of any update like new releases or updates. My emaililan.nyska@microsoft.com LinkedInhttps://www.linkedin.com/in/ilan-nyska/ Celebrating 30,000 Downloads! Please consider sharing your anonymous feedback <-- [~ 40 seconds to complete]
Application Gateway WAFv2 Custom Rules disappeared.
Hello All, We have a AGW with WAFv2 running. A while back we were working on adding new custom rules, but after saving the new rule, all of our existing WAF custom rules were deleted. Checking with Azure support, we came to know that the delete operation also works as a PUT operation for updating and/or deleting details. But we couldn't get a clear picture on what caused our rules to be deleted instead of adding the new rule. We are still in the process of exploring options to understand what could have caused this anomaly. Have any of you faced any such scenario(s)? Any insights or suggestions are welcome and much appreciated.How we can create SessionHost in Azure Virtual Desktop using Azure Powershell?
Hi I am having one requirement where I need to create SessionHost VMs using Command Line , I followed references on Az.DesktopVirtualization Module where we have Cmdlet like New-AzWVDHostPool. Do we have similar cmdlet to create SessionHost as well? Kindly helpHow to Sync Area and Iteration Paths Between Jira and Azure DevOps
Azure DevOps area and iteration paths do not have a direct replica on the Jira side. So to sync information between both systems, the area and iteration path data has to be mapped to a custom field in the Jira issue. For this to work, you need a customizableAI-powered integrationsolution likeExalate. This solution will help you generate the script for mapping paths and maintaining the relationships between the work item and the issue. What is an Area Path? An area path establishes a hierarchy for work items related to a specific project. It helps you group work items based on team, product, or feature. Organizations working on a product or feature can use area paths to establish a hierarchy between teams at every level of involvement. You can assign the same area path to multiple teams. What is an Iteration Path? An iteration path assigns work items at the project level based on time-related intervals. Teams can share them to keep track of ongoing projects, specifically for sprints, releases, and subreleases. When new work items are added to the Sprint backlog, they become accessible via the existing iteration path. You can add up to 300 iteration paths per team. Sync Area and Iteration Paths: Jira to Azure DevOps Use Case You can create a custom field in your Jira instance to reflect the data from the iteration and area paths. How does this help your organization? Syncing this data gives more context about the teams involved on the Azure DevOps side. It provides context about the timelines and stages of progress for the mapped projects and entities. Primary Requirements Obtaining the right information from the API on both sides. Writing or generating the correctsync rulesfor both the incoming and outgoing data. Creatingtriggersto update the custom fields on Jira automatically. Fetching the right string from the area or iteration path. How Exalate Handles Jira to Azure DevOps Syncs Exalate supports one-way andtwo-way integrationbetween Jira and Azure DevOps as well as with Zendesk, ServiceNow, Salesforce, GitHub, etc. Exalate also supports AI-poweredGroovy scriptingwith the help of a chatbot. Users can also create trigger-based integrations for real-time syncs and bulk operations. To use Exalate, first install it on both Jira and Azure DevOps. Since this use case requires scripting, you need to set up a connection in the Script Mode. To configure the sync, open Exalate in your Azure DevOps dashboard, go to the connection you want to edit and click on the “Edit connection” icon. You have two options: Outgoing sync (on the Azure DevOps side) refers to the data being sent over to Jira. Incoming sync (on the Jira side) refers to the data to be received from the work item on Azure DevOps. Outgoing Sync (Azure DevOps): Send Area and Iteration Path Details from Azure DevOps to Jira To send out the area and iteration paths from the Azure DevOps work item, use the code below: replica.areaPath = workItem.areaPath replica.iterationPath = workItem.iterationPath Thereplicaretrieves the values of the area and iteration paths from the work item and saves them as a string. On the remote side, you can store the area/iteration path in a custom field using a type string or select list. Incoming Sync (Jira): Set Area Path from Azure DevOps as a Custom Field in Jira Let’s start with the area path. The area path starts with the name of the project. For example, an Azure DevOps project called AzureProject handled by Exalate’s dev team could have an area path:AzureProject\\ExalateDev. To set the area path based on the value received from the remote side text field, use the code below: issue.customFields."Area Path".value = replica.areaPath The issue.customFields."area-path".value retrieves data from the work item and stores it in the designated custom field on Jira. Incoming Sync (Jira) Set Iteration Path from Azure DevOps as a Custom Field in Jira The iteration path shows the name of the project as well as the specific sprint. For example, an Azure DevOps project called AzureProject in the first sprint could have an area path:AzureProject//Sprint1 If you don’t set the value for the Area field in the Sync Rules, Exalate uses the default area that has the same name as the project. To set the iteration path based on the value received from the remote side text field, use the code below: issue.customFields."iPath".value = replica.iterationPath Theissue.customFields."iPath".valueretrieves data from the work item and stores it in the designated custom field on Jira. Congratulations! You have successfully mapped the area and iteration path to a Jira custom field. If you still have questions or want to see how Exalate is tailored to your specific use case, drop a comment below or reach out to our engineers.
Peering Virtual Network Access to OpenAI resources?
In tenant A, I have an existing OpenAI resource that is set to only be accessible from a virtual network, in which a VM rests in place. The VM can connect to the OpenAI resource successfully. Now to access other OpenAI resources from another tenant B, I have added a peering connection to connect the virtual network of the OpenAI resources from tenant B with the virtual network in tenant A. (the typical 10.0.0.0-10.1.0.0 example) But when trying to call the API endpoint in the VM using REST or JavaScript SDK, 403 error occurs: "Access denied due to Virtual Network/Firewall rules" The peering network is confirmed to be successfully through the network interface of the VM. Is linking to Azure OpenAI resources like this possible, or do we have to use a private endpoint?How to Create an AI Model for Streaming Data
A Practical Guide with Microsoft Fabric, Kafka and MLFlow Intro In today’s digital landscape, the ability to detect and respond to threats in real-time isn’t just a luxury—it’s a necessity. Imagine building a system that can analyze thousands of user interactions per second, identifying potential phishing attempts before they impact your users. While this may sound complex, Microsoft Fabric makes it possible, even with streaming data. Let’s explore how. In this hands-on guide, I’ll walk you through creating an end-to-end AI solution that processes streaming data from Kafka and employs machine learning for real-time threat detection. We’ll leverage Microsoft Fabric’s comprehensive suite of tools to build, train, and deploy an AI model that works seamlessly with streaming data. Why This Matters Before we dive into the technical details, let’s explore the key advantages of this approach: real-time detection, proactive protection, and the ability to adapt to emerging threats. Real-Time Processing: Traditional batch processing isn’t enough in today’s fast-paced threat landscape. We need immediate insights. Scalability: With Microsoft Fabric’s distributed computing capabilities, our solution can handle enterprise-scale data volumes. Integration: By combining streaming data processing with AI, we create a system that’s both intelligent and responsive. What We’ll Build I’ve created a practical demonstration that showcases how to: Ingest streaming data from Kafka using Microsoft Fabric’s Eventhouse Clean and prepare data in real-time using PySpark Train and evaluate an AI model for phishing detection Deploy the model for real-time predictions Store and analyze results for continuous improvement The best part? Everything stays within the Microsoft Fabric ecosystem, making deployment and maintenance straightforward. Azure Event Hub Start by creating an Event Hub namespace and a new Event Hub. Azure Event Hubs have Kafka endpoints ready to start receiving Streaming Data. Create a new Shared Access Signature and utilize the Python i have created. You may adopt the Constructor to your own idea. import uuid import random import time from confluent_kafka import Producer # Kafka configuration for Azure Event Hub config = { 'bootstrap.servers': 'streamiot-dev1.servicebus.windows.net:9093', 'sasl.mechanisms': 'PLAIN', 'security.protocol': 'SASL_SSL', 'sasl.username': '$ConnectionString', 'sasl.password': 'Endpoint=sb://<replacewithyourendpoint>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxxxxxx', } # Create a Kafka producer producer = Producer(config) # Shadow traffic generation def generate_shadow_payload(): """Generates a shadow traffic payload.""" subscriber_id = str(uuid.uuid4()) # Weighted choice for subscriberData if random.choices([True, False], weights=[5, 1])[0]: subscriber_data = f"{random.choice(['John', 'Mark', 'Alex', 'Gordon', 'Silia' 'Jane', 'Alice', 'Bob'])} {random.choice(['Doe', 'White', 'Blue', 'Green', 'Beck', 'Rogers', 'Fergs', 'Coolio', 'Hanks', 'Oliver', 'Smith', 'Brown'])}" else: subscriber_data = f"https://{random.choice(['example.com', 'examplez.com', 'testz.com', 'samplez.com', 'testsite.com', 'mysite.org'])}" return { "subscriberId": subscriber_id, "subscriberData": subscriber_data, } # Delivery report callback def delivery_report(err, msg): """Callback for delivery reports.""" if err is not None: print(f"Message delivery failed: {err}") else: print(f"Message delivered to {msg.topic()} [partition {msg.partition()}] at offset {msg.offset()}") # Topic configuration topic = 'streamio-events1' # Simulate shadow traffic generation and sending to Kafka try: print("Starting shadow traffic simulation. Press Ctrl+C to stop.") while True: # Generate payload payload = generate_shadow_payload() # Send payload to Kafka producer.produce( topic=topic, key=str(payload["subscriberId"]), value=str(payload), callback=delivery_report ) # Throttle messages (1500ms) producer.flush() # Ensure messages are sent before throttling time.sleep(1.5) except KeyboardInterrupt: print("\nSimulation stopped.") finally: producer.flush() You can run this from your Workstation, an Azure Function or whatever fits your case. Architecture Deep Dive: The Three-Layer Approach When building AI-powered streaming solutions, thinking in layers helps manage complexity. Let’s break down our architecture into three distinct layers: Bronze Layer: Raw Streaming Data Ingestion At the foundation of our solution lies the raw data ingestion layer. Here’s where our streaming story begins: A web service generates JSON payloads containing subscriber data These events flow through Kafka endpoints Data arrives as structured JSON with key fields like subscriberId, subscriberData, and timestamps Microsoft Fabric’s Eventstream captures this raw streaming data, providing a reliable foundation for our ML pipeline and stores the payloads in Eventhouse Silver Layer: The Intelligence Hub This is where the magic happens. Our silver layer transforms raw data into actionable insights: The EventHouse KQL database stores and manages our streaming data Our ML model, trained using PySpark’s RandomForest classifier, processes the data SynapseML’s Predict API enables seamless model deployment A dedicated pipeline applies our ML model to detect potential phishing attempts Results are stored in Lakehouse Delta Tables for immediate access Gold Layer: Business Value Delivery The final layer focuses on making our insights accessible and actionable: Lakehouse tables store cleaned, processed data Semantic models transform our predictions into business-friendly formats Power BI dashboards provide real-time visibility into phishing detection Real-time dashboards enable immediate response to potential threats The Power of Real-Time ML for Streaming Data What makes this architecture particularly powerful is its ability to: Process data in real-time as it streams in Apply sophisticated ML models without batch processing delays Provide immediate visibility into potential threats Scale automatically as data volumes grow Implementing the Machine Learning Pipeline Let’s dive into how we built and deployed our phishing detection model using Microsoft Fabric’s ML capabilities. What makes this implementation particularly interesting is how it combines traditional ML with streaming data processing. Building the ML Foundation First, let’s look at how we structured the training phase of our machine learning pipeline using PySpark: Training Notebook Connect to Eventhouse Load the data from pyspark.sql import SparkSession # Initialize Spark session (already set up in Fabric Notebooks) spark = SparkSession.builder.getOrCreate() # Define connection details kustoQuery = """ SampleData | project subscriberId, subscriberData, ingestion_time() """ # Replace with your desired KQL query kustoUri = "https://<eventhousedbUri>.z9.kusto.fabric.microsoft.com" # Replace with your Kusto cluster URI database = "Eventhouse" # Replace with your Kusto database name # Fetch the access token for authentication accessToken = mssparkutils.credentials.getToken(kustoUri) # Read data from Kusto using Spark df = spark.read \ .format("com.microsoft.kusto.spark.synapse.datasource") \ .option("accessToken", accessToken) \ .option("kustoCluster", kustoUri) \ .option("kustoDatabase", database) \ .option("kustoQuery", kustoQuery) \ .load() # Show the loaded data print("Loaded data:") df.show() Separate and flag Phishing payload Load it with Spark from pyspark.sql.functions import col, expr, when, udf from urllib.parse import urlparse # Define a UDF (User Defined Function) to extract the domain def extract_domain(url): if url.startswith('http'): return urlparse(url).netloc return None # Register the UDF with Spark extract_domain_udf = udf(extract_domain) # Feature engineering with Spark df = df.withColumn("is_url", col("subscriberData").startswith("http")) \ .withColumn("domain", extract_domain_udf(col("subscriberData"))) \ .withColumn("is_phishing", col("is_url")) # Show the transformed data df.show() Use Spark ML Lib to Train the model Evaluate the Model from pyspark.sql.functions import col from pyspark.ml.feature import Tokenizer, HashingTF, IDF from pyspark.ml.classification import RandomForestClassifier from pyspark.ml import Pipeline from pyspark.ml.evaluation import MulticlassClassificationEvaluator # Ensure the label column is of type double df = df.withColumn("is_phishing", col("is_phishing").cast("double")) # Tokenizer to break text into words tokenizer = Tokenizer(inputCol="subscriberData", outputCol="words") # Convert words to raw features using hashing hashingTF = HashingTF(inputCol="words", outputCol="rawFeatures", numFeatures=100) # Compute the term frequency-inverse document frequency (TF-IDF) idf = IDF(inputCol="rawFeatures", outputCol="features") # Random Forest Classifier rf = RandomForestClassifier(labelCol="is_phishing", featuresCol="features", numTrees=10) # Build the ML pipeline pipeline = Pipeline(stages=[tokenizer, hashingTF, idf, rf]) # Split the dataset into training and testing sets train_data, test_data = df.randomSplit([0.7, 0.3], seed=42) # Train the model model = pipeline.fit(train_data) # Make predictions on the test data predictions = model.transform(test_data) # Evaluate the model's accuracy evaluator = MulticlassClassificationEvaluator( labelCol="is_phishing", predictionCol="prediction", metricName="accuracy" ) accuracy = evaluator.evaluate(predictions) # Output the accuracy print(f"Model Accuracy: {accuracy}") Add Signature to AI Model from mlflow.models.signature import infer_signature from pyspark.sql import Row # Select a sample for inferring signature sample_data = train_data.limit(10).toPandas() # Create a Pandas DataFrame for schema inference input_sample = sample_data[["subscriberData"]] # Input column(s) output_sample = model.transform(train_data.limit(10)).select("prediction").toPandas() # Infer the signature signature = infer_signature(input_sample, output_sample) Run – Publish Model and Log Metric: Accuracy import mlflow from mlflow import spark # Start an MLflow run with mlflow.start_run() as run: # Log the Spark MLlib model with the signature mlflow.spark.log_model( spark_model=model, artifact_path="phishing_detector", registered_model_name="PhishingDetector", signature=signature # Add the inferred signature ) # Log metrics like accuracy mlflow.log_metric("accuracy", accuracy) print(f"Model logged successfully under run ID: {run.info.run_id}") Results and Impact Our implementation achieved: 81.8% accuracy in phishing detection Sub-second prediction times for streaming data Scalable processing of thousands of events per second Yes, that's a good start ! Now let's continue our post by explaining the deployment and operation phase of our ML solution: From Model to Production: Automating the ML Pipeline After training our model, the next crucial step is operationalizing it for real-time use. We’ve implemented one Pipeline with two activities that process our streaming data every 5 minutes: All Streaming Data Notebook # Main prediction snippet from synapse.ml.predict import MLFlowTransformer # Apply ML model for phishing detection model = MLFlowTransformer( inputCols=["subscriberData"], outputCol="predictions", modelName="PhishingDetector", modelVersion=3 ) # Transform and save all predictions df_with_predictions = model.transform(df) df_with_predictions.write.format('delta').mode("append").save("Tables/phishing_predictions") Clean Streaming Data Notebook # Filter for non-phishing data only non_phishing_df = df_with_predictions.filter(col("predictions") == 0) # Save clean data for business analysis non_phishing_df.write.format("delta").mode("append").save("Tables/clean_data") Creating Business Value What makes this architecture particularly powerful is the seamless transition from ML predictions to business insights: Delta Lake Integration: All predictions are stored in Delta format, ensuring ACID compliance Enables time travel and data versioning Perfect for creating semantic models Real-Time Processing: 5-minute refresh cycle ensures near real-time threat detection Automatic segregation of clean vs. suspicious data Immediate visibility into potential threats Business Intelligence Ready: Delta tables are directly compatible with semantic modeling Power BI can connect to these tables for live reporting Enables both historical analysis and real-time monitoring The Power of Semantic Models With our data now organized in Delta tables, we’re ready for: Creating dimensional models for better analysis Building real-time dashboards Generating automated reports Setting up alerts for security teams Real-Time Visualization Capabilities While Microsoft Fabric offers extensive visualization capabilities through Power BI, it’s worth highlighting one particularly powerful feature: direct KQL querying for real-time monitoring. Here’s a glimpse of how simple yet powerful this can be: SampleData | where EventProcessedUtcTime > ago(1m) // Fetch rows processed in the last 1 minute | project subscriberId, subscriberData, EventProcessedUtcTime This simple KQL query, when integrated into a dashboard, provides near real-time visibility into your streaming data with sub-minute latency. The visualization possibilities are extensive, but that’s a topic for another day. Conclusion: Bringing It All Together What we’ve built here is more than just a machine learning model – it’s a complete, production-ready system that: Ingests and processes streaming data in real-time Applies sophisticated ML algorithms for threat detection Automatically segregates clean from suspicious data Provides immediate visibility into potential threats The real power of Microsoft Fabric lies in how it seamlessly integrates these different components. From data ingestion through Eventhouse ad Lakehouse, to ML model training and deployment, to real-time monitoring – everything works together in a unified platform. What’s Next? While we’ve focused on phishing detection, this architecture can be adapted for various use cases: Fraud detection in financial transactions Quality control in manufacturing Customer behavior analysis Anomaly detection in IoT devices The possibilities are endless with our imagination and creativity! Stay tuned for the Git Repo where all the code will be shared ! References Get Started with Microsoft Fabric Delta Lake in Fabric Overview of Eventhouse CloudBlogger: A guide to innovative Apps with MS FabricIssues with FSLogix Profiles
Hey guys we have a AVD environment with Windows 11 23h2 Multiuser session host's in a pooled Hostpool configured. The users are connecting via ThinClients with IgelOS and the AVD Client to the AVD Desktop. We have hybrid joined session host's which worked just fine till the 12th of December. We discovered that from different locations where the users are connected the have some issues with the FSLogix Profiles. It seems that the client lost connection to the session host when using short path with udp. then the users are trying to reconnect and get a temporary profile. There was no change currently. We were able to solve the issue with just create new session host with Hydra and remove the old ones. But the issue is getting back of course. In the logs in Azure I see the error: ShortpathTransportNetworkDrop The session host lost connection to the client because of an unexpected network error in the Shortpath transport. Check your network quality and configurations, then try again. Does anybody have similar issues or know what could cause that? I checked with our Network Team and they don't see any issues on their side. Many thanks for any feedback Best regards, MarcRun Logic app if new virtual machine is created
Hello, I'm building logic app that get triggered on resource creation event by connecting it to event grid. my goal is only to run this if new vm is created however logic app get executed on every create success event. I noticed whenever there is deletion or creation on VM the logic app get triggered. Even in the event payload there is no difference between create and delete VM. how to limit the call of logic app only if new VM is created?Azure Policy - Seeking Guidance: Adding "Destination Table" to Built-In Azure Policy
Hi Team, I am sharing the existing Built-In Azure Policy that previously sent logs to Log Analytics Workspace. However, it currently lacks the option to choose "Destination Table" with selections such as "Azure Diagnostics" or "Resource Specific." I would like to include this option in the policy. Could you please advise on how to achieve this? Built-in Policy Name : Enable logging by category group for Application gateways (microsoft.network/applicationgateways) to Log Analytics Policy Code { "mode": "Indexed", "policyRule": { "if": { "allOf": [ { "field": "type", "equals": "microsoft.network/applicationgateways" }, { "anyOf": [ { "value": "[first(parameters('resourceLocationList'))]", "equals": "*" }, { "field": "location", "in": "[parameters('resourceLocationList')]" } ] } ] }, "then": { "effect": "[parameters('effect')]", "details": { "type": "Microsoft.Insights/diagnosticSettings", "evaluationDelay": "AfterProvisioning", "existenceCondition": { "allOf": [ { "count": { "field": "Microsoft.Insights/diagnosticSettings/logs[*]", "where": { "allOf": [ { "field": "Microsoft.Insights/diagnosticSettings/logs[*].enabled", "equals": "[equals(parameters('categoryGroup'), 'allLogs')]" }, { "field": "microsoft.insights/diagnosticSettings/logs[*].categoryGroup", "equals": "allLogs" } ] } }, "equals": 1 }, { "field": "Microsoft.Insights/diagnosticSettings/workspaceId", "equals": "[parameters('logAnalytics')]" } ] }, "roleDefinitionIds": [ "/providers/Microsoft.Authorization/roleDefinitions/92aaf0da-9dab-42b6-94a3-d43ce8d16293" ], "deployment": { "properties": { "mode": "incremental", "template": { "$schema": "http://schema.management.azure.com/schemas/2019-08-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { "diagnosticSettingName": { "type": "string" }, "logAnalytics": { "type": "string" }, "categoryGroup": { "type": "String" }, "resourceName": { "type": "string" } }, "variables": {}, "resources": [ { "type": "microsoft.network/applicationgateways/providers/diagnosticSettings", "apiVersion": "2021-05-01-preview", "name": "[concat(parameters('resourceName'), '/', 'Microsoft.Insights/', parameters('diagnosticSettingName'))]", "properties": { "workspaceId": "[parameters('logAnalytics')]", "logs": [ { "categoryGroup": "allLogs", "enabled": "[equals(parameters('categoryGroup'), 'allLogs')]" } ], "metrics": [] } } ], "outputs": { "policy": { "type": "string", "value": "[concat('Diagnostic setting ', parameters('diagnosticSettingName'), ' for type Application gateways (microsoft.network/applicationgateways), resourceName ', parameters('resourceName'), ' to Log Analytics ', parameters('logAnalytics'), ' configured')]" } } }, "parameters": { "diagnosticSettingName": { "value": "[parameters('diagnosticSettingName')]" }, "logAnalytics": { "value": "[parameters('logAnalytics')]" }, "categoryGroup": { "value": "[parameters('categoryGroup')]" }, "resourceName": { "value": "[field('name')]" } } } } } } }, "parameters": { "effect": { "type": "String", "metadata": { "displayName": "Effect", "description": "Enable or disable the execution of the policy" }, "allowedValues": [ "DeployIfNotExists", "AuditIfNotExists", "Disabled" ], "defaultValue": "DeployIfNotExists" }, "diagnosticSettingName": { "type": "String", "metadata": { "displayName": "Diagnostic Setting Name", "description": "Diagnostic Setting Name" }, "defaultValue": "setByPolicy-LogAnalytics" }, "categoryGroup": { "type": "String", "metadata": { "displayName": "Category Group", "description": "Diagnostic category group - none, audit, or allLogs." }, "allowedValues": [ "audit", "allLogs" ], "defaultValue": "audit" }, "resourceLocationList": { "type": "Array", "metadata": { "displayName": "Resource Location List", "description": "Resource Location List to send logs to nearby Log Analytics. A single entry \"*\" selects all locations (default)." }, "defaultValue": [ "*" ] }, "logAnalytics": { "type": "String", "metadata": { "displayName": "Log Analytics Workspace", "description": "Log Analytics Workspace", "strongType": "omsWorkspace", "assignPermissions": true } } } }
Recent Blogs
- As cloud networking grows in complexity, managing and operating these services efficiently can be tedious and time consuming. That’s where Copilot in Azure for Networking steps in, a generative AI to...Dec 27, 2024
- Co-written with Arushi Sharma, Sr. PM @ Microsoft, Azure Portal UX Overview Azure offers many Virtual Machine (VM) size options based on your workload needs. Selecting precisely which VM size is...Dec 27, 2024
