Join the Fabric Partner Community for this Week's Fabric Engineering Connection calls!
Are you a Microsoft partners that is interested in data and analytics? Be sure to join us for the next Fabric Engineering Connection calls! 🎉 The Americas & EMEA call will take place Wednesday, October 22, from 8-9 am PT and will feature presentations from Teddy Bercovitz and Gerd Saurer on Fabric Extend Workload Developer Kit, followed by a presentation on Data Protection Capabilities from Yael Biss. The APAC call is Thursday, October 23, from 1-2 am UTC/Wednesday, October 2, from 5-6 pm PT. Tamer Farag, Trilok Rajesh and Shreya Ghosh will be presenting on Modernizing Legacy Analytics & BI Platforms. This is your opportunity to learn more, ask questions, and provide feedback. To join the call, you must be a member of the Fabric Partner Community Teams channel. To join, complete the participation form at https://aka.ms/JoinFabricPartnerCommunity. We look forward to seeing you later this week!Join the Fabric Partner Community for this Week's Fabric Engineering Connection calls!
Are you a Microsoft partner that is interested in data and analytics? Be sure to join us for the next Fabric Engineering Connection calls! 🎉 Sujata Narayana will be sharing a recap of Power BI announcements from FabCon Europe, followed by the latest updates on AI Functions from Virginia Roman. The Americas & EMEA call will take place Wednesday, October 15, from 8-9 am PT and the APAC call is Thursday, October 16, from 1-2 am UTC/Wednesday, October 15, from 5-6 pm PT. This is your opportunity to learn more, ask questions, and provide feedback. To join the call, you must be a member of the Fabric Partner Community Teams channel. To join, complete the participation form at https://aka.ms/JoinFabricPartnerCommunity. We look forward to seeing you later this week!
Diagnose performance issues in Spark jobs through Spark UI.
Agenda Introduction Overview of Spark UI Navigating to Spark UI Jobs Timeline Opening Jobs Timeline Reading Event Timeline Failing Jobs or Executors Diagnosing Failing Jobs Diagnosing Failing Executors Scenario - Memory Issues Scenario - Long Running Jobs Scenario - Identifying Longest Stage Introduction Diagnosing performance issues of job using Spark UI This guide walks you through how to use the Spark UI to diagnose performance issues Overview of Spark UI Job Composition Composed of multiple stages Stages may contain more than one task Task Breakdown Tasks are broken into executors Navigating to Spark UI: Navigating to Cluster's Page Navigate to your cluster’s page: Navigating to Spark UI: Clicking Spark UI Click Spark UI: Jobs Timeline Jobs timeline The jobs timeline is a great starting point for understanding your pipeline or query. It gives you an overview of what was running, how long each step took, and if there were any failures along the way Opening Jobs Timeline Accessing the Jobs Timeline Navigate to the Spark UI Click on the Jobs tab Viewing the Event Timeline Click on Event Timeline Highlighted in red in the screenshot Example Timeline Shows driver and executor 0 being added Failing Jobs or Executors: Example of Failed Job Failed Job Example Indicated by a red status Shown in the event timeline Removed Executors Also indicated by a red status Shown in the event timeline Failing Jobs or Executors: Common Reasons for Executors Being Removed Autoscaling Expected behavior, not an error See Enable autoscaling for more details Compute configuration reference - Azure Databricks | Microsoft Learn Spot instance losses Cloud provider reclaiming your VMs Learn more about Spot instances here Executors running out of memory Diagnosing Failing Jobs: Steps to Diagnose Failing Jobs Identifying Failing Jobs Click on the failing job to access its page Reviewing Failure Details Scroll down to see the failed stage Check the failure reason Diagnosing Failing Jobs: Generic Errors You may get a generic error. Click on the link in the description to see if you can get more info: Diagnosing Failing Jobs: Memory Issues Task Failure Explanation Scroll down the page to see why each task failed Memory issue identified as the cause Scenario – Spot instance , Auto-scaling Diagnosing Failing Executors: Checking Event Log Check Event Log Identify any explanations for executor failures Spot Instances Cloud provider may reclaim spot instances Diagnosing Failing Executors: Navigating to Executors Tab Check Event Log for Executor Loss Look for messages indicating cluster resizing or spot instance loss Navigate to Spark UI Click on the Executors tab Diagnosing Failing Executors: Getting Logs from Failed Executors Here you can get the logs from the failed executors: Scenario - Memory Issues Memory Issues Common cause of problems Requires thorough investigation Quality of Code Potential source of memory issues Needs to be checked for efficiency Data Quality Can affect memory usage Must be organized correctly Spark memory issues - Azure Databricks | Microsoft Learn Identifying Longest Stage Identify the longest stage of the job Scroll to the bottom of the job’s page Locate the list of stages Order the stages by duration Identifying Longest Stage Identify the longest stage of the job Scroll to the bottom of the job’s page Locate the list of stages Order the stages by duration Stage I/O Details High-Level Data Overview Input Output Shuffle Read Shuffle Write Number of Tasks in Long Stage Identifying the number of tasks Helps in pinpointing the issue Look at the specified location to determine the number of tasks Investigating Stage Details Investigate Further if Multiple Tasks Check if the stage has more than one task Click on the link in the stage’s description Get More Info About Longest Stage Click on the link provided Gather detailed information Conclusion Potential Data Skew Issues Data skew can impact performance May cause uneven distribution of data Spelling Errors in Data Incorrect spelling can affect data processing Ensure data accuracy for optimal performance Learn More Navigate to Skew and Spill - Skew and spill - Azure Databricks | Microsoft LearnJoin the Fabric Partner Community for this Week's Fabric Engineering Connection calls!
Are you a Microsoft partner that is interested in data and analytics? Be sure to join us for the next Fabric Engineering Connection calls! 🎉 Tom Peplow will be discussing OneLake Diagnostics and Sarab Dua will be joining to cover recent releases and roadmap for network security. Both promise to be presentations you won't want to miss! The Americas & EMEA call will take place Wednesday, October 8, from 8-9 am PT and the APAC call is Thursday, October 9, from 1-2 am UTC/Wednesday, October 8, from 5-6 pm PT. This is your opportunity to learn more, ask questions, and provide feedback. To join the call, you must be a member of the Fabric Partner Community Teams channel. To join, complete the participation form at https://aka.ms/JoinFabricPartnerCommunity. We look forward to seeing you later this week!Reducing SQL Connection Latency for Apps Using Azure AAD Authentication
Challenge: connection latency and token overhead Consider a cloud-native application deployed in Azure App Service or Kubernetes (AKS) that needs to query an Azure SQL Database for real-time data. The application uses Azure Active Directory (AAD) for secure authentication, but every time the application establishes a new connection to the database, it requests a new AAD token. In high-traffic environments where thousands of requests are processed per second, this repetitive token issuance introduces latency and performance degradation. This delay becomes particularly problematic for time-sensitive applications where every millisecond counts. Each token request impacts response times and creates unnecessary resource consumption. Solution: token caching and expiration management To mitigate these delays, we can optimize the authentication process by caching the AAD token and reusing it for the duration of its validity (typically 1 hour to 24 hours). Instead of requesting a new token for every database connection, the token is fetched only when the existing one is near expiration. This approach eliminates the repeated authentication overhead and ensures that the application can maintain seamless connectivity to the database without the performance hit of generating a new token for each request. In addition to reducing latency, this approach reduces the number of HTTP calls made to the Azure Active Directory service, resulting in better resource utilization and lower operational costs. Concrete performance gains: optimized SQL client connection As part of the mitigation, we provide a custom code implementation that uses SqlClient, a supported library, to optimize the connection time. The test was conducted with the S0 database, where using a single process and using connection pooling, we opened a connection, executed the SELECT 1, and closed the connection. During the testing phase with a connection pooler script running for 96 hours (without the AAD token cache), the following results were observed: 10 connections took 1 second, representing 0.866% of total connections. 1 connection took 4 seconds, representing 0.0866%. 1.144 connections took less than 1 second, representing 99.05% of total connections. All executions of SELECT 1 were completed in 0 seconds. These results demonstrate how caching AAD tokens and reusing them effectively reduced connection overhead and improved performance. None of the connections exceeded 5 seconds in duration, while with the default behavior, connections were reaching 30 seconds and more, depending on the environment complexity. Step-by-step implementation Here’s a step-by-step guide on how to implement this solution using C# and the Microsoft.Data.SqlClient package to optimize SQL database connections: Obtain and cache a token: Instead of requesting a new AAD token with every connection, we obtain a token once and cache it. This is done by leveraging Azure Managed Identity to authenticate the application, which eliminates the need to repeatedly authenticate with Azure Active Directory for every database connection. In this step, we fetch the token once and store it securely for reuse. Renew the token only when it’s near expiry We will refresh the token only when it is nearing expiration or has already expired. The application checks the token’s expiration time before attempting to use it. If the token is still valid, it continues to be reused. If it's close to expiration, a new token is fetched. Reuse a single token across multiple connections: The cached token can be used for multiple database connections during its lifetime. Rather than requesting a new token for each new connection, the application will use the same token across all connections until the token is about to expire. Code example: optimized SQL connection management Here’s an example of how you can implement token caching in a C# application using Microsoft.Data.SqlClient. using System; using System.Data.SqlClient; using System.Diagnostics; using System.Threading; using Azure.Identity; namespace SqlConnectionOptimization { public class SqlConnectionManager { private string _accessToken; private DateTimeOffset _tokenExpiration; private readonly string _connectionString = "Server=tcp:servername.database.windows.net,1433;Initial Catalog=DBName;..."; private readonly Stopwatch _stopwatch = new Stopwatch(); public SqlConnectionManager() { _accessToken = string.Empty; _tokenExpiration = DateTimeOffset.UtcNow; } public void Run() { while (true) { // Refresh token if necessary if (IsTokenExpired()) { RefreshToken(); } // Establish connection and perform operations using (var connection = CreateConnection()) { LogExecutionTime("Connected"); ExecuteQuery(connection); LogExecutionTime("Query Executed"); } // Simulate some idle time between operations Log("Waiting before next operation..."); Thread.Sleep(1000); } } private bool IsTokenExpired() { return string.IsNullOrEmpty(_accessToken) || DateTimeOffset.UtcNow.AddMinutes(5) >= _tokenExpiration; } private void RefreshToken() { _stopwatch.Start(); try { var result = FetchAccessToken(); _accessToken = result.Token; _tokenExpiration = result.Expiration; LogExecutionTime("Token Refreshed"); Log($"Token expires at: {_tokenExpiration}"); } catch (Exception ex) { Log($"Error fetching token: {ex.Message}"); } } private (string Token, DateTimeOffset Expiration) FetchAccessToken() { var managedIdentityCredential = new ManagedIdentityCredential(); var tokenRequestContext = new Azure.Core.TokenRequestContext(new[] { "https://database.windows.net/" }); var accessToken = managedIdentityCredential.GetTokenAsync(tokenRequestContext).Result; return (accessToken.Token, accessToken.ExpiresOn.UtcDateTime); } private SqlConnection CreateConnection() { var connection = new SqlConnection(_connectionString) { AccessToken = _accessToken }; int retries = 0; while (true) { try { connection.Open(); return connection; } catch (Exception ex) { retries++; if (retries > 5) { Log($"Error connecting after multiple retries: {ex.Message}"); throw; } Log($"Connection attempt failed. Retrying in {retries} seconds..."); Thread.Sleep(retries * 1000); } } } private void ExecuteQuery(SqlConnection connection) { var query = "SELECT 1"; // Simple query, replace with real logic as needed int retries = 0; while (true) { try { using (var command = new SqlCommand(query, connection)) { command.CommandTimeout = 5; // Adjust timeout for more complex queries command.ExecuteScalar(); } return; } catch (Exception ex) { retries++; if (retries > 5) { Log($"Max retries reached for query execution: {ex.Message}"); throw; } Log($"Query execution failed. Retrying in {retries} seconds..."); Thread.Sleep(retries * 1000); } } } private void Log(string message) { Console.WriteLine($"{DateTime.Now:yyyy-MM-dd HH:mm:ss.fff}: {message}"); } private void LogExecutionTime(string action) { _stopwatch.Stop(); var elapsed = _stopwatch.Elapsed; Log($"{action} - Elapsed time: {elapsed:hh\\:mm\\:ss\\.fff}"); _stopwatch.Reset(); } public static void Main(string[] args) { var manager = new SqlConnectionManager(); manager.Run(); } } } Key points in the code Token Expiration Check: The IsTokenExpired() method checks whether the token has expired by comparing it to the current time. We’ve added a 5-minute buffer for token expiration. This can be adjusted based on your needs. Managed Identity Authentication: The application uses Azure Managed Identity to authenticate and fetch the token, ensuring secure and scalable access to Azure SQL Database without requiring client secrets. Retry Logic: In the event of a connection failure or query execution failure, the system retries a set number of times with exponential backoff, making it resilient to transient network or authentication issues. Conclusion By implementing a token caching and expiration management strategy, applications can dramatically improve the performance and scalability of their database interactions, especially in environments with high request volumes. By leveraging Azure Managed Identity for secure, reusable tokens, you can reduce authentication latency and improve the overall efficiency of your SQL database connections. This approach can also be adapted to any service using Azure SQL Database and Azure Active Directory for authentication. Next steps Benchmarking: Test the implementation in your environment to quantify the performance gains. Error Handling: Extend the retry logic and error handling to better handle transient failures, especially in production environments. Resources: Introducing Configurable Retry Logic in Microsoft.Data.SqlClient v3.0.0-Preview1 Configurable retry logic in SqlClient Troubleshoot transient connection errors Scaling: Consider how this strategy can be applied across multiple services in larger architectures. Consider reading and applying managed identity best practices. Resources: Managed identity best practice recommendationsJoin the Fabric Partner Community for this Week's Fabric Engineering Connection calls!
Are you a Microsoft partner that is interested in data and analytics? Be sure to join us for the next Fabric Engineering Connection calls! 🎉 Miguel Llopis and Mark Kromer will be providing a recap of the Data Factory Announcements made during FabCon Europe, followed by Ambika J. presenting on Data Recovery Features in Fabric DW. The Americas & EMEA call will take place Wednesday, October 1, from 8-9 am PT and the APAC call is Thursday, October 2, from 1-2 am UTC/Wednesday, October 1, from 5-6 pm PT. This is your opportunity to learn more, ask questions, and provide feedback. To join the call, you must be a member of the Fabric Partner Community Teams channel. To join, complete the participation form at https://aka.ms/JoinFabricPartnerCommunity. We look forward to seeing you later this week!Partner Know Before You Go to the 2025 European Microsoft Fabric Community Conference!
We can’t wait to see you at FabCon Europe 2025, taking place 15-18 September at the Austria Center Vienna in Vienna, Austria! With more than 130 expert-led sessions over three days, plus workshops 15 September, this is the largest Microsoft tech conference in Europe! Our team has been hard at work planning several partner-exclusive activities throughout the event, to help ensure the best experience possible for you, our valued partners. This Know Before You Go guide will provide all the details on how to participate in: Partner Day Partner Happy Hour 1:1 Meetings Partner Booth at Ask the Experts Partner AMA Partner Photo Scavenger Hunt Cvent Event App If you have any questions, please feel free to reach out to our team at mailto:FabricPartnersTeam@microsoft.com or through the Cvent app. See you very soon! Complete details and entry form are available at https://aka.ms/FabConEuropePartnerPhotoHunt.Join the Fabric Partner Community for an AMA with Arun Ulag!
🚨 Mark your calendars now! 📅 New this year in the Fabric Partner Community, an AMA (Ask Me Anything) call series with members of the Fabric Leadership Team! 🥳 Arun Ulag, CVP of Azure Data, will kick of this new call series Thursday, September 25, from 8-9 am PT. You will not want to miss this opportunity to ask all your questions, including those related to the announcements made at #FabConEurope, provide your feedback, and more! 👏 To join theses calls, you must be a member of the Fabric Partner Community Teams Channel. Not yet part of the Fabric Partner Community? Join now by submitting the form at https://aka.ms/JoinFabricPartnerCommunity.Join the Fabric Partner Community for this Week's Fabric Engineering Connection calls!
Are you a Microsoft partner that is interested in data and analytics? Be sure to join us for this week's Fabric Engineering Connection calls! 🎉 Jia Ma will be providing a deep dive on Real-Time Hub and Jenny Jiang will be sharing the latest monitoring updates for Spark jobs. The Americas & EMEA call will take place Wednesday, September 10, from 8-9 am PT and the APAC call is Thursday, September 11, from 1-2 am UTC/Wednesday, September 10, from 5-6 pm PT. This is your opportunity to learn more, ask questions, and provide feedback. To join the call, you must be a member of the Fabric Partner Community Teams channel. To join, complete the participation form at https://aka.ms/JoinFabricPartnerCommunity. We look forward to seeing you next week!
