How to Query Spark Tables from Serverless SQL Pools in Azure Synapse
Introduction Say goodbye to constantly running Spark clusters! With the shared metadata functionality, you can shut down your Spark pools while still be able to query your Spark external tables using Serverless SQL Pool. In this blog we dive into, how Serverless SQL Pool streamlines your data workflow by automatically synchronizing metadata from your Spark pools. Shared Metadata functionality Azure Synapse Analytics allows the different workspace computational engines to share databases and tables between its Apache Spark pools and serverless SQL pool. When we create tables in Apache Spark Pool, whether managed or external, the Serverless SQL pool automatically synchronizes its metadata. This metadata synchronization automatically creates a corresponding external table in a serverless SQL pool database. Then after a short delay, we can see the table in our Serverless SQL pool. Creating a managed table in Spark and querying from Serverless SQL Pool Now we can shut down our Spark pools and still be able to query Spark external tables from Serverless SQL Pool. NOTE: Azure Synapse currently only shares managed and external Spark tables that store their data in Parquet, DELTA, or CSV format. Tables backed by other formats are not automatically synced. You may be able to sync such tables explicitly yourself as an external table in your own SQL database if the SQL engine supports the table's underlying format. Also, External tables created in Spark are not available in dedicated SQL pool databases. Why we get an error if you use dbo schema in Spark pool or if you don’t use dbo schema in Serverless SQL pool? The dbo schema (short for “database owner”) is the default schema in SQL Server and Azure Synapse SQL pools. Spark pool only supports user-defined schemas. Means, it does not recognize dbo as a valid schema name. While in Serverless SQL Pool, all the tables belong to the dbo schema, regardless of their original schema in Spark pool or other sources.Enhance Cost Optimization in Azure Cosmos DB Without Compromising Service
Optimizing Azure cosmos DB involves using strategies and best practices to reduce the overall spending on the service while maintaining and improving performance and availability. In this blog, I’ll explore actionable strategies and tools for reducing your Azure Cosmos DB costs without sacrificing the speed, scalability, or reliability of your database. Whether you're managing large-scale applications or developing on a budget, these insights will help you make the most out of Azure Cosmos DB. The main sub-topics we shall dive into includes: Pricing model Free development Plan for OptimizationBuild the "Take a break reminder" app with Microsoft Graph - November 14th
Join us November 14th! This is the third week of the 30 Days of Microsoft Graph Challenge! It's not too late, join the challenge: https://aka.ms/30DaysOfMSGraph/Challenge. 30 Days of Microsoft Graph is a gamified learning experience for beginners to get started building apps with Microsoft Graph and JavaScript.
Create Your Azure Infrastructure with Cloud Development Kit for Terraform - LIVE SESSION 15th Sept
This series will teach you how to use Cloud Development Kit for Terraform (CDK-TF) and Microsoft Azure serverless services. First, you will learn how to build everything as code, and we will teach you how to solve several real-world problems with just Azure Functions and short-lived Azure container instance. All projects covered in the series are opensource.Seamless assessment management with Azure LTI Assessment App
The Azure LTI Assessment App aims to reduce time spent by educators on assessment management. It works independently from any LMS, provides a unified user interface, and eliminates the need to transfer the questions from one format to another when switching between different LMS.
