How to Query Spark Tables from Serverless SQL Pools in Azure Synapse
Introduction Say goodbye to constantly running Spark clusters! With the shared metadata functionality, you can shut down your Spark pools while still be able to query your Spark external tables using Serverless SQL Pool. In this blog we dive into, how Serverless SQL Pool streamlines your data workflow by automatically synchronizing metadata from your Spark pools. Shared Metadata functionality Azure Synapse Analytics allows the different workspace computational engines to share databases and tables between its Apache Spark pools and serverless SQL pool. When we create tables in Apache Spark Pool, whether managed or external, the Serverless SQL pool automatically synchronizes its metadata. This metadata synchronization automatically creates a corresponding external table in a serverless SQL pool database. Then after a short delay, we can see the table in our Serverless SQL pool. Creating a managed table in Spark and querying from Serverless SQL Pool Now we can shut down our Spark pools and still be able to query Spark external tables from Serverless SQL Pool. NOTE: Azure Synapse currently only shares managed and external Spark tables that store their data in Parquet, DELTA, or CSV format. Tables backed by other formats are not automatically synced. You may be able to sync such tables explicitly yourself as an external table in your own SQL database if the SQL engine supports the table's underlying format. Also, External tables created in Spark are not available in dedicated SQL pool databases. Why we get an error if you use dbo schema in Spark pool or if you don’t use dbo schema in Serverless SQL pool? The dbo schema (short for “database owner”) is the default schema in SQL Server and Azure Synapse SQL pools. Spark pool only supports user-defined schemas. Means, it does not recognize dbo as a valid schema name. While in Serverless SQL Pool, all the tables belong to the dbo schema, regardless of their original schema in Spark pool or other sources.Create fake data for a demo or presentation
In preparation for a demo in his talk, Leon Welicki needed "safe data", meaning data that looks legit but is fake. This post shares how I use Azure SQL Database templates, ChatGTP, and the .NET package Bogus to create data safe to be used, random, but that fits the demo theme and requirement.Game of Learner Clinics for Data 2022
The Game of Learners Clinics for Data is a 5-week skilling initiative, which aims to help you learn and gain technical skills on how to work with data in the cloud. This Initiative will also provide you with the option to get to receive a free voucher to sit for the DP-900 Azure Data Fundamentals Exam. Register at https://aka.ms/dataclinics/registerGetting Started with Databases: SQL and Data Visualizations Basics
The importance of learning SQL cannot be argued against. The requirement for SQL skills on job description expands to more than just traditional developer roles. This follow-along session will help you get started with learning SQL by showing real examples of how to read and write to a SQL database. Data is only important by what we can learn from it so we will also cover how you can take data, explore it with SQLlite in an R notebook and build visualizations that tell a story, with the tidyverse framework.Augmented Reality for children at Great Ormond Street Hospital
First published on MSDN on May 10, 2018 Guest blog by Ayushmaan Seth Microsoft Student Partner and 1st Year Computer Science, University College LondonAbout me I am a 1st year Computer Science Undergraduate student at University College London.
