Using Azure Synapse Serverless Pools for Data Engineering | Data Exposed: MVP Edition

Published Oct 26 2021 09:00 AM 1,118 Views
Microsoft
Data engineering is one of the most involved, dull, and repetitive tasks in the data industry today. Bringing up the subject is a sure way to instantly clear the room at a party or make your friends groan in despair. The work includes a lot of complicated tools, many, many lines of code, and a not-insignificant amount of gnashing of teeth. But what if I told you there is a way to quickly, easily, and cheaply do both data engineering and data exploration on most data without even having to import it into a database? What if I told you that Azure Synapse has a just about magical tool that can instantly connect to flat files and make them available as nice, well-behaved tables? I'm talking about Azure Synapse serverless pools, a great tool for both initial exploration and surprisingly complex data engineering. I'll walk you through what it is, how to set it up, and give you a few examples of basic data exploration, as well as a tad more complex data engineering. Oh, and did I mention you can actually store the end result in several different formats as well...?

 

Watch on Data Exposed


Resources:
%3CLINGO-SUB%20id%3D%22lingo-sub-2883504%22%20slang%3D%22en-US%22%3EUsing%20Azure%20Synapse%20Serverless%20Pools%20for%20Data%20Engineering%20%7C%20Data%20Exposed%3A%20MVP%20Edition%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2883504%22%20slang%3D%22en-US%22%3E%3CDIV%3E%3CSPAN%3EData%20engineering%20is%20one%20of%20the%20most%20involved%2C%20dull%2C%20and%20repetitive%20tasks%20in%20the%20data%20industry%20today.%20Bringing%20up%20the%20subject%20is%20a%20sure%20way%20to%20instantly%20clear%20the%20room%20at%20a%20party%20or%20make%20your%20friends%20groan%20in%20despair.%20The%20work%20includes%20a%20lot%20of%20complicated%20tools%2C%20many%2C%20many%20lines%20of%20code%2C%20and%20a%20not-insignificant%20amount%20of%20gnashing%20of%20teeth.%20But%20what%20if%20I%20told%20you%20there%20is%20a%20way%20to%20quickly%2C%20easily%2C%20and%20cheaply%20do%20both%20data%20engineering%20and%20data%20exploration%20on%20most%20data%20without%20even%20having%20to%20import%20it%20into%20a%20database%3F%20What%20if%20I%20told%20you%20that%20Azure%20Synapse%20has%20a%20just%20about%20magical%20tool%20that%20can%20instantly%20connect%20to%20flat%20files%20and%20make%20them%20available%20as%20nice%2C%20well-behaved%20tables%3F%20I'm%20talking%20about%20Azure%20Synapse%20serverless%20pools%2C%20a%20great%20tool%20for%20both%20initial%20exploration%20and%20surprisingly%20complex%20data%20engineering.%20I'll%20walk%20you%20through%20what%20it%20is%2C%20how%20to%20set%20it%20up%2C%20and%20give%20you%20a%20few%20examples%20of%20basic%20data%20exploration%2C%20as%20well%20as%20a%20tad%20more%20complex%20data%20engineering.%20Oh%2C%20and%20did%20I%20mention%20you%20can%20actually%20store%20the%20end%20result%20in%20several%20different%20formats%20as%20well...%3F%3C%2FSPAN%3E%3C%2FDIV%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%3E%3CFONT%20size%3D%225%22%3E%3CA%20href%3D%22https%3A%2F%2Fyoutu.be%2FymV8PNbwYHU%3FWT.mc_id%3Ddataexposed-c9-niner%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3EWatch%20on%20Data%20Exposed%3C%2FA%3E%3C%2FFONT%3E%3CBR%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CDIV%3E%3CEM%3E%3CBR%20%2F%3EResources%3A%3C%2FEM%3E%3C%2FDIV%3E%0A%3CDIV%3E%0A%3CDIV%3E%3CA%20href%3D%22https%3A%2F%2Fwww.datahai.co.uk%2Ftag%2Fsql-serverless%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EAndy%20Cutler's%20blog%20on%20serverless%20pools%3C%2FA%3E%3CBR%20%2F%3E%3CA%20href%3D%22https%3A%2F%2Fam2.co%2F2019%2F10%2Fhow-to-create-date-buckets-in-t-sql%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EAndy%20Mallon's%20blog%20on%20bucketing%20with%20T-SQL%3C%2FA%3E%3C%2FDIV%3E%0A%3CDIV%3E%0A%3CDIV%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%3E%3CSPAN%3EView%2Fshare%20our%20latest%26nbsp%3Bepisodes%20on%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fchannel9.msdn.com%2FShows%2FData-Exposed%3FWT.mc_id%3Ddataexposed-c9-niner%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EChannel%209%3C%2FA%3E%3CSPAN%3E%26nbsp%3Band%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fwww.youtube.com%2Fplaylist%3Flist%3DPLlrxD0HtieHieV7Jls72yFPSKyGqycbZR%26amp%3BWT.mc_id%3Ddataexposed-c9-niner%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EYouTube%3C%2FA%3E%3CSPAN%3E!%3C%2FSPAN%3E%3C%2FDIV%3E%0A%3C%2FDIV%3E%0A%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2883504%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22MSDataExposed.jpg%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F201130iE9AAF33173512AF8%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22MSDataExposed.jpg%22%20alt%3D%22MSDataExposed.jpg%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3ELearn%20how%20to%20do%20data%20engineering%20using%20Azure%20Synapse%20Serverless%20Pools.%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2883504%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20SQL%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Co-Authors
Version history
Last update:
‎Oct 25 2021 01:32 PM
Updated by: