Using the workspace MSI to authenticate a Synapse notebook when accessing an Azure Storage account

Published 05-06-2021 11:04 AM 1,271 Views
Microsoft

Background

 

When a Synapse notebook accesses Azure storage account it uses an AAD identity for authentication.

 

How the notebook is run controls with AAD identity is used:

  • If a user is interactively running the notebook, then the user's AAD identity is used. We often call this "AAD passthrough" because it "passes the user's AAD identity through to Azure Storage" 
  • If the notebook is run through the pipeline, the workspace MSI is used. 

This blog will show you how force the notebook to always use the workspace MSI.

 

Audience

This is for beginners with some knowledge of the workspace configuration using linked services.

 

STEP 1: Ensure the workspace MSI must have the permissions to access the data in the storage account.

 

The easiest way of doing this is to assign the workspace to the Storage Blob Data Contributor role on the storage account.

 

STEP 2: Configuring the storage account firewall (if needed)

 

If you have enabled the firewall on the storage account, you need to follow these instructions:  Configure Azure Storage firewalls and virtual networks | Microsoft Docs

 

Here is an example with firewall enabled on the storage account:

post.png

When you grant access to trusted Azure services inside of the storage networking, you will grant the following types of access:

 

  • Trusted access for select operations to resources that are registered in your subscription.
  • Trusted access to resources based on system-assigned managed identity.

Additional information on this topic can be found in this document: Connect to a secure storage account from your Azure Synapse workspace – Azure Synapse Analytics | Mi...

 

 

Liliam_Leme_0-1620285112785.png

 

 

Step 3: Configuring the Linked Service

Open Synapse Studio and configure the Linked Service to use the workspace MSI:

 

Liliam_Leme_2-1620285112806.png

 

STEP 4: Test the configuration and see if it is successful

 

Click Test connection to verify that you have configured everything correctly.

 

STEP 5: Update the notebook code to use the Linked Service configuration 

 

val linked_service_name = “LinkedServerName” 
// replace with your linked service name
%%spark
// Allow SPARK to access from Blob remotely
val sc = spark.sparkContext
spark.conf.set(“spark.storage.synapse.linkedServiceName”, linked_service_name)
spark.conf.set(“fs.azure.account.oauth.provider.type”, “com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider”) 
//replace the container and storage account names
val df = “abfss://Container@StorageAccount.dfs.core.windows.net/”

print(“Remote blob path: ” + df)

mssparkutils.fs.ls(df)

 


Additional Resources

Learn more about how the Synapse workspaces performs authentication and uses managed identities by reading these documents:

 

 

That is it!

Liliam UK Engineer

%3CLINGO-SUB%20id%3D%22lingo-sub-2330029%22%20slang%3D%22en-US%22%3EUsing%20the%20workspace%20MSI%20to%20authenticate%20a%20Synapse%20notebook%20when%20accessing%20an%20Azure%20Storage%20account%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2330029%22%20slang%3D%22en-US%22%3E%3CP%20style%3D%22font-weight%3A%20inherit%3B%20font-style%3A%20inherit%3B%22%3E%3CSTRONG%3EBackground%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%20style%3D%22font-weight%3A%20inherit%3B%20font-style%3A%20inherit%3B%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22font-weight%3A%20inherit%3B%20font-style%3A%20inherit%3B%22%3EWhen%20a%20Synapse%20notebook%20accesses%20Azure%20storage%20account%20it%20uses%20an%20AAD%20identity%20for%20authentication.%3C%2FP%3E%0A%3CP%20style%3D%22font-weight%3A%20inherit%3B%20font-style%3A%20inherit%3B%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22font-weight%3A%20inherit%3B%20font-style%3A%20inherit%3B%22%3EHow%20the%20notebook%20is%20run%20controls%20with%20AAD%20identity%20is%20used%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%20style%3D%22font-weight%3A%20inherit%3B%20font-style%3A%20inherit%3B%22%3EIf%20a%20user%20is%20interactively%20running%20the%20notebook%2C%20then%20the%20user's%20AAD%20identity%20is%20used.%20We%20often%20call%20this%20%22AAD%20passthrough%22%20because%20it%20%22%3CSTRONG%3Epasses%3C%2FSTRONG%3E%20the%20user's%20AAD%20identity%20%3CSTRONG%3Ethrough%3C%2FSTRONG%3E%20to%20Azure%20Storage%22%26nbsp%3B%3C%2FLI%3E%0A%3CLI%20style%3D%22font-weight%3A%20inherit%3B%20font-style%3A%20inherit%3B%22%3EIf%20the%20notebook%20is%20run%20through%20the%20pipeline%2C%20the%20workspace%20MSI%20is%20used.%26nbsp%3B%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EThis%20blog%20will%20show%20you%20how%26nbsp%3B%3CSPAN%20style%3D%22font-family%3A%20inherit%3B%22%3Eforce%20the%20notebook%20to%20%3C%2FSPAN%3E%3CSTRONG%20style%3D%22font-family%3A%20inherit%3B%22%3Ealways%3C%2FSTRONG%3E%3CSPAN%20style%3D%22font-family%3A%20inherit%3B%22%3E%20use%20the%20workspace%20MSI.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20style%3D%22font-weight%3A%20inherit%3B%20font-style%3A%20inherit%3B%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22font-weight%3A%20inherit%3B%20font-style%3A%20inherit%3B%22%3E%3CSTRONG%3EAudience%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%20style%3D%22font-weight%3A%20inherit%3B%20font-style%3A%20inherit%3B%22%3EThis%20is%20for%20beginners%20with%20some%20knowledge%20of%20the%20workspace%20configuration%20using%20linked%20services.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ESTEP%201%3A%20Ensure%20%3C%2FSTRONG%3E%3CSTRONG%3Ethe%20workspace%20MSI%20must%20have%20the%20permissions%20to%20access%20the%20data%20in%20the%20storage%20account%3C%2FSTRONG%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20easiest%20way%20of%20doing%20this%20is%20to%20assign%20the%20workspace%20to%20the%26nbsp%3B%3CSTRONG%3EStorage%20Blob%20Data%20Contributor%3C%2FSTRONG%3E%20role%20on%20the%20storage%20account.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ESTEP%202%3A%20Configuring%20the%20storage%20account%20firewall%20(if%20needed)%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EIf%20you%20have%20enabled%20the%20firewall%20on%20the%20storage%20account%2C%20you%20need%20to%20follow%20these%20instructions%3C%2FSTRONG%3E%3A%20%26nbsp%3B%3CA%20style%3D%22background-color%3A%20%23ffffff%3B%22%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fdocs.microsoft.com%252Fen-us%252Fazure%252Fstorage%252Fcommon%252Fstorage-network-security%253Ftabs%253Dazure-portal%2523exceptions%26amp%3Bdata%3D04%257C01%257Clilem%2540microsoft.com%257C5a8c5a3914b84759674108d90c1acbcc%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637554131494043972%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C1000%26amp%3Bsdata%3DpjP5r0u16ZCuLCW18SUHqNNBje0PxnY8WdXp4ydWNJU%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EConfigure%20Azure%20Storage%20firewalls%20and%20virtual%20networks%20%7C%20Microsoft%20Docs%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EHere%20is%20an%20example%20with%20firewall%20enabled%20on%20the%20storage%20account%3A%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22post.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F278674i1751425B2000950C%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22post.png%22%20alt%3D%22post.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EWhen%20you%20grant%20access%20to%20trusted%20Azure%20services%20inside%20of%20the%20storage%20networking%2C%20you%20will%20grant%20the%20following%20types%20of%20access%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3ETrusted%20access%20for%20select%20operations%20to%20resources%20that%20are%20registered%20in%20your%20subscription.%3C%2FLI%3E%0A%3CLI%3ETrusted%20access%20to%20resources%20based%20on%20system-assigned%20managed%20identity.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EAdditional%20information%20on%20this%20topic%20can%20be%20found%20in%20this%20document%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fdocs.microsoft.com%252Fen-us%252Fazure%252Fsynapse-analytics%252Fsecurity%252Fconnect-to-a-secure-storage-account%2523access-a-secured-storage-account%26amp%3Bdata%3D04%257C01%257Clilem%2540microsoft.com%257C5a8c5a3914b84759674108d90c1acbcc%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637554131494053967%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C1000%26amp%3Bsdata%3D%252FrARceiT21I7oKTmEch4ds1CL9zo5nHO2OkK15R%252BpTE%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EConnect%20to%20a%20secure%20storage%20account%20from%20your%20Azure%20Synapse%20workspace%20%E2%80%93%20Azure%20Synapse%20Analytics%20%7C%20Microsoft%20Docs%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Liliam_Leme_0-1620285112785.png%22%20style%3D%22width%3A%20596px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F278671iAC5CE65ED02BEB14%2Fimage-dimensions%2F596x79%3Fv%3Dv2%22%20width%3D%22596%22%20height%3D%2279%22%20role%3D%22button%22%20title%3D%22Liliam_Leme_0-1620285112785.png%22%20alt%3D%22Liliam_Leme_0-1620285112785.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EStep%203%3A%20Configuring%20the%20Linked%20Service%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EOpen%20Synapse%20Studio%20and%20configure%20the%20Linked%20Service%20to%20use%20the%20workspace%20MSI%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Liliam_Leme_2-1620285112806.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F278673i9C4DE669ABF944DF%2Fimage-size%2Fmedium%3Fv%3Dv2%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Liliam_Leme_2-1620285112806.png%22%20alt%3D%22Liliam_Leme_2-1620285112806.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ESTEP%204%3A%20Test%20the%20configuration%20and%20see%20if%20it%20is%20successful%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EClick%20%3CSTRONG%3ETest%20connection%3C%2FSTRONG%3E%20to%20verify%20that%20you%20have%20configured%20everything%20correctly.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ESTEP%205%3A%20Update%20the%20notebook%20code%20to%20use%20the%20Linked%20Service%20configuration%26nbsp%3B%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-java%22%3E%3CCODE%3Eval%20linked_service_name%20%3D%20%E2%80%9CLinkedServerName%E2%80%9D%20%0A%2F%2F%20replace%20with%20your%20linked%20service%20name%0A%25%25spark%0A%2F%2F%20Allow%20SPARK%20to%20access%20from%20Blob%20remotely%0Aval%20sc%20%3D%20spark.sparkContext%0Aspark.conf.set(%E2%80%9Cspark.storage.synapse.linkedServiceName%E2%80%9D%2C%20linked_service_name)%0Aspark.conf.set(%E2%80%9Cfs.azure.account.oauth.provider.type%E2%80%9D%2C%20%E2%80%9Ccom.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider%E2%80%9D)%20%0A%2F%2Freplace%20the%20container%20and%20storage%20account%20names%0Aval%20df%20%3D%20%E2%80%9Cabfss%3A%2F%2FContainer%40StorageAccount.dfs.core.windows.net%2F%E2%80%9D%0A%0Aprint(%E2%80%9CRemote%20blob%20path%3A%20%E2%80%9D%20%2B%20df)%0A%0Amssparkutils.fs.ls(df)%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CBR%20%2F%3E%3CSTRONG%3EAdditional%20Resources%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3ELearn%20more%20about%20how%20the%20Synapse%20workspaces%20performs%20authentication%20and%20uses%20managed%20identities%20by%20reading%20these%20documents%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fdocs.microsoft.com%252Fen-us%252Fazure%252Fsynapse-analytics%252Fspark%252Fapache-spark-secure-credentials-with-tokenlibrary%253Fpivots%253Dprogramming-language-scala%26amp%3Bdata%3D04%257C01%257Clilem%2540microsoft.com%257C5a8c5a3914b84759674108d90c1acbcc%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637554131494043972%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C1000%26amp%3Bsdata%3DUHDGJqIGHIcf%252FDSiGHTXMBolVOUC9jm0adh4mMIUDAg%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3ESecure%20credentials%20with%20linked%20services%20using%20the%20TokenLibrary%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fdocs.microsoft.com%252Fazure%252Fdata-factory%252Fconnector-azure-blob-storage%253Ftoc%253D%252Fazure%252Fsynapse-analytics%252Ftoc.json%2526bc%253D%252Fazure%252Fsynapse-analytics%252Fbreadcrumb%252Ftoc.json%2523managed-identity%26amp%3Bdata%3D04%257C01%257Clilem%2540microsoft.com%257C02a98953d5f74072650c08d9109ef2f0%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637559097155820323%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C1000%26amp%3Bsdata%3DSgaSsejHmio4Yf22IVeNjMHWXYKvkVDW4HTmxtno8Hk%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EManaged%20identities%20for%20Azure%20resource%20authentication%3C%2FA%3E%26nbsp%3B%26nbsp%3B%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fdocs.microsoft.com%252Fazure%252Fsynapse-analytics%252Fspark%252Fapache-spark-secure-credentials-with-tokenlibrary%253Fpivots%253Dprogramming-language-scala%2523tokenlibrary-for-other-linked-services%26amp%3Bdata%3D04%257C01%257Clilem%2540microsoft.com%257C02a98953d5f74072650c08d9109ef2f0%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637559097155830310%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C1000%26amp%3Bsdata%3D5T5cYLrQqMKNPm8v3ShsxDgKNZjOc5BGDtEZ4MnhNFQ%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EADLS%20Gen2%20storage%20with%20linked%20services%26nbsp%3B%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fdocs.microsoft.com%252Fen-us%252Fazure%252Fsynapse-analytics%252Fspark%252Fmicrosoft-spark-utilities%26amp%3Bdata%3D04%257C01%257Clilem%2540microsoft.com%257C5a8c5a3914b84759674108d90c1acbcc%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637554131494063965%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C1000%26amp%3Bsdata%3DAp7sfJ40CwaJjqtNvFpkp0t%252FbzLsApDFjTmG%252FL%252FPGj0%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EIntroduction%20to%20Microsoft%20Spark%20utilities%20%E2%80%93%20Azure%20Synapse%20Analytics%20%7C%20Microsoft%20Docs%3C%2FA%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThat%20is%20it!%3C%2FP%3E%0A%3CP%3ELiliam%20UK%20Engineer%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2330029%22%20slang%3D%22en-US%22%3E%3CP%3EConfigure%20notebooks%20to%20always%20use%20the%20workspace%20MSI%20for%20authentication.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2330029%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ESynapse%20Spark%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Co-Authors
Version history
Last update:
‎May 07 2021 08:00 PM
Updated by: