Notebook - This request is not authorized to perform this operation. , 403

[Liliam_C_Leme]

This a quick post about this failure and how to fix: Error: org.apache.spark.sql.AnalysisException: java.lang.RuntimeException:
The operation failed: 'This request is not authorized to perform this operation.', 403

First, let's just add some context : 

When you are working on synapse workspace with the managed identity you would need to give Storage Blob Data contributor permission to the workspace that represents the managed identity permission:

https://docs.microsoft.com/en-us/azure/synapse-analytics/security/how-to-grant-workspace-managed-ide...

More information here: https://docs.microsoft.com/en-us/azure/synapse-analytics/security/synapse-workspace-managed-identity

Speaking of managed identities -quick review on it: "A common challenge when building cloud applications is how to manage the credentials in your code for authenticating to cloud services. You can use the identity to authenticate to any service that supports Azure AD authentication, including Key Vault, without any credentials in your code."

More here: https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview?...

So there you are with your workspace with the managed identity permissions granted running a notebook to create a database on Spark.

Note you are the one running the notebook on Synapse Studio and you are not using any credentials as you are under Synapse workspace:

%%spark
spark.sql("CREATE DATABASE IF NOT EXISTS nyctaxi")

It fails:

Error : org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: Operation failed: "This request is not authorized to perform this operation using this permission.", 403, HEAD, https://StorageAccountName.dfs.core.windows.net/ContainerName/tmp/hive?upn=false&action=getStatus&ti...;

Solution:

Add the RBAC Storage Blob Data Contributor to the user that is running the notebook, or your user.

Steps here: https://docs.microsoft.com/en-us/azure/synapse-analytics/security/how-to-grant-workspace-managed-ide... (Grant permissions to managed identity after workspace creation)

In this case, the script is running under the user that is executing the notebook. So this user needs permission as well.

That is it!

Liliam Uk Engineer.

[GauravKhattar]

Hi @Liliam_C_Leme 

I have assigned following permissions:

1. Myself and Synapse identity as Storage Blob Contributor
2. Myself as Synapse Admin
3. Added Myself and Synapse identity to ACL with Read, Write and Execute permissions on container

Still I'm getting this error

[Liliam_C_Leme]

Hi,@GauravKhattar  do you have Vnet enable? is the error the same with or without the firewall?  Is this storage account on the same subscription? Can you connect on it with the Synapse linked server? is Synapse configured to pass through the firewall ( if it is enabled)? Connect to a secure storage account from your Azure Synapse workspace - Azure Synapse Analytics | Mi...

Did you create this workspace on top of SQLDW ( former). If that is the case, can you try to add a private endpoint?

[Nathan_Zimmerman]

I get the same message running azcopy at the command line. I have authorized myself as owner of the container, storage account and the resource group and have logged into azcopy using the AD tenant, per instructions. It works fine when I use SAS but not with Active Directory authentication. Frankly, I don't see how it can work because the AD tenant it not tied to the container but those are the instructions.

[GauravKhattar]

@Liliam_C_Leme 

1. I don't have VNet
2. There are no firewall restrictions
3. No Synapse and storage are not in same subscription
4. Yes I can browse folder through Integration dataset created by Linked service
5. Error is intermittent not consistent
   1. In most runs I get error and in some runs it succeeds
6. I am connecting using SPN through spark conf:
   1. spark.conf.set("spark.storage.synapse.linkedServiceName", linkedServiceName)
      spark.conf.set("fs.azure.account.oauth.provider.type", "com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider")

[pranp]

I am currently having the same problem. The Storage Account that it is trying to access was created together with Azure Synapse Analytics. I have the contributor permission for the Storage Account as well. The problem goes away when I change the firewall setting to 'All Network'. I have a private endpoint from the Storage Account set in the Managed Vnet too.

[GSunita]

Hi  @Liliam_Leme 

I am trying to read a file in storage account from by Spark notebook.

I am able to create Linked service to the storage , read and write files using copy data , but not able to read it from spark notebook.

I gave  Synapse MSI and user group "Storage Blob Data Contributor access" on the storage

Added Synapse workspace in the firewall rule. 

Still getting the same error ""This request is not authorized to perform this operation.", 403"

[GSunita]

Hi ,

Able to solve the issue of reading the data from ADLS gen2 by using the native capability of synapse to create a linked server connection to the file.

This works same as the linked service from synapse pipeline. So we reuse those linked service to connection for reading the files.

When the type of LS s managed identity , we can use the below code to read the file (this can be extended for json , parquet and other formats)

%%pyspark

# Python code

spark.conf.set("spark.storage.synapse.linkedServiceName", "<lINKED SERVICE NAME>") spark.conf.set("fs.azure.account.oauth.provider.type", "com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider") df = spark.read.csv('abfss://<CONTAINER>@<ACCOUNT>.dfs.core.windows.net/<DIRECTORY PATH>') df.show()

Hope this helps.

[Liliam_C_Leme]

thanks @GSunita 

[Liliam_C_Leme]

@GauravKhattar  "

1. Error is intermittent not consistent
   1. In most runs I get error and in some runs it succeeds" -> you mentioned this and considering that fact. Please open a Case to microsoft support, maybe there is  more than just adjust the configuration in your scenario. ( sorry the delay in answer)

[Liliam_C_Leme]

@pranp  did you add Synapse workspace on the storage network to whitelist when you enable the firewall?

( Storage Account -> network ->Resource instances -> Microsoft. synapse/workspace -> Name of your workspace -> save)

[GSunita]

Hi ,

Please refer to the link below for further details on using the Linked services for reading external ADLS gen2 storage from Synapse notebook.

Secure access credentials with Linked Services in Apache Spark for Azure Synapse Analytics - Azure S...

[joshuadmatthews]

When I had this same 403 issue it was because I hadn't approved a private endpoint connection request from synapse on my storage account. The request to DFS storage from the spark notebook / SQL pool was routed to the public storage endpoint and blocked by the firewall due to IP restrictions. Once I approved the private endpoint connection from synapse it creates a private endpoint within the azure managed network that the SQL pools run inside to route directly to storage account without going through its public endpoint firewall.

[Liliam_C_Leme]

@GauravKhattar if the notebook is not using MSI, it is running under the AAD passthrough. so the user that is executing the notebook is the one that gets the permissions validated against the container. This user needs to have the RBAC Blob data contributos on the storage account( please try this also and if it works check later ACL granularity permission) and the other point is the network. Whether it is using the public internet or not. Please check if there is any difference in the notebook execution using public or not public internet configuration on the storage ( if it is possible to check that). 

If this does not help to address where is blocking. Open a Microsoft Support ticket, then the engineer can review it with you in more detail. thanks for your comment

[Liliam_C_Leme]

thanks @joshuadmatthews . Good point

[luluskyw971]

Hi @Liliam_C_Leme  did you get in touch with Microsoft for solving this issue ? I am facing the same error and I check everything from MSI / storage blob data contributor. However it doesn't work still :(

[Liliam_C_Leme]

@luluskyw971  question.. Do you have VNET enabled or not? If VNET is not enabled it will not work anyway, unless you use the public internet on the storage account. 

Connect to a secure storage account from your Azure Synapse workspace - Azure Synapse Analytics | Mi...

"Access a secured storage account Synapse operates from networks that cannot be included in your network rules. The following needs to be done to enable access from your workspace to your secure storage account.

Create an Azure Synapse workspace with a managed virtual network associated with it and create managed private endpoints from it to the secure storage account"

P.S. Feel free to open a case to Microsoft, if you want to double confirm. 

[ajay041994]

Hi @Liliam_C_Leme ,
I have given the 'Storage Blob Data Contributor' right to the Synapse Workspace on the Storage Account. I have mentioned 'Microsoft.Synapse/workspaces' in the resource type and the workspace's name in the resource instance (Storage Account -> network ->Resource instances -> Microsoft. synapse/workspace -> Name of your workspace -> save). I have allowed 'Azure Services' to connect to the Storage Account. I have contributor access to the storage account. My Linked Service can connect to the Storage Account and I can see the files in the 'Linked' service of Data in Synapse Studio. Also, the storage account is the 'primary' storage account for Synapse. I have created the mount point using LinkedService, and Account Key, but each time I get the error:  "This request is not authorized to perform this operation.", 403, HEAD,. Could you please advise here.