Error: File 'https://storageaccount.dfs.core.windows.net/parquettest/filename.snappy.parquet' cannot be opened because it does not exist or it is used by another process.
Scenario: Storage is with the firewall set to use Selected Network\Allow Trusted Microsoft services. Be aware If that is the case you will need to create the MSI. In order to create MSI on SQL On Demand follow the script example steps.
The error happens when you try to run the Select on the file that exist inside of the storage account from Synapse Studio using SQL On Demand.
Before I proceed with the step by step. Another link that may help you: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/resources-self-help-sql-on-demand
Here it goes - how to do it:
Fig. 1 Data Lake
Follow the script:
CREATE DATABASE [YourDatabase]
GO
-------Change to YourDatabase
CREATE MASTER KEY ENCRYPTION BY PASSWORD= 'XXXXX!0000';
-- Create a database scoped credential.
CREATE DATABASE SCOPED CREDENTIAL AppCred WITH IDENTITY = 'Managed Identity'
Note my Storage Account is called Criativity, you should use for the MSI .dfs instead of .blob.
CREATE EXTERNAL DATA SOURCE YourDataSourceName_criativity
WITH (
LOCATION = 'https://criativity.dfs.core.windows.net',
cREDENTIAL = [AppCred]
)
SELECT TOP 100 *
FROM OPENROWSET(
BULK 'parquet/*.parquet',
DATA_SOURCE = 'YourDataSourceName_criativity',
FORMAT = 'PARQUET'
) as X
That is it!
Thanks to the collegues Charl Roux and Veljko Vasic. :)
Liliam Leme
UK Engineer
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.