There are various scenarios wherein you would need to access data on Azure Storage or secrets from Azure Key Vault from a Data Factory pipeline or your applications. Often there is a security requirement to prevent any unknown sources from accessing the Storage account or the Azure Key Vault service. In such circumstances, you can use the ‘Allow trusted Microsoft services...’ setting in the firewall to enable access to your data from 'Trusted Services' without requiring you to allow connections from all network. For more details on 'Trusted Services', please refer azure storage and azure key vault documentation.
Data Factory is now part of ‘Trusted Services’ in Azure Key Vault and Azure Storage. Integration runtime (Azure, Self-hosted, and SSIS) can now connect to Storage/ Key Vault without having to be inside the same virtual network or requiring you to allow all inbound connections to the service.
Note: Both Data Movement and Mapping Data flows are also supported as ‘Trusted Services’.
Common data integration security requirements
- Use the Internet to connect to data stores/ secrets store over TLS
- Security – secure data using all supported Auth mechanism
- Recommendation – Use Azure IR/ SSIS IR
- Use the Internet to connect to data stores/ secrets store over TLS only from known sources using ‘Trusted Services’ firewall exception
- Security – secure data using MSI Auth + Service Firewall
- Recommendation – Use ‘Allow Trusted Services…’ in Storage/ Key Vault firewall + Azure IR/ Self-hosted IR/ SSIS IR
- Use a private network/ virtual network to connect to data stores over TLS
- Security – secure data using Auth + compute injection/ peering with the private network
- Recommendation – Use Self-hosted IR/ SSIS IR within your Virtual Network/ Private network.
Note: We are actively working on adding the capability to add/ peer an Azure IR inside VNET.
Steps to connect as ‘Trusted Service’
-
Connecting to Azure Storage (using Azure blob or Azure Data lake Gen2 linked service)
- Grant Data Factory’s Managed identity access to read data in storage’s access control. For more detailed instructions, please refer this.
- Create the linked service using Managed identities for Azure resources authentication
- Modify the firewall settings in Azure Storage account to select ‘Allow trusted Microsoft Services…’.
Note: Only Managed Identity authentication is supported when using ‘Trusted Service’ functionality in storage to allow Azure Data Factory to access its data.
-
Connecting to Azure Key Vault (using Azure Key Vault linked service)
- Create linked service with managed identity authentication and grant appropriate permissions in Azure Key Vault Access Policies as mentioned in the article.
- Modify the firewall settings in the Azure Key Vault to select ‘Allow Trusted Microsoft Services…’
Next Steps
See the following related articles for more details:
- Azure Key Vault ‘Trusted Services’
- Azure Storage ‘Trusted Microsoft Services’
- Managed identity for Data Factory