Azure Data Factory supports managed virtual network with the scalable Azure Integration Runtime. Azure Integration Runtime is deployed and managed by Microsoft, eliminating the need to have a self-hosted integration runtime by the customer. Azure Integration Runtime managed virtual network uses private endpoints to securely connect to Snowflake, utilizing the Azure Private Link for Snowflake.
NOTE: Snowflake Business Critical edition or higher is required to setup Azure private link with Snowflake.
Setting up Azure Managed Virtual Network Integration Runtime
The Azure managed virtual network integration runtime is recommended to be deployed in the same region where Snowflake is deployed. For details on how to setup managed virtual network for Data Factory, please refer to this Microsoft guide.
Setting up managed endpoint for Snowflake Private Link
Azure Private Link provides private connectivity to Snowflake by ensuring that access to Snowflake is through a private IP address. The Network traffic flows to the Snowflake Virtual Network using the Microsoft backbone and avoids the public Internet. This significantly simplifies the network configuration by keeping access rules private while providing secure and private communication.
1. Execute SYSTEM$WHITELIST and SYSTEM$WHITELIST_PRIVATELINK to obtain the SNOWFLAKE_DEPLOYMENT, SNOWFLAKE_DEPLOYMENT_REGIONLESS and OCSP_CACHE values for public and whitelist hosts. You can also execute the following query with a warehouse to get it in a readable format:
use role accountadmin;
select t.value:type as type,
t.value:host as host,
pt.value:host as privatelink_host,
t.value:port as port
from table(flatten(input=>parse_json(system$whitelist()))) t,
table(flatten(input=>parse_json(system$whitelist_privatelink()))) pt
where t.value:type = pt.value:type;
|
Property |
Public Host |
Private Link Host |
|
SNOWFLAKE_DEPLOYMENT |
<account_name>.<region_id>.azure.snowflakecomputing.com |
<account_name>.<region_id>.privatelink.snowflakecomputing.com |
|
SNOWFLAKE_DEPLOYMENT_REGIONLESS |
<org_name>-<account_name>.snowflakecomputing.com |
<org_name>-<account_name>.privatelink.snowflakecomputing.com |
|
OCSP_CACHE |
ocsp.snowflakecomputing.com |
ocsp.<account_name>.<region_id>.privatelink.snowflakecomputing.com |
SNOWFLAKE_DEPLOYMENT_REGIONLESS will be available only for Organization accounts. If organization URL is not used to connect, then it could be skipped.
2. Execute SYSTEM$GET_SNOWFLAKE_PLATFORM_INFO() as ACCOUNTADMIN to obtain the snowflake-vnet-subnet-ids values. The Subscription ID for the Private Link of the Snowflake’s Azure tenant is obtained from this. You can also execute the following query with a warehouse to get it in a readable format.
use role accountadmin;
select t.key, v.value
from table(flatten(input=>parse_json(system$get_snowflake_platform_info()))) t, table(flatten(t.value)) v;
|
Property |
Value |
|
snowflake-vnet-subnet-id |
/subscriptions/ae0c1e4e-d49e-4115-b3ba-888d77ea97a3/resourceGroups/azure-prod/providers/Microsoft.Network/virtualNetworks/azure-prod/subnets/xp |
The subscription id is the one which has the resource group azure-prod or if it is not present, then it would be the deployment-infra-rg or something similar. The Resource group and Subscription ID may change in the future. Please contact Snowflake support to get the actual Subscription ID details. In this case, the Subscription ID is ae0c1e4e-d49e-4115-b3ba-888d77ea97a3.
3. Create a private endpoint for private link under the Managed private endpoints section in the Manage menu of the Data Factory Studio. For more information on Data Factory managed private endpoints, please refer to the Microsoft documentation.
4. Select Private link service to configure the managed endpoint for the Snowflake private link.
5. The resource ID for the target Snowflake private link is of the following format:
/subscriptions/<subscription_id>/resourcegroups/az<region>-privatelink/providers/microsoft.network/privatelinkservices/sf-pvlinksvc-az<region>
The Subscription ID is from Step 2 and the region is the Snowflake region obtained from the region_id of Step 1 without any hyphen (-).
For example, if the Snowflake region id is east-us-2, then it would be eastus2. So, the URL would look like:
/subscriptions/ae0c1e4e-d49e-4115-b3ba-888d77ea97a3/resourcegroups/azeastus2-privatelink/providers/microsoft.network/privatelinkservices/sf-pvlinksvc-azeastus2
6. For the fully qualified name, add the SNOWFLAKE_DEPLOYMENT, SNOWFLAKE_DEPLOYMENT_REGIONLESS and OCSP_CACHE of both the public and private link host from Step 1.
7. The Provisioning state should be Succeeded, and the Approval state should be pending. If the Provisioning state has failed, check the values provided and repeat the process with the correct values.
8. Since the Private link is in Snowflake’s Azure Subscription, the support ticket needs to be raised to Snowflake to approve the private endpoint connection. Provide the managed private endpoint resource id and the managed approvals link in the Azure portal.
9. Once the Snowflake support confirmed the connection is approved, the managed endpoint approval state should be approved and ready to be utilized.
10. Execute SYSTEM$GET_PRIVATELINK_AUTHORIZED_ENDPOINTS() to check the connection in Snowflake. You can also execute the following query with a warehouse to get it in a readable format:
use role accountadmin;
select value: endpointId as endpoint
from table(flatten(input => parse_json(system$get_privatelink_authorized_endpoints())));
11. The linked service is configured with the Snowflake connection, and it will make use of the Private endpoint to connect to Snowflake via private link.
Setting up Snowflake Network policies
The Snowflake network policies affect the private link connections even though the communications are through private Azure backbone. If network policies are enabled at account level, then the private CIDR 10.0.0.0/8 of the Azure managed virtual network should be allowed. This is a private IP range and would only allow the applications that is configured via private endpoint to Snowflake private link.
For additional security, network policies can be enforced for the Snowflake user accounts configured in the Data Factory Linked Services to allow the CIDR 10.0.0.0/8.
Hope this guide helps!
Updated Oct 13, 2022
Version 1.0
