Run your Azure Stream Analytics job inside your Azure Virtual Network (Public Preview)
Published May 23 2023 08:00 AM 3,177 Views

Overview

Virtual network support enables you to lock down access to Azure Stream Analytics to your virtual network infrastructure.  This capability provides you with the benefits of network isolation and can be accomplished by deploying a containerized instance of your Azure Stream Analytics job inside your Virtual Network.  Your VNET injected ASA job can then privately access your resources within the virtual network via:

 

 

Availability

Currently, this capability is only available in select Azure regions.  See here for an updated list of VNET enabled regions. 

If you are interested in enabling VNET integration in your region, please let us know by completing this form.  

 

ASA Requirements for VNET Integration Support

 

1. A General Purpose V2 Storage account is required for VNET injected ASA jobs.  
  • VNET injected ASA jobs require access to metadata such as checkpoints to be stored in Azure tables for operational purposes. 

  • If you already have a GPV2 account provisioned with your ASA job, no additional steps are required. 

  • Premium storage accounts are not supported for this requirement, users will Premium storage are still required to provide a GPV2 storage account. 

  • If you wish to protect your storage accounts from public IP based access, consider configuring it using Managed Identity and Trusted Services.
  • For more information on storage accounts, see Storage account overview and Create a storage account.

 

2. Please provide or create an Azure Virtual Network for ASA to use.

 

IMPORTANT

ASA VNET injected jobs use an internal container injection technology provided by Azure networking.  At this time, Azure Networking recommends that all customers set up Azure NAT Gateway for security and reliability. 

Azure NAT Gateway is a fully managed and highly resilient Network Address Translation (NAT) service.  Azure NAT Gateway simplifies outbound Internet connectivity for virtual networks.  When configured on a subnet, all outbound connectivity uses the NAT gateway's static public IP addresses.

Anasheh_Boisvert_9-1684817726321.png

 

To learn about setup and pricing, please visit Azure NAT Gateway.

 

Subnet Requirements

Virtual network integration depends on a dedicated subnet.  When you create a subnet, the Azure subnet consumes five IPs from the start.

You must take into consideration the IP range associated with your delegated subnet as you think about future needs required to support your ASA workload.  Because subnet size can't be changed after assignment, use a subnet that's large enough to accommodate whatever scale your job(s) might reach.

 

Take the following into consideration as you estimate your IP range:

  • Make sure the subnet range does not collide with ASA’s subnet range.  Avoid IP range 10.0.0.0 to 10.0.255.255 as it is used by ASA.
  • Reserve:
    • 5 IP addresses for Azure Networking
    • 1 IP address to facilitate features such as sample data, test connection and metadata discovery for jobs associated with specified subnet.
    • 2 IP addresses for every 6 SU or 1 SU V2 (a single streaming node).  For details on ASA's new pricing structure (V2) launchin on July 1st 2023, please see this blog.

 

When you indicate VNET integration with your Azure Stream Analytics job, Azure Portal will automatically delegate the subnet to the ASA service.  Azure Portal will undelegate the subnet in the following scenarios:

  • You inform us that VNET integration is no longer needed for the last job associated with specified subnet via the ASA portal (see ‘how to’ section).
  • You delete the last job associated with the specified subnet.

 

Last job

Several ASA jobs may utilize the same subnet.  The last job here refers to no other jobs utilizing the specified subnet.  When the last job has been deleted or removed by associated, Azure Stream Analytics will release the subnet as a resource which was delegated to ASA as a service.  Please allow several minutes for this action to be completed.

 

How to Setup VNET integration

From the Azure portal, navigate to Networking from menu bar and select ‘Run this job in virtual network’. 

This step informs us that your job must work with a VNET:

 

Anasheh_Boisvert_10-1684817726339.png

 

Configure the settings as prompted and click ‘save’.

 

In VSCode, reference the subnet within your ASA job.  This step tells your job that it must work with a subnet.

In the JobConfig.json, setup your VirtualNetworkConfiguration as below:

 

Anasheh_Boisvert_11-1684817726348.png

 

Setting up an associated Storage Account

Add a storage account under Configure > Storage account settings > Add storage account:

 

Anasheh_Boisvert_13-1684817726357.png

 
 

Follow the prompts and configure your storage account settings.

 

IMPORTANT

  • To authenticate with connection string, you must disable the storage account firewall settings.
  • To authenticate with Managed Identity, you must add your Stream Analytics job to the storage account's access control list with the Storage Blob Data Contributor role.  If you do not give your job access, the job will not be able to perform any operations.  For more information on how to grant access, see Use Azure RBAC to assign a managed identity access to another resource.

 

Permissions

You must have at least the following Role-based access control permissions on the subnet or at a higher level to configure virtual network integration through Azure portal, CLI or when setting the virtualNetworkSubnetId site property directly:

 

Action

Description

Microsoft.Network/virtualNetworks/read

Read the virtual network definition

Microsoft.Network/virtualNetworks/subnets/read

Read a virtual network subnet definition

Microsoft.Network/virtualNetworks/subnets/join/action

Joins a virtual network

Microsoft.Network/virtualNetworks/subnets/write

Optional.  Only required if you need to perform delegation operations.

 

IMPORTANT
If the virtual network is in a different subscription than your ASA job, you must ensure that the subscription with the virtual network is registered for the Microsoft.Web resource provider.  You can explicitly register the provider by following this documentation, but it's automatically registered when creating the job in a subscription.

 

Limitations

  • VNET jobs require a minimum of 1 SU V2 (new pricing model) or 6 SUs (current).
  • Make sure the subnet range does not collide with ASA subnet range (i.e. do not use subnet range 10.0.0.0/16).
  • ASA job(s) and the virtual network must be in the same region.
  • The delegated subnet can only be used by Azure Stream Analytics.
  • You cannot delete a virtual network when it is integrated with ASA. You must disassociate or remove the last job* on the delegated subnet.
  • We do not support DNS refreshes currently.  If DNS configurations of your VNET are changed, you must re-deploy all ASA jobs in that VNET (subnets will also need to be disassociated from all jobs and reconfigured).  See here for more information.  

 

Access on-premises resources

No extra configuration is required for the virtual network integration feature to reach through your virtual network to on-premises resources. You simply need to connect your virtual network to on-premises resources by using ExpressRoute or a site-to-site VPN.

 

Pricing details

Outside of basic requirements listed in this document, virtual network integration has no extra charge for use beyond the Azure Stream Analytics pricing charges.

 

Troubleshooting

The feature is easy to set up, but that doesn't mean your experience is problem free.  If you encounter problems accessing your desired endpoint, contact Microsoft Support.

 

Feedback

For direct feedback on this capability, please reach out to askasa@microsoft.com.

 

 

Co-Authors
Version history
Last update:
‎Jul 31 2023 03:52 PM
Updated by: