Introducing the Management Hub to Azure Data Factory
The management hub, accessed by the Manage tab in the Azure Data Factory UX, is a portal that hosts global management actions for your data factory. You can manage your connections to data stores and external computes, source control configuration, and trigger settings.How to startup your data flows execution in less than 5 seconds! (Public Preview)
Data Flow is Azure's low-code visual data transformation feature found in Azure Data Factory and Azure Synapse Analytics that makes building and deploying ETL super-easy by leveraging serverless Spark environments. Now you can execute data flows from a pipeline that startup in just seconds.ADF Release Notes 07/31/2020
The Azure Data Factory team is proud to announce the following features now available in the ADF UX: Array support for mapping data flow parameters Run a pipeline debug run of a data flow using a job cluster Filter by run-id in monitoring Parameterize key columns for by array for update operations in database sink Updated feedback experienceSecuring outbound traffic with Azure Data Factory's outbound network rules
The Outbound Rules feature in Azure Data Factory allows organizations to exercise granular control over outbound traffic, thereby strengthening network security. By integrating with Azure Policy, this feature also improves overall governance.
Announcing public preview of Time-To-Live (TTL) in managed virtual network
Today we’re excited to share the public preview of Time-To-Live (TTL) in managed virtual network. Background Managed virtual network provides customers with a secure and manageable data integration solution. But due to the limitation of architecture, we need to provision computes in managed virtual network each time we execute an activity. This can lead to relatively long queue times. Especially when you have small jobs that are executed sequentially, it’s not very efficient. So we introduce a TTL feature that allows users to reserve computes and these computes won’t be released within TTL period after the last activity execution. How does it work You can configure TTL settings in the integration runtime creation or edit page if this integration runtime enables managed virtual network. Since we use different computes from interactive authoring, copy activity and pipeline/external activity, so you can set different TTL value respectively. You can see more introductions and one demo in this video Learn more about managed virtual network and TTL Managed virtual network and managed private endpoints - Azure Data Factory | Microsoft Docs
Best practices for Azure Data Factory Integration Runtime
Integration runtime is a core component of Azure Data Factory. Users can use the integration runtime created by default in Azure Data Factory or create it themselves, depending on the actual situation. Since there are multiple types of Integration runtimes, it is necessary to properly select the most suitable type during actual use. We're excited to share a new article to help you determine the right integration runtime configuration for your scenario.How to access on premises data stores from ADF managed virtual network using Private Endpoint
Azure Data Factory managed virtual network is designed to allow you to securely connect Azure Integration Runtime to your stores via Private Endpoint. Your data traffic between Azure Data Factory Managed Virtual Network and data stores goes through Azure Private Link which provides secured connectivity and eliminates your data exposure to the public internet. Now we have a solution which leverages Private Link Service and Load Balancer to access on premises data stores or data stores in another virtual network from ADF managed virtual network. To learn more about this solution, visit Tutorial - access on premises SQL Server. You can also use this approach to access Azure SQL Database Managed Instance, see more in Tutorial - access Azure SQL Database Managed Instance.
