Accessing On-Premise File Systems from Azure Logic Apps

VeeraReddy · ‎Jun 09 2020

Overview:

In today's world, hybrid integration is prominent with many challenges on accessing on-premise resources on Cloud environment. Here, I thought of sharing my insights on available options to access On-Premise File Systems as many users have various concerns like Size limitations , access to storage accounts behind firewall. Let's see what are the options we have to access On-Premise file systems and how we can access them in Logic Apps.

It's as of my knowledge and there might be other ways to access On-Premise files through Express route , VNET peering and others. I am more over concentrating on the major concrete solutions we have at present in terms of respective services/resources.

On-Premise Data Gateway
Azure File Share - File Sync service (How about when its behind firewall)
Express route , VNET peering with Integration Service Environment (ISE)
SFTP& FTP servers

On-Premise Data gateway:

OPDG is windows service which can be deployed in on-premise windows servers to access LOB systems on premise such as File Systems (Even Microsoft File over clustering is supported), SQL Server , IBM MQ server , SAP etc. OPDG service underlying service is Azure Service Bus which is securely connects to the LOB systems on on-premise without opening additional firewalls and ports except the ports specific to LOB systems. OPDG service currently works for LogicApps,Power Apps and Power BI.

Pro's:

No need to open additional firewall and ports
Easy to configure and doesn't need much effort ,One gateway is enough o access all data sources
Fail over clustering for high availability
Can be installed in any server with in the domain network has access to the resources but performance could be impacted.

Limitations:

As it's underlying architecture is on Azure Service Bus , the request size limit is max of 2 MB and Response limit is 8 MB except for File systems which is 30 MB for all I/O operations except Create file which is 30 MB
Connection timeout will be 30 seconds
User can be part of only one tenant directory

Azure File Share-File Sync Service:

In most of scenarios, we use Azure File Server for accessing the common tools, Config files, logs file etc.. In addition to that Microsoft provides the Azure File Sync service which can be deployed on your windows machined to mount the Azure File services on your on-premise file systems. So you can use the Azure File share when you would like to use the file systems on both Microsoft cloud and On-Premise.

Bit difficult to configure as various components has to be setup
Easy to mount on multiple windows servers to access common tools setup files etc..
File sync service does cache the Azure File server for quick access of the files
Limits are bound to Azure File Share limits which is in TB's Tera bytes.

However, I am not going to details about installation and configuration and quickly jump on how we could access these from Logic Apps.

First thing that comes to out mind is security when we move data to cloud and accessing it from other Azure resources or any other clients. So storage account has Firewall settings along with the access keys or SAS keys for restricting the access and authorizing operations on storage account. When we consider this how could we access the storage accounts in Logic Apps which are behind the firewall and both are region same region. We can use the REST API's with SAS URL of Azure File Share to overcome this scenario as we won't be able to see the out of box connectors when they are in same region.

ISE Environment: Dedicated environment isolate from multi tenant shared environment.

ISE environment as we know can be used to access the all resources with in same virtual network which can have Express route or VNET to peer configuration to access data resources in on-premise servers. It's pretty straight forward to access on-premise file systems using ISE version connectors.

Dedicated resources will be allocated and isolate from the shared environment
All resources with in the same virtual network or access to other networks if peer networking or Express route established can be accessed
Cost is primary concern as its fixed amount respective to SKU's selected
Flexibility to scale up and scale down with additional units
Expose with Static IP

SFTP /FTP server:

Most resilient approaches for accessing file servers hosted in either Windows or Linux servers through SFTP/FTP servers is pretty common scenario now. Using SFTP/FTP servers we can access the files of size upto 1 GB with support of chunking.

Secure file transfer with SSH -2.0 protocol
Trigger file size limit is 15 MB and chunking doesn't support on trigger as expected by deign
Actions can be used to read data in streams of 15 to 50 MB chunks which can be set according to network speed. Limit is of 1 MB
Connection timeout is 20 seconds.
New filed to get the metadata of the files on both triggers and actions which can be suppressed in performance issues.

Refrences: