IoT Device connectivity to Azure via FTP

[PBradz]

Hi All,

I'm working with 30+ waterflow meters and the connectivity option is limited to FTP (I'm hoping SFTP is supported but haven't found a definitive yes/no in the Doc). The data to be passed is simply csv files from several sensors.

The current method of retrieving data is via a Tech visiting once a month and downloading the data via USB-to-Serial. They visit to check status and clear debris as well, so data collection is not the only task. The meters all have Cellular Modems and can connect to the Internet. Limiting factor is FTP being the only data transfer method supported. What are options for getting Data into Azure via FTP (ideally into IoT Hub to match the workflows we use for other sensors)?

[#2]

@PBradz I'll work through some possible solutions and hopefully this may be beneficial to someone else.

I/we are looking at 4 options to resolve our connectivity issue. The workflow for other devices, that fortunately support more than just FTP, is as follows;

• Connect to Azure IoT Hub (via one of the supported methods), connection(mostly JSON data) passed to Azure Logic App
• Azure Logic App identifies connection/data and routes data to 2 locations; a 3rd party Platform(integrates to SalesForce, and external partner Orgs) and our internal db
• Web Services use the internal db info to present data to internal and external users via webpages, visualizations, map overlays, etc.

The 4 options so far are as follows, in a successively more complex order (for us):

1. Route data in via another path; possibilities include going to our 3rd Party Platform or the Web Services platform, and pulling the data into our db from there.
2. Utilizing an App Services Web App for the FTP upload, have our Logic App monitor the folders for changes (uploads) and move that data per other workflows.
3. Use an Azure ACI with Azure persistent Storage as the FTP target, have our Logic App monitor the folders for changes (uploads) and move that data per other workflows.
4. Build a custom Azure IoT Protocol Gateway to act as a Bridge to Azure IoT Hub; accept FTP via the Gateway and pass to IoT Hub via AMQP. From there the workflow is the same as other devices.

Happy to hear any other suggestions or recommendations!

[#3]

Hi @PBradz,
I am currently having the same problem in my company where we want to let weather stations ingest data via FTP. Currenlty the folder structure on the FTP server offers one folder per device. So my idea was to use:
your option 3)
Host a FTP server which provides FTP protocol support via a Azure Service / VM. The backing storage is monitored by some serverless logic on Azure and "microbatched"/parsed/forwared to the streaming (fast path) architecture.
My idea hear is to use IoT Hub also for "provisioning" the FTP server by having each weather station defined as "Iot device". The device agent (sideloaded with the ftp server) would react to new "device registrations" and create a new folder in the FTP server data store as well as provision the ACLs for the folder.

or

your option 4)
forget about a "real" ftp server but write a "facade FTP Bridge to Azure IoT Hub" which from the outside looks like a FTP server but has no storage built-in. But this would only allso for "write" scenarios not if some other user whats to use the FTP to read weather data from.

Did you choose one solution for your use-case? If yes, did you re-use opensource components for is?

Thanks in advance,
Dominic

[#4]

Dominic, apologies for delay in responding...I'll go back and check for correct info, however, as I remember it we ended up with a linux VM w/ IoT Edge installed; FTP went to Linux VM and Edge would transfer "new" files added to a Dir into IoT Hub which would direct as needed. For your case you could also have files copied to Azure Blob storage that is shared. I'll dig up my notes and add any details or corrections....