User Profile
RamonMA1360
Copper Contributor
Joined Nov 06, 2020
User Widgets
Recent Discussions
Setting up a new Data Factory
Hi all, I'm not a data expert/novice, but I do love data. I'm looking to up my companies game when it comes to ingesting and storing data from our various sources. We currently have a few on prem databases for applications we run, MSSQL and mySQL. But we have tons of software as a services, all with their own REST APIs. I have a hodgepodge of Power Automate Flows, CDS DataFlows, manual imports, csv dumps, etc etc, and these could point to a number of places. I do like the Common Data Service Dataflows feature which uses Power/M Query to transform the data and put it in a CDS entity, however I don't like CDS entities for use outside of the M365 ecosystem. My end result I want is no matter where the data lives I can get it into a Azure SQL Database. I stumbled upon Data Factory this week and it seems to fit the bill at a high level and I can do a basic copy data pipeline. My question is, how well is it dealing with REST APIs that are not official integrations? I'm wanting to transform (Data Flows?) the data without having to store that data in a intermediate area. Do data flows really only support "live" transformations from the select few sources? For our cloud infrastructure we are a Azure shop, so I'd like to keep our solution within Azure. Any assistance would be appreciate and if I sound like a complete noob that has no idea about what I'm getting myself into, it's because that is the case 🙂894Views0likes1Comment
Recent Blog Articles
No content to show