User Profile
RamonMA1360
Copper Contributor
Joined 5 years ago
User Widgets
Recent Discussions
Understanding AlertRow in Dataflows
I'm revisiting Azure Data Factory after the introduction of REST Apis as supported sources for dataflows. We manage multiple client M365/Azure environments and using the MS Graph API I'm trying to pull different types of reports and info from their Tenant. My pipeline will start with pulling a list of our Tenant Ids, foreach tenant, run the dataflow that pulls their user list and then sinks that to our Azure SQL DB. Main parts works as expected, and inserts run just fine, however I want to delete users that no longer exist in the API source from the SQL table or update them if they already exits and of course insert if they are new. I can't seem to figure out how to accomplish this. Currently works with Inserting only. Sink Settings when trying to do updates/deletes/inserts. Alter Row Transformation I can't for the life of me figure out what expressions to put in here. Id column is a GUID from MS for that specific user. My thought was I needed to add a 2nd source to compare against, that 2nd source being the target table for my sink, but my google fu is proving very weak and can't seem to figure out how to use the expression language to reference columns from another source in the AlertRow step. Am I going about this the wrong way? My sink table is going to have multiple clients in it so I can't drop the table each time I run the data flow, I need it to be to flexible, as there might be times I just want to update a single client's data.Setting up a new Data Factory
Hi all, I'm not a data expert/novice, but I do love data. I'm looking to up my companies game when it comes to ingesting and storing data from our various sources. We currently have a few on prem databases for applications we run, MSSQL and mySQL. But we have tons of software as a services, all with their own REST APIs. I have a hodgepodge of Power Automate Flows, CDS DataFlows, manual imports, csv dumps, etc etc, and these could point to a number of places. I do like the Common Data Service Dataflows feature which uses Power/M Query to transform the data and put it in a CDS entity, however I don't like CDS entities for use outside of the M365 ecosystem. My end result I want is no matter where the data lives I can get it into a Azure SQL Database. I stumbled upon Data Factory this week and it seems to fit the bill at a high level and I can do a basic copy data pipeline. My question is, how well is it dealing with REST APIs that are not official integrations? I'm wanting to transform (Data Flows?) the data without having to store that data in a intermediate area. Do data flows really only support "live" transformations from the select few sources? For our cloud infrastructure we are a Azure shop, so I'd like to keep our solution within Azure. Any assistance would be appreciate and if I sound like a complete noob that has no idea about what I'm getting myself into, it's because that is the case 🙂
