Forum Discussion
Rahtset
Dec 18, 2021Copper Contributor
Anyone have ideas on how I can use ADF to compare two files?
I have a scenario where I need to compare two blob files to see if their contents have changed and take action if they have. I tried this in Power Automate/Logic Apps, but the files can be around 130+ MB, which isn’t supported natively as the files have to then be chunked.
Is it possible to do this in ADF prior to copying data to a sink?
Is it possible to do this in ADF prior to copying data to a sink?
1 Reply
- azharamir13Brass ContributorCreate an ADF pipeline and add two Copy activities to the pipeline. Configure the source for each Copy activity to be the location of the file you want to compare, and configure the destination for each Copy activity to be a temporary location in your Azure environment, such as an Azure Blob Storage container. Add a Lookup activity to the pipeline and configure it to read the contents of the two files you copied in the previous step. You can specify the file format and delimiter in the Lookup activity's settings. Add a Conditional Split activity to the pipeline and configure it to compare the contents of the two datasets. For example, you could use an expression like @equals(activity('Lookup1').output.firstRow.Field1, activity('Lookup2').output.firstRow.Field1) to compare the first row of each dataset based on a specific field. You can define multiple conditions and specify the outcome for each condition using the Conditional Split activity's settings. Add a Sink activity to the pipeline and configure it to write the results of the comparison to a destination location, such as an Azure Blob Storage container or a SQL Server database. Save and publish the pipeline.