Forum Discussion
AjayGopu
May 10, 2024Copper Contributor
How to Achieve Coupe of functionalities in Azure Data Factory Dyanmically.
Hi Team. I have below scenarios as part of my business requirement. These requirements has to achieved dyanmically using Azure Data Factory Data Flows or Pipelines. Note : Requirement is not...
balasubramanim
Feb 06, 2025Iron Contributor
You can achieve this dynamically in Azure Data Factory (ADF) using Pipelines and Mapping Data Flows without Function Apps or Databricks.
Steps:
1. Loop through files – Use Get Metadata + ForEach to iterate over CSVs in Blob Storage.
2. Get row count – Use Aggregate transformation in Data Flow.
3. Apply conditional logic – Use If Condition to check if row count > 0.
4. Validate rows – Use Derived Column + Filter transformation to check for misaligned columns (unquoted commas).
5. Move valid/invalid rows –
Valid Rows → Convert to Parquet
Invalid Rows → Save as CSV in another folder
This keeps the pipeline fully dynamic and automated