Forum Discussion
SrikanthSolanki
Oct 29, 2026Copper Contributor
Azure Data Factory: How do you handle late-arriving records in incremental loads?
I am looking at approaches for implementing incremental loads in Azure Data Factory when records can arrive late or existing records can be updated after the initial load.
A watermark based on a source timestamp works for many scenarios, but a strict timestamp > last_watermark filter can miss records if multiple rows share the same timestamp or if updates arrive outside the expected window.
For a reliable pipeline, would you recommend:
- Using an overlapping extraction window and deduplicating records in the target?
- Tracking a source-specific change version or change tracking mechanism?
- Using a combination of a watermark and periodic reconciliation?
The source is a relational database, and the target needs to remain consistent without reloading the entire dataset every time.
What approach would you recommend, and what edge cases should be considered when implementing this in Azure Data Factory?
No RepliesBe the first to reply