Forum Discussion
[Design Pattern] Handling race conditions and state in serverless data pipelines
Hello community, I recently faced a tricky data engineering challenge involving a lot of Parquet files (about 2 million records) that needed to be ingested, transformed, and split into different ent...
Using Durable Entities is a strong choice here. They act as stateful actors and let you maintain a strictly sequential counter while running large-scale parallel transformations. This avoids external locking, reduces complexity, and prevents ID collisions.
Common alternatives in similar ETL/serverless designs include:
- SQL or Cosmos DB sequences when a transactional database already exists.
- Queue-based atomic increments, though these are less reliable for strict ID ordering.
- ADF sequential mode, which is safe but often too slow for high-volume ingestion.
Your Durable Entities approach is the right balance of scalability, correctness, and simplicity for a serverless-only stack.