Forum Discussion
Looking for guidance on designing an Azure data analytics pipeline for reporting
I’m working on modernizing an old reporting workflow that currently runs on a few on-premises databases and scheduled scripts.
The current process collects operational data from multiple systems, performs some basic transformation and aggregation, and then generates reports for different business teams. As the data volume is growing, the existing setup is becoming difficult to maintain and slow to refresh.
I’m looking for an Azure-based architecture that can ingest data from different sources, store both raw and processed data, run scheduled transformations, and make the final datasets available for reporting tools like Power BI.
Would appreciate any suggestions on the recommended architecture, especially around data storage, transformation, refresh performance, and cost control.
Thanks
1 Reply
Hi, for a modern Azure reporting pipeline I would usually start with ADLS Gen2 as the landing zone and keep a bronze/silver/gold layout for raw, cleaned, and curated data. Use Data Factory or Fabric Data Factory for ingestion, then Databricks, Synapse, or Fabric notebooks depending on your team skills. Put secrets in Key Vault, monitor the pipelines with Azure Monitor, and keep Power BI pointed at curated tables instead of raw operational data.