Forum Discussion
TonDeli1337
Jan 16, 2024Copper Contributor
How to automate processing files for Power BI?
First a disclaimer because I'm totally noob with all the Azure stuff...
I'm getting finance transactions in Excel and csv format from several stakeholders. Currently I'm doing all this manually using e-mails, OneDrive and have to do some manual data transformation before I visualize data with Power BI. I'd like to automate this whole process because during this year amount of files is going to increase that I don't have any time to do this manually.
My current plan:
- BlobStorage > Collecting Excel and csv files from stakeholders
- Data Factory > Transform data into readable and uniform format
- Azure PostgreSQL > Store data
- Power BI > Data modeling and data visualization
I was hoping to test Synapse or DataLake for this solution but I have no idea how the first step is used with them.
Any suggestions how should I build up my solution and I also appreciate if you can share links where I can find more information.
2 Replies
Sort By
- Senthil GopalCopper ContributorYour plan is on the right track, and you've identified the key components needed for automating the processing of files for Power BI. Here's a more detailed breakdown of how you can implement your solution:
Blob Storage: Use Azure Blob Storage as your central storage repository for receiving Excel and CSV files from stakeholders. You can create a container for each stakeholder or group of stakeholders to organize the files.
Azure Data Factory: Set up an Azure Data Factory pipeline to automate the ingestion and transformation of data from Blob Storage to Azure PostgreSQL. You can use Data Factory's copy activity to move files from Blob Storage to your PostgreSQL database. Additionally, you can define data transformation activities using Data Flows or Mapping Data Flows to clean and prepare the data before loading it into your database.
Azure PostgreSQL: Store your transformed data in Azure PostgreSQL. You can create tables in PostgreSQL to represent your data schema and load the cleaned data from Data Factory into these tables.
Power BI: Connect Power BI to your Azure PostgreSQL database to access and visualize the data. You can create Power BI reports and dashboards based on the tables in your PostgreSQL database, allowing you to analyze the financial transactions efficiently.
Regarding your consideration of Synapse or Data Lake, while they offer powerful data analytics capabilities, they might be overkill for your current requirements. However, if you anticipate needing advanced analytics or handling very large volumes of data in the future, they could be worth exploring.
For more information and guidance on each component:
Azure Blob Storage: Azure Blob Storage documentation
Azure Data Factory: Azure Data Factory documentation
Azure PostgreSQL: Azure Database for PostgreSQL documentation
Power BI: Power BI documentation
Additionally, you may find tutorials and guides on Microsoft Learn or in the Azure Architecture Center that walk you through similar scenarios and provide step-by-step instructions for setting up your solution. Good luck with automating your data processing workflow!