Forum Discussion
Need help with large data sets
Hi all wondering if someone can help me with a solution!
We have 10+ suppliers that send us multiple excel spreadsheets on a monthly basis with millions of lines of data. The data unfortunately doesn't come formatted the same from each one but does contain the same information.
What we need to do which I need help with please!:
Combine all information from each supplier so it is formatted the same ( really stuck as its millions of lines and don't know the best way to store it without maxing excel and making documents slow)
Create a report or pivot on all the data on a monthly and combined yearly basis (issue is the amount of data being millions if lines and presenting this without excel saying you have gone over the maximum amount of lines.
Please could someone help me with the above i would really appreciate it.
We have 10+ suppliers that send us multiple excel spreadsheets on a monthly basis with millions of lines of data. The data unfortunately doesn't come formatted the same from each one but does contain the same information.
What we need to do which I need help with please!:
Combine all information from each supplier so it is formatted the same ( really stuck as its millions of lines and don't know the best way to store it without maxing excel and making documents slow)
Create a report or pivot on all the data on a monthly and combined yearly basis (issue is the amount of data being millions if lines and presenting this without excel saying you have gone over the maximum amount of lines.
Please could someone help me with the above i would really appreciate it.
3 Replies
Aread62127 Have a look at PowerQuery (PQ). It allows you to connect to data sources without loading everything into Excel. Since you ask the question, I assume that you are not familiar with it. Perhaps wise to have a look at the information in the attached link. Then you will get an idea of what PQ can do.
- Thanks for the reply
Can I ask though, when I create a query and then want to present the results or produce a pivot etc, if the results are 1m+ lines will this cause an issue for excel?Aread62127 No problem, you can load the data into the Data Model. More about that in, for example, the link below:
