User Profile
rajkm42
Copper Contributor
Joined 2 years ago
User Widgets
Recent Discussions
How to build lake data warehouse
Hi, We need to build new data warehouse. I am hearing multiple comments on this. We have very little budgets on Azure. 1. We have Azure elastic pool with 1 TB storage 2. D365 is our main application whose data is pushed to ADLS Gen2 using synapse link using dataverse 3. We have many other external applications(third party) which we are importing into elastic pool using SSIS running on Azure VM. Options being MS suggests to use Modern DWH where we don't need to store the data in sql databases. We are able to query Datalake using Synapse serverless pool but the performance is really bad. Moreover, we need to transform(apply various business logics) data after querying from datalake and also need to join data from external applications and then sql views are created to feed thru PBI reports. In serverless pool, how can we apply business logic? I mean stored procedures/complex logic... Is it possible to use Serverless pool to build our data warehouse? We are really struck here and not able to argue with many commenters. Please suggest an idea431Views0likes1CommentRe: Challenge: Modern data warehouse solution vs traditional data warehouse
Hi Ronen_Ariely Thanks for the reply. My question is actually 1. What is the best way to migrate to Azure Synapse based on my current process a. Using SSIS to pull data from datalake (Common Data Model) to Azure SQL DB b. Synapse is serverless pool c. What is the best way to implement business logic like Sql Views, Sql SPs d. Serverless pool offers $5 to process 1TB of data 2. How do I merge or query data between 2 datalakes(D365 CE and D365 FinOps) a. I mean cross database reference b. Is the serverless pool performance good with massive data files in lake Regards1KViews0likes0CommentsChallenge: Modern data warehouse solution vs traditional data warehouse
Dear all, I am new to Azure synapse analytics I mean know a little bit. We have D365 CE data exported to ADLS gen2. When I open Synapse studio, I can Built-in serverless sql pool. We currently have traditional datawarehouse(SQL server). ETL pushes data from various sources(Operational, csv, excel, third party apps) to sql server. How can I move to azure synapse analytics workspace to save costs and performance. Serverless pool is cheaper but it doesn't hold data physically in a table. It is a reference to csv file(Common Data Model/normal third party csv files). I have seen performance is not good when querying a table with 800k rows. I am googling to find how Azure Synapse analytics workspace replaces traditional dw. I am getting confused with the results. Basically, I need to include all the data sources in serverless pool what I used to get into my SSMS(as mentioned above). And also, I need to apply business logic(Stored procedures or views) before feeding to reporting platform. Target views will be a mixture of more than 1 sources of data in SSMS. I am very confused, How a serverless pool is best for me? Any reply is highly appreciable.1.1KViews0likes2CommentsChallenge: Modern sql datawarehouse(replace to sql server data warehouse)
Dear all, I am new to Azure synapse analytics I mean know a little bit. We have D365 CE data exported to ADLS gen2. When I open Synapse studio, I can Built-in serverless sql pool. We currently have traditional datawarehouse(SQL server). ETL pushes data from various sources(Operational, csv, excel, third party apps) to sql server. How can I move to azure synapse analytics workspace to save costs and performance. Serverless pool is cheaper but it doesn't hold data physically in a table. It is a reference to csv file(Common Data Model/normal third party csv files). I have seen performance is not good when querying a table with 800k rows. I am googling to find how Azure Synapse analytics workspace replaces traditional dw. I am getting confused with the results. Basically, I need to include all the data sources in serverless pool what I used to get into my SSMS(as mentioned above). And also, I need to apply business logic(Stored procedures or views) before feeding to reporting platform. Target views will be a mixture of more than 1 sources of data in SSMS. I am very confused, How a serverless pool is best for me? Any reply is highly appreciable.483Views0likes0Comments
Recent Blog Articles
No content to show