Forum Discussion
Event Driven Centralized Metadata lake using Azure Services
Sagar_Lad Could you describe in more details what you want to build?
Hi DennesTorres ,
Thanks for the reply.
I would like to build a an event driven product as follows :
- Create a centralized data lake for Metadata
- Metadata is being maintained by different teams in different type of databases like SQL DB, Synapse etc
- End goal is to push metadata from different data sources based on any change/update using event via API to centralized metadata lake
- Metadata available in a centralized metadata lake should be available in near real time
Sagar_Lad You explanation is not enough to give you an answer that I would be confident it's the right one. Probably the best option for you would be hiring a consulting service to analyze the details of your environment and propose a solution.
I'm not confident you need to move this metadata to another place, although in fact I don't know what this metadata is. The systems holding this metadata, such as Synapse, are used to hold BIG data warehouses, so you don't have the need to move them again, that's the idea of a data lake.
As long as these are not production systems, you can build a virtual data lake using Synapse serverless and pointing to this data, even building aggregations of the data. But again, I don't understand your scenario, I'm not sure if that's what you need.Besides that, logic apps and azure functions can deal with events, receiving events and triggering actions.