In the world of Azure data platform, Cosmos DB and Azure Data Explorer are two big data systems that complement each other across operational and analytical workloads respectively. You could make better business decisions if you get an ability to analyse data in near real-time and that’s what the integration of these two systems empowers you to achieve.
Azure Data Explorer(ADX) is an append-only big data analytical database for low latency near real-time analytics scenarios. Azure Cosmos DB is a globally distributed, multi-model NoSQL database.
In traditional data analytics solutions, you have to go through the time consuming curation processes to bring data in a shape to be consumed by the end users.
Now with these advanced systems, it is practically possible to ingest and query raw operational data in easy and effective manner as depicted in the following solution architecture –
Benefits of this pattern
Benefits of Azure Data Explorer in this pattern
Key Features of Azure Data Explorer
There are numerous potential benefits of this architecture from business growth perspective. Just to give you an idea on its value proposition which is applicable to most organizations across diverse industries, sharing few examples -
Similarly in health, finance and many other industries, heaps of scenarios where you could make better business decisions using this pattern.
The next obvious question would be around data redundancy and cost impact of this solution. You could optimize the cost of this solution by managing the data retention policies across all these services. For example, Cosmos DB is an operational hot store where data is stored for few days, Azure Data Explorer is an analytical warm store where frequently accessed data is stored, export rest of the old data to cold storage which is Azure BLOB or data lake storage in this solution. It is very easy to configure caching and data retention policies in ADX so you could easily change it, seamlessly query across warm and cold store depending on your requirements.
Demonstration of solution with hands on lab
To help you understand the end-to-end flow of this solution, hands on lab with step by step guidance has been put together along with working code samples so you can try and test it on your own with the simulated data. Brief on what is being covered in this lab –
The lab is publicly available here at GitHub.
Try it out and share your feedback!
Near real-time analytics solution can be built in multiple ways using different azure services, this lab describes one of the possible scenarios. Similar outcomes can be achieved using other azure services which are not covered in this lab.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.