Forum Discussion

Amila_19's avatar
Amila_19
Copper Contributor
Jul 12, 2020

Running Summarizing jobs on ADX

Hello,

We are trying to use ADX to build an event processing solution and are currently evaluating some of its capabilities. Wanted to know whether ADX provides a way to run a scheduled job and do Data Summarization.

 

We are building an event processing solution where we capture events from a set of microservices running on-prem  and Publish to Event Hubs. ADX pulls data from the EventHubs and store in respective Databases. Currently what we store is raw events. From these events we need to prepare several charts, which will give certain aggregates like sum count grouped by different event attributes and rolled up by different time scales.  We've been planning to prepare summaries while querying data , but given the type of queries the system would receive and the data volume ADX was scanning to produce a single summary, we felt it'd be better both performance wise and cost wise to summarize data offline and just retrieve results while querying. So the question is, whether ADX offers in-built feature to do this sort of processing? What would be the recommended way to summarize data with ADX? If the ADX is not the correct solution to do this sort of processing, what would be the correct product?

Thanks in advance!!

1 Reply

  • Amila_19 hey per your description it seems that you have couple of options 

    1. update policy - which provide an light ETL capabilities on your service for new ingested data 

    2. metalized viewexpose an aggregation query over a source table. Materialized views always return an up-to-date result of the aggregation query (always fresh).

     

Resources