Over the last several months I've had a number of conversations with customers and partners who wished to explore native timeseries services in Azure to support their critical analytical application requirements. To help this process we collated the common asks we received into a single list of typical requirements, and mapped these against the native capabilities inherently available in Azure Data Explorer (ADX).
Without going into detail about these scenarios – this blog post explores the types of core functionality that typical timeseries data processing applications seek, and how “out of the box” functionality built into ADX aligns extremely well to meet these challenges head-on.
So what is timeseries anyway?
Well there are a number of definitions I've come across...
One I like is outlined on Wikipedia– “time series is a series of data points indexed (or listed or graphed) in time order. Most commonly, a time series is a sequence taken at successive equally spaced points in time. Thus it is a sequence of discrete-time data“.
Microsoft also publish a great article discussing in detail here - "Time series data is a set of values organized by time. Examples of time series data include sensor data, stock prices, click stream data, and application telemetry. Time series data can be analyzed for historical trends, real-time alerts, or predictive modeling."
However before we step into mapping Timeseries Requirements across into Azure Data Explorer, we first need to cover off exactly what that is.
What is Azure Data Explorer (ADX)?
In the context of timeseries data management...
Azure Data Exploreris aVERYfast andVERYhighly scalable data exploration service for log and telemetry data (which is timeseries in nature). It helps handle the many data streams emitted by modern software, so you can collect, store, and analyze data.
Azure Data Exploreris ideal for analyzing large volumes of diverse data from any data source, such as websites, applications, IoT devices, and more. This data is used for diagnostics, monitoring, reporting, machine learning, and additional analytics capabilities.
For more information - please see the ADX MS Docs referenceshere
Personally I’ve usedAzure Data Explorerin several large scale timeseries related projects, and it hashands-down nailed it each and every time.In fact, don’t take it from me, check out one of our recent case studiesherefor asuper coolandinnovative Virtual Power Plant (VPP)solution ingesting data at massive scale.
What are the typical timeseries requirements that applications need?
Now we’ve talked to the backstory, what are the typical areas that timeseries data processing applications tend to look out for?
These are outlined below…
Core - Functions
Support for Advanced Data Queries
Core - DataManagement
Ingestion Source; Internal to Azure
Ingestion Source; External to Azure
Data Lifecycle Management
Data Concurrency and Consistency
High Ingest Rate (write)
High Query Rate (read)
Linear Engine Scalability
Encryption (in transit) – Transport Layer Security (TLS)
Authentication and Authorization
Encryption (at rest)
Backup / Restore
High Availability (HA) and Fault Tolerance (resiliency)
Rolling Upgrades and Restarts
Integrate with BI Visualization Tools / Dashboard Platforms
Integrate with External Tools / Frameworks
Management Tools – Administration
Management Tools – Deployment Automation
Monitoring / Audit / Logs
Naturally we're not saying this list is finite per se, its just a summary of the common functionality we've been asked to address in the past.
Application timeseries requirements assessment paper download
If the above sounds of interest – you candownloadtheADX Assessment Paper attached at the bottom of this blog post.
This whitepaper outlines29 key functionality areasexpected for typical timeseries data processing and analytics applications, and how these are met via the core capabilities provided withinAzure Data Explorer (ADX) - the massive scale PaaS timeseries database hosted in the Azure Cloud.
Information presented in this whitepaper provides areference viewon core ADX “out of the box” functionality against a set of common timeseries analytics and applications requirements. All information provided in this paper has been referenced back to official MS Docs pages.
OK, so there you have it, ashort summaryof thecore ADX functionalityavailable to store and process large scaletimeseries data.
We hope this paper helps you explore how ADX can help solve your largest and most complex timeseries needs.