Over the last several months I've had a number of conversations with customers and partners who wished to explore native timeseries services in Azure to support their critical analytical application requirements. To help this process we collated the common asks we received into a single list of typical requirements, and mapped these against the native capabilities inherently available in Azure Data Explorer (ADX).
Without going into detail about these scenarios – this blog post explores the types of core functionality that typical timeseries data processing applications seek, and how “out of the box” functionality built into ADX aligns extremely well to meet these challenges head-on.
So what is timeseries anyway?
Well there are a number of definitions I've come across...
- One I like is outlined on Wikipedia – “time series is a series of data points indexed (or listed or graphed) in time order. Most commonly, a time series is a sequence taken at successive equally spaced points in time. Thus it is a sequence of discrete-time data“.
- Microsoft also publish a great article discussing in detail here - "Time series data is a set of values organized by time. Examples of time series data include sensor data, stock prices, click stream data, and application telemetry. Time series data can be analyzed for historical trends, real-time alerts, or predictive modeling."
However before we step into mapping Timeseries Requirements across into Azure Data Explorer, we first need to cover off exactly what that is.
What is Azure Data Explorer (ADX)?
In the context of timeseries data management...
- Azure Data Explorer is a VERY fast and VERY highly scalable data exploration service for log and telemetry data (which is timeseries in nature). It helps handle the many data streams emitted by modern software, so you can collect, store, and analyze data.
- Azure Data Explorer is ideal for analyzing large volumes of diverse data from any data source, such as websites, applications, IoT devices, and more. This data is used for diagnostics, monitoring, reporting, machine learning, and additional analytics capabilities.
For more information - please see the ADX MS Docs references here
Personally I’ve used Azure Data Explorer in several large scale timeseries related projects, and it has hands-down nailed it each and every time. In fact, don’t take it from me, check out one of our recent case studies here for a super cool and innovative Virtual Power Plant (VPP) solution ingesting data at massive scale.
What are the typical timeseries requirements that applications need?
Now we’ve talked to the backstory, what are the typical areas that timeseries data processing applications tend to look out for?
These are outlined below…
| Category | Functionality |
| Core - Functions | Statistical Functions |
| Support for Advanced Data Queries | |
| Down Sampling | |
| Time-Stamped Measurement | |
| Summarization | |
| Metadata Tagging | |
| Core - Data Management | Ingestion Source; Internal to Azure |
| Ingestion Source; External to Azure | |
| Data Compression | |
| Data Back-Filling | |
| Data Lifecycle Management | |
| Data Concurrency and Consistency | |
| Reference Data | |
| Performance | High Ingest Rate (write) |
| High Query Rate (read) | |
| Linear Engine Scalability | |
| Security | Encryption (in transit) – Transport Layer Security (TLS) |
| Authentication and Authorization | |
| Encryption (at rest) | |
| Availability, Resiliency | Backup / Restore |
| High Availability (HA) and Fault Tolerance (resiliency) | |
| Rolling Upgrades and Restarts | |
| Integration | Migration Tools |
| Integrate with BI Visualization Tools / Dashboard Platforms | |
| Integrate with External Tools / Frameworks | |
| Administration | Management Tools – Administration |
| Management Tools – Deployment Automation | |
| Monitoring / Audit / Logs | |
| Maintenance (SLA) |
Naturally we're not saying this list is finite per se, its just a summary of the common functionality we've been asked to address in the past.
Application timeseries requirements assessment paper download
If the above sounds of interest – you can download the ADX Assessment Paper attached at the bottom of this blog post.
- This whitepaper outlines 29 key functionality areas expected for typical timeseries data processing and analytics applications, and how these are met via the core capabilities provided within Azure Data Explorer (ADX) - the massive scale PaaS timeseries database hosted in the Azure Cloud.
- Information presented in this whitepaper provides a reference view on core ADX “out of the box” functionality against a set of common timeseries analytics and applications requirements. All information provided in this paper has been referenced back to official MS Docs pages.
Conclusion
OK, so there you have it, a short summary of the core ADX functionality available to store and process large scale timeseries data.
We hope this paper helps you explore how ADX can help solve your largest and most complex timeseries needs.
Adios for now…
|
Rolf Tesmer. Data & AI Cloud Solution Architect (CSA). Customer Success Unit (CSU). Australia. |