High throughput streaming ingestion to Synapse SQL pools
Published Sep 22 2020 06:00 AM 3,763 Views

Now Generally Available: High throughput streaming ingestion into Synapse SQL pools

 

Azure Synapse is an analytics service that seamlessly brings together enterprise data warehousing and Big Data analytics workloads. It gives customers the freedom to query data using on-demand or provisioned resources at scale, and with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs. A Synapse SQL pool refers to the enterprise data warehousing features that are generally available in Synapse.

 

As more and more customers require their Modern Data Warehouse patterns and reporting environments to be real-time representations of their business, streaming ingress into the warehouse is becoming a very core requirement. 

 

Azure Stream Analytics is a market leading Serverless PaaS offering for real-time ingress and analytics on Streaming data. Starting at a very low price point of USD $0.11 per Streaming Unit per hour, it helps customers process, and reason over streaming data to detect anomalies and trends of interest with ultra-low latencies. Azure Stream Analytics is used across a variety of industries and scenarios such as Industrial IoT for remote monitoring and predictive maintenance, Application telemetry processing, Connected vehicle telematics , Clickstream analytics, Fraud detection etc.

 

Today, we are announcing the General Availability of high throughput streaming data ingestion (and inline analytics) to Synapse SQL pools from Azure Synapse Analytics, that can handle throughput rates even exceeding 200MB/sec. This is expected to support even the most demanding warehouse workloads such as real-time reporting, dashboarding and many more. More details can be found in the feature documentation.

 

With Azure Stream Analytics, in addition to high throughput ingress, customers can also run in-line analytics such as JOINs, temporal aggregations, filtering, real-time time inferencing with pre-trained ML models, Pattern matching, Geospatial analytics and many more. Custom de-serialization capabilities in Stream Analytics help with ingress and analytics on any custom or binary streaming data formats. Additionally, developers and data engineers can express their most complex analytics logic using a simple SQL language, that is further extensible via Javascript and C# UDFs. 

 

For IoT specific scenarios, Azure Stream Analytics enables portability of the same SQL query between cloud and IoT edge deployments. This provides several opportunities for customers such as pre-processing, filtering or anonymization of data the edge.

 

pic1.PNG

  Azure Stream Analytics output to Synapse SQL table

 

Feedback and engagement

Engage with us and get early glimpses of new features by following us on Twitter at @AzureStreaming. The Azure Stream Analytics team is highly committed to listening to your feedback and letting the user's voice influence our future investments. We welcome you to join the conversation and make your voice heard via our UserVoice page.

Version history
Last update:
‎Sep 22 2020 02:21 PM
Updated by: