Public Preview: Azure Monitor pipeline transformations

The Azure Monitor pipeline extends the data collection capabilities of Azure Monitor to edge and multi-cloud environments. It enables at-scale data collection (data collection over 100k EPS), and routing of telemetry data before it's sent to the cloud. The pipeline can cache data locally and sync with the cloud when connectivity is restored and route telemetry to Azure Monitor in cases of intermittent connectivity.  Learn more about this here - Configure Azure Monitor pipeline - Azure Monitor | Microsoft Learn

 Why transformations matter

• Lower Costs: Filter and aggregate before ingestion to reduce ingestion volume and in turn lower ingestion costs 

• Better Analytics: Standardized schemas mean faster queries and cleaner dashboards. 

• Future-Proof: Built-in schema validation prevents surprises during deployment. 

Azure Monitor pipeline solves the challenges of high ingestion costs and complex analytics by enabling transformations before ingestion, so your data is clean, structured, and optimized before it even hits your Log Analytics Workspace.  

Check out a quick demo here - 

 

Key features in public preview

1. Schema change detection  

One of the most exciting additions is schema validation for Syslog and CEF :  

• Integrated into the “Check KQL Syntax” button in the Strato UI. 

• Detects if your transformation introduces schema changes that break compatibility with standard tables. 

• Provides actionable guidance: 

• • Option 1: Remove schema-changing transformations like aggregations. 

• • Option 2: Send data to a custom tables that support custom schemas. 

This ensures your pipeline remains robust and compliant with analytics requirements.  

For example, in the picture below, extending to new columns that don't match the schema of the syslog table throws an error during validation and asks the user to send to a custom table or remove the transformations.  

 

 

While in the case of the example below, filtering does not modify the schema of the data at all and so no validation error is thrown, and the user is able to send it to a standard table directly.  

 

2. Pre-built KQL templates 

• Apply ready-to-use templates for common transformations. 

• Save time and minimize errors when writing queries. 

 

 

3. Automatic schema standardization for syslog and CEF 

• Automatically schematize CEF and syslog data to fit standard tables without any added transformations to convert raw data to syslog/CEF from the user.  

4. Advanced filtering

• Drop unwanted events based on attributes like:  

• • Syslog: Facility, ProcessName, SeverityLevel. 

• • CEF: DeviceVendor, DestinationPort. 

• Reduce noise and optimize ingestion costs. 

5. Aggregation for high-volume logs

• Group events by key fields (e.g., DestinationIP, DeviceVendor) into 1-minute intervals. 

• Summarize high-frequency logs for actionable insights. 

6. Drop unnecessary fields

• Remove redundant columns to streamline data and reduce storage overhead. 

 Supported KQL sunctions   

1. Aggregation 

• summarize (by), sum, max, min, avg, count, bin  

2. Filtering  

• where, contains, has, in, and, or, equality (==, !=), comparison (>, >=, <, <=)  

3. Schematization   

• extend, project, project-away, project-rename, project-keep, iif, case, coalesce, parse_json  

4. Variables for Expressions or Functions  

• let  

5. Other Functions  

• String: strlen, replace_string, substring, strcat, strcat_delim, extract  
• Conversion: tostring, toint, tobool, tofloat, tolong, toreal, todouble, todatetime, totimespan 

 Get started today  

Head to the  Azure Portal and explore the new Azure Monitor pipeline transformations UI. Apply templates, validate your KQL, and experience the power of Azure Monitor pipeline transformations. Find more information on the public docs here -  Configure Azure Monitor pipeline transformations - Azure Monitor | Microsoft Learn