Blog Post

Azure Observability Blog
7 MIN READ

Schema at PR speed: schemaless log ingestion in Azure Monitor

Ivan_Varnitski's avatar
Nov 17, 2026

Developers add log fields in a pull request; DCR changes wait for a ticket. Turn on schemaless ingestion in Azure Monitor (public preview) to store new fields without a DCR update.

It's a quiet Monday, and a coding agent opens a pull request titled "Upgrade payments client to v3." It bumps the SDK version, renames two methods and fixes a test the new version broke. Fourteen files, all checks green. Your teammate approves it in six minutes.

No logging code changed in that diff. What changed was the provider's response, which your checkout service spreads into every log event. Version 3 adds decline_reason, retry_after_ms and an issuer object. The Logs Ingestion API accepts every batch with a 204 and stores none of the three, because your data collection rule (DCR) has never heard of them.

Three weeks later, card declines in one region jump to 9 percent. The on-call engineer queries the logs and finds 4,000 rows that say card_declined. The provider's support team replies within the hour: the issuer was timing out, and every response told you to retry after 30 seconds. The answer had been in decline_reason and retry_after_ms on every response for three weeks.

Nobody made a mistake here. The new fields came in with a dependency upgrade, so they never appeared in a diff anyone could review.

Log fields now change as fast as pull requests merge, including pull requests written by agents, while DCRs change through tickets. In this post, we'll show how schemaless ingestion in Azure Monitor, new in public preview at Ignite, stores the fields that arrive in between, how to promote and shape them, and what changes if you're moving off the HTTP Data Collector API.

At a glance

  • What: Schemaless ingestion in Azure Monitor gives custom logs a dynamic schema: new, undeclared top-level properties land in a declared dynamic column, the overflow column, without a data collection rule (DCR) change per field. No new table columns.
  • Status: Public preview, Microsoft Ignite 2026 (November 2026). 
  • Configure: Use DCR API 2025-05-11 or later. Enable within header.CustomStream in DCR transformations
  • Shape the data: Use KQL bag functions pack_all(), bag_remove_keys(), bag_merge().
  • For: DCR owners, teams sending custom JSON logs, and teams leaving the HTTP Data Collector API.
  • Start: Enable schemaless ingestion.

Your log schema runs on two clocks

Schema drift in your logs and changes to your DCR run on different clocks, and schemaless ingestion is built for the gap between them. The checkout service changed its log shape on a Monday morning, in a pull request nobody needed to think hard about. The DCR that stores those logs needs a request, an update to its stream declaration, transformation and table, and a deployment, and it's slow on purpose, because every change affects the queries and alerts built on the table.

The trouble is that a stream declaration does two jobs. It's a contract: the names and types your queries, alerts and workbooks rely on, which deserve review. It's also an allowlist: any top-level property it doesn't list isn't stored. Platform teams review DCR changes because of the first job, and the second job is why the new payment fields were dropped.

 

 Contract (declared columns)Long tail (overflow column)
Example fields
OrderId, Status, DeclineCode
decline_reason, retry_after_ms, issuer
TypesDeclared in the streamWhatever arrives, stored as keys in a dynamic bag
How it changesReviewed DCR changeArrives with the data, no DCR change
Who owns itPlatform teamService teams and their dependencies

Schemaless ingestion keeps the contract and drops the allowlist role for everything else. Declared columns stay the contract, with the same review process, and everything else lands in a property bag in a declared dynamic column, the overflow column. When a field starts to matter, you promote it through the same reviewed DCR change, with weeks of its data already stored.

New at Ignite: schemaless ingestion in Azure Monitor (public preview)

Schemaless ingestion in Azure Monitor adds two settings, dynamicSchema and overflowColumn, to the header.CustomStream processor, the first step of a DCR transformation, which reads each incoming record. With dynamicSchema set to true, the header maps declared properties of a custom stream to their columns as before and collects every undeclared top-level property into the overflow column, a dynamic column you declare in the stream. The KQL processors run next and can read, reshape or project the bag like any other dynamic value.

The destination table doesn't change as new fields arrive: you add one dynamic column to your Log Analytics table once, project the bag into it, and each new source property becomes a new key there. A record with no undeclared properties gets null, a value sent in the overflow column itself is replaced, property names are case-sensitive, and capture applies only to records ingested after you turn it on. For the full behavior, see Schemaless ingestion.

Here's a transformation for a custom stream of checkout events:

{
  "name": "captureCheckoutAttributes",
  "headerProcessor": {
    "processor": "header.CustomStream",
    "configuration": {
      "streamId": "Custom-CheckoutEvents",
      "dynamicSchema": true,
      "overflowColumn": "OverflowAttributes"
    }
  },
  "processors": [
    {
      "processor": "transform.KQL",
      "configuration": {
        "expression": "source | project TimeGenerated, Body, Attributes = OverflowAttributes"
      }
    }
  ]
}

The stream declaration lists TimeGenerated, Body and OverflowAttributes (type dynamic), and the data flow's transform property names captureCheckoutAttributes. Use DCR API version 2025-05-11 or later; the Azure portal doesn't support configuring schemaless ingestion.

Promote a field when it joins the contract

With schemaless ingestion, promotion is the DCR change you make when a field joins the contract. Add the property to the stream declaration, and later records map it to a typed column instead of the bag. Update the output projection and table in the same change, or the value leaves the bag and isn't stored anywhere. Records already stored keep the field in the bag.

To get a typed value without promoting, extract it with an explicit cast. The key stays in the bag:

source
| project TimeGenerated, Body, Attributes = OverflowAttributes,
    Ring = tostring(OverflowAttributes.deployment_ring),
    BuildVersion = tostring(OverflowAttributes.context.build.version)

In our test run, this stored Ring as canary and BuildVersion as 2.0, with the bag unchanged.

Promote a field when you filter, alert or join on it, or when it needs a fixed type.

Shape the bag with bag functions

Bag functions in Azure Monitor transformations decide what the overflow column from schemaless ingestion keeps. bag_remove_keys() removes top-level keys, bag_merge() adds defaults, and pack_all(true) packs a record's populated columns into a bag (true skips null and empty values). This transformation drops retry_count and adds a default deployment_ring:

source
| project TimeGenerated, Body,
    Attributes = bag_merge(
        bag_remove_keys(OverflowAttributes, parse_json('["retry_count"]')),
        parse_json('{"deployment_ring":"unknown"}'))

In our test run, a record with deployment_ring set to canary, retry_count set to 3 and a nested context object stored Attributes as {"deployment_ring":"canary","context":{"region":"westus","build":{"version":"2.0"}}}. The captured canary wins because bag_merge() keeps the leftmost value.

bag_remove_keys() accepts top-level keys only, and bag_merge() is shallow. pack_all() can't bring back a property that was dropped before KQL ran because the stream didn't declare it, which is why capture happens in the header processor. See Bag functions.

Moving from the HTTP Data Collector API

If you're moving custom logs off the HTTP Data Collector API, schemaless ingestion in Azure Monitor (public preview) lets your services keep sending new fields without a DCR change for each one. Support for the Data Collector API ended September 14, 2026; the API still accepts data from clients that use TLS 1.2 or later, but it's no longer supported. It created a type-suffixed column for each new property, such as deployment_ring_s. The Logs Ingestion API stores only the properties your DCR declares, so a straight migration means declaring every field every service sends.

 HTTP Data Collector APILogs Ingestion APILogs Ingestion API with schemaless ingestion
New propertyCreates a type-suffixed column, such as deployment_ring_sNot stored until you declare it in the DCRStored as a key in the overflow column, such as deployment_ring
TypesOne suffix per type; a new type can create another columnDeclared in the streamDeclared columns are typed; bag keys aren't
Table schemaChanges automaticallyYou manage itYou manage it, with one dynamic column for the bag

A migration that uses schemaless ingestion looks like this:

  1. Declare the fields your queries, alerts and workbooks already use. If you migrate an existing table, project to its existing column names, such as deployment_ring_s, so those queries keep working.
  2. Capture everything else in the overflow column, and project it to a dynamic column in the table.
  3. Promote a captured field when it becomes part of the contract.

Fields that arrive after the migration are keys in the bag, so queries read them as tostring(Attributes.new_field) instead of from a new _s column. Existing column names don't change automatically.

You can migrate an existing table, which is one-way, or create a new one and run both side by side while the old data ages out. If you still send data to a migrated table through the Data Collector API, don't change its schema, because adding a column breaks legacy ingestion. For the full procedure, see Migrate from the HTTP Data Collector API to the Logs Ingestion API.

Start with one stream

Schemaless ingestion is in public preview, with no signup. Pick a custom stream whose fields change often, add a dynamic column to its table and follow Enable schemaless ingestion. Then send a record with a property your DCR has never seen and check the stored row, because a 204 only means the request was accepted. For more queries on captured fields, see Work with captured attributes.

Review what lands in the bag: captured keys aren't type-checked, and new fields can carry values you'd rather not store. Declare fields that need a type, drop unwanted keys with bag_remove_keys(), and remember that a collector running before ingestion can still drop fields.

Back to that Monday. With capture on, the payments upgrade still merges without a DCR change, and three weeks later the on-call engineer finds the issuer timeouts in decline_reason with one query. The platform team promotes the field the next day.

How do you govern the structure of logs in your organization? Tell us in the comments.

Updated Nov 12, 2026
Version 1.0
No CommentsBe the first to comment